ci: introduce renovate to deal with bumping our dependencies

.github/RENOVATE_SETUP.md

@@ -0,0 +1,180 @@
+# Renovate Setup Documentation
+
+This repository uses [Renovate](https://docs.renovatebot.com/) to automatically update dependencies, including git submodules and Python packages managed in `pyproject.toml`.
+
+## What Renovate Does
+
+Renovate automatically:
+1. **Updates git submodules** by tracking the configured branches
+2. **Updates a small allowlist of Python dependencies** in `pyproject.toml`:
+   - `vllm`, `torch`, and `ray` for the core training stack
+   - `transformer-engine` and `flash-attn` for xformers compatibility
+   - `transformers` so we can track upstream releases
+   - _Everything else is frozen unless explicitly requested._
+3. **Syncs `3rdparty/*/setup.py` files** with their corresponding submodule dependencies
+4. **Regenerates `uv.lock`** after dependency updates
+5. **Pre-clones git submodules with full history** so Renovate can checkout new commits (works around `shallow=true` in `.gitmodules`)
+6. **Creates a single PR** that automatically triggers the full CI pipeline (`cicd-main.yml`)
+
+## Setup Requirements
+
+You need to set up authentication for Renovate. Choose one of the following options:
+
+### Option 1: Personal Access Token (PAT) - Quick Start
+
+**This is the easiest way to get started:**
+
+1. Create a GitHub Personal Access Token (PAT):
+   - Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
+   - Click "Generate new token (classic)"
+   - Give it a descriptive name (e.g., "Renovate Bot")
+   - Select scopes:
+     - ✅ `repo` (Full control of private repositories)
+     - ✅ `workflow` (Update GitHub Action workflows - required for github-actions manager)
+   - Click "Generate token" and copy it
+
+2. Add the token as a repository secret:
+   - Go to your repository → Settings → Secrets and variables → Actions
+   - Click "New repository secret"
+   - Name: `RENOVATE_TOKEN`
+   - Value: Paste your PAT
+   - Click "Add secret"
+
+3. You're done! The workflow will use the PAT automatically.
+
+### Option 2: GitHub App (Recommended for Organizations)
+
+**Better for rate limits and security, but requires more setup:**
+
+1. Create a GitHub App:
+   - Go to Organization Settings → Developer settings → GitHub Apps → New GitHub App
+   - Or use an existing Renovate GitHub App
+
+2. Configure the app with these permissions:
+   - Repository permissions:
+     - Contents: Read & Write
+     - Pull requests: Read & Write
+     - Workflows: Read & Write (if using github-actions manager)
+     - Metadata: Read-only
+
+3. Install the app on your repository
+
+4. Add these secrets to your repository:
+   - `RENOVATE_APP_ID`: The app ID (found on the app's settings page)
+   - `RENOVATE_APP_PRIVATE_KEY`: The app's private key (PEM format)
+
+5. The workflow will automatically detect and use the GitHub App token
+
+### 2. Grant Workflow Permissions
+
+Ensure the Renovate workflow has permission to:
+- Create and update pull requests
+- Read and write to the repository
+- Access secrets
+
+This can be configured in: `Settings` → `Actions` → `General` → `Workflow permissions`
+
+## Configuration Files
+
+### `.github/renovate.json`
+Main configuration file that defines:
+- Update schedule (daily during business hours PST)
+- Package grouping rules
+- Branch naming conventions
+- PR labels (`dependencies`, `CI:L2`)
+
+### `.github/workflows/renovate.yml`
+GitHub Actions workflow that:
+- Runs daily at 9 AM UTC (1 AM PST / 2 AM PDT)
+- Can be manually triggered with `workflow_dispatch`
+- Sets up the environment (Python, uv)
+- Executes Renovate with proper credentials
+
+### `.github/scripts/sync_submodule_dependencies.py`
+Python script that:
+- Reads dependencies from `3rdparty/*/pyproject.toml` files in submodules
+- Updates `CACHED_DEPENDENCIES` in corresponding `setup.py` files
+- Ensures consistency between submodule requirements and wrapper packages
+
+### `.github/scripts/renovate_post_update.sh`
+Bash script that runs after Renovate updates dependencies:
+1. Syncs submodule dependencies to setup.py files
+2. Runs `uv lock` to regenerate the lock file
+3. Stages changes for commit
+
+## Manual Workflow Trigger
+
+You can manually trigger Renovate at any time:
+
+1. Go to `Actions` → `Renovate` in GitHub
+2. Click `Run workflow`
+3. Optional parameters:
+   - **Log level**: Set to `debug` for verbose output
+   - **Dry run**: Enable to preview changes without creating PRs
+
+## Update Strategy
+
+Renovate now produces **one consolidated PR at a time**:
+
+| Branch prefix | Contents | Notes |
+|---------------|----------|-------|
+| `renovate/allowlist-…` | Git submodules, Docker/GitHub Action updates, and the allowlisted Python packages above | Runs on the configured weekday schedule; no other dependencies are touched until explicitly re-enabled. Renovate's built-in vulnerability PRs are disabled so everything funnels through this branch. |
+
+## Debug vs. Production Settings
+
+- `prHourlyLimit` is currently `0` **only while debugging** so Renovate can recreate PRs immediately. Set it back to `1` once we're satisfied with the configuration to avoid noisy PR bursts.
+- `prConcurrentLimit` stays at `1` to preserve the "one PR at a time" contract; raise it temporarily if you ever need parallel testing.
+
+## CI Integration
+
+When Renovate creates a PR:
+1. The PR is automatically labeled with `CI:L2` to trigger full CI testing
+2. `cicd-main.yml` runs the complete test suite
+3. All L2 tests must pass before the PR can be merged
+4. The lock file and setup.py changes are included in the PR
+
+## Troubleshooting
+
+### Renovate workflow fails
+- Check that secrets `RENOVATE_APP_ID` and `RENOVATE_APP_PRIVATE_KEY` are set
+- Verify the GitHub App is installed on the repository
+- Check workflow logs for specific error messages
+
+### Dependencies not syncing
+- Ensure submodules are properly initialized
+- Check `.github/scripts/sync_submodule_dependencies.py` logs
+- Verify that submodule `pyproject.toml` files exist and are valid
+
+### uv lock fails
+- Ensure `uv` version in workflow matches project requirements
+- Check for dependency conflicts in the update
+- Review the post-update script logs
+
+### PRs not triggering CI
+- Verify PR has the `CI:L2` label
+- Check `cicd-main.yml` configuration
+- Ensure PR is targeting the `main` branch
+
+## Customization
+
+To modify Renovate behavior:
+1. Edit `.github/renovate.json` for scheduling, grouping, or update rules
+2. Update `.github/workflows/renovate.yml` for workflow settings
+3. Modify `.github/scripts/renovate_post_update.sh` for custom post-update logic
+
+## Testing Changes
+
+Before committing Renovate config changes:
+1. Use the workflow's dry-run mode to test
+2. Check the Renovate logs for validation errors
+3. Test the post-update script locally:
+   ```bash
+   .github/scripts/renovate_post_update.sh
+   ```
+
+## References
+
+- [Renovate Documentation](https://docs.renovatebot.com/)
+- [Renovate Configuration Options](https://docs.renovatebot.com/configuration-options/)
+- [GitHub Action for Renovate](https://github.com/renovatebot/github-action)
+

.github/renovate.json

@@ -0,0 +1,119 @@
+{
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  "extends": [
+    "config:best-practices",
+    ":dependencyDashboard"
+  ],
+  "timezone": "America/Los_Angeles",
+  "cloneSubmodules": true,
+  "customEnvVariables": {
+    "NRL_AUTO_SYNC_DEPS": "1"
+  },
+  "prConcurrentLimit": 1,
+  "prHourlyLimit": 0,
+  "semanticCommits": "disabled",
+  "commitMessagePrefix": "[Renovate]",
+  "branchPrefix": "renovate/",
+  "labels": [
+    "dependencies",
+    "CI:L2"
+  ],
+  "assignees": [],
+  "reviewers": [],
+  "postUpgradeTasks": {
+    "commands": [
+      ".github/scripts/renovate_post_update.sh"
+    ],
+    "fileFilters": [
+      "uv.lock",
+      "3rdparty/**/setup.py"
+    ],
+    "executionMode": "update"
+  },
+  "packageRules": [
+    {
+      "description": "Disable Python dependency updates unless explicitly allowlisted",
+      "matchManagers": [
+        "pep621"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "Allowlisted training stack dependencies (from initial requirements)",
+      "matchManagers": [
+        "pep621"
+      ],
+      "matchPackageNames": [
+        "vllm",
+        "transformer-engine",
+        "flash-attn"
+      ],
+      "enabled": true
+    },
+    {
+      "description": "Disable Python version updates (pyenv, github-actions python)",
+      "matchManagers": [
+        "pyenv"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "Disable Python runtime version updates in GitHub Actions",
+      "matchPackageNames": [
+        "python"
+      ],
+      "matchManagers": [
+        "github-actions"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "Disable alternate Python managers that duplicate pep621 coverage",
+      "matchManagers": [
+        "pip_setup",
+        "poetry",
+        "pixi",
+        "uv"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "Disable Docker base image updates by default (enable manually when needed)",
+      "matchManagers": [
+        "dockerfile"
+      ],
+      "enabled": false
+    },
+    {
+      "description": "Group every allowed update into a single PR",
+      "matchPackagePatterns": [
+        "*"
+      ],
+      "groupName": "dependency updates",
+      "additionalBranchPrefix": "deps-",
+      "separateMajorMinor": false,
+      "separateMultipleMajor": false
+    }
+  ],
+  "git-submodules": {
+    "enabled": true
+  },
+  "pep621": {
+    "enabled": true,
+    "fileMatch": ["(^|/)pyproject\\.toml$"],
+    "updateLockFiles": false
+  },
+  "lockFileMaintenance": {
+    "enabled": false
+  },
+  "vulnerabilityAlerts": {
+    "enabled": false
+  },
+  "osvVulnerabilityAlerts": false,
+  "rangeStrategy": "bump",
+  "platformAutomerge": false,
+  "rebaseWhen": "behind-base-branch",
+  "recreateWhen": "always",
+  "forkProcessing": "enabled",
+  "includeForks": true
+}

.github/scripts/renovate_cmd.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+# Custom Renovate command that configures git safe.directory before running renovate
+# This is needed because the pre-cloned repo is owned by a different user than the container user
+
+# Mark all directories as safe (required for pre-cloned repos with different ownership)
+git config --global --add safe.directory '*'
+
+# Enable auto-sync of CACHED_DEPENDENCIES in 3rdparty setup.py files
+# This allows submodule updates to proceed - the setup.py will use submodule deps directly
+export NRL_AUTO_SYNC_DEPS=1
+
+# Run the actual renovate command
+exec renovate "$@"
+

.github/scripts/renovate_post_update.sh

@@ -0,0 +1,77 @@
+#!/bin/bash
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(realpath "$SCRIPT_DIR/../..")"
+
+echo "==================================="
+echo "Renovate Post-Update Script"
+echo "==================================="
+echo ""
+
+cd "$REPO_ROOT"
+
+# Step 1: Sync submodule dependencies to setup.py files
+echo "Step 1: Syncing submodule dependencies..."
+python3 "$SCRIPT_DIR/sync_submodule_dependencies.py"
+if [ $? -ne 0 ]; then
+    echo "ERROR: Failed to sync submodule dependencies"
+    exit 1
+fi
+echo ""
+
+# Step 2: Run uv lock to regenerate lock file
+echo "Step 2: Running uv lock..."
+
+# Install uv if not available (needed when running inside Renovate's Docker container)
+if ! command -v uv &> /dev/null; then
+    echo "uv not found, installing..."
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+    export PATH="$HOME/.local/bin:$PATH"
+fi
+
+if ! command -v uv &> /dev/null; then
+    echo "ERROR: uv is not installed or not in PATH after installation attempt"
+    exit 1
+fi
+
+# Run uv lock to regenerate the lock file
+# Note: Workspace members are defined in pyproject.toml [tool.uv.workspace].members
+# Some members point to submodule paths (e.g., Automodel-workspace/Automodel), not the parent dir
+uv lock
+if [ $? -ne 0 ]; then
+    echo "ERROR: uv lock failed"
+    exit 1
+fi
+echo ""
+
+# Step 3: Stage all changes for commit
+echo "Step 3: Staging changes..."
+git add -A 3rdparty/*/setup.py uv.lock
+if [ $? -ne 0 ]; then
+    echo "WARNING: Failed to stage files with git add"
+    # Don't exit, as Renovate might handle git operations differently
+fi
+echo ""
+
+echo "==================================="
+echo "Post-update completed successfully"
+echo "==================================="
+echo ""
+echo "Changed files:"
+git diff --cached --name-only || git status --short || echo "(git status unavailable)"
+