-
-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Summary
Add caching support for container mode in setup-environment to speed up lecture-specific package installation.
Current State
Container mode currently installs packages fresh each run via conda env update. This takes ~30-60 seconds per build.
Problem with actions/cache
The naive approach of using actions/cache to cache /opt/conda/envs/... doesn't work because:
actions/cacheis a JavaScript action - It runs on the host runner, not inside the container- Container filesystem is isolated - Paths like
/opt/conda/...exist inside the container but are not visible to the host - Cache restore/save fails silently - The action may appear to succeed but won't actually cache anything
Potential Solutions
Option 1: Workspace-relative cache
Copy installed packages to a workspace directory (e.g., .cache/conda-pkgs/), then restore them on subsequent runs:
- name: Cache packages in workspace
uses: actions/cache@v4
with:
path: .cache/conda-pkgs
key: lecture-pkgs-${{ hashFiles('environment.yml') }}
- name: Restore packages from workspace cache
if: steps.cache.outputs.cache-hit == 'true'
run: cp -r .cache/conda-pkgs/* /opt/conda/envs/quantecon/lib/python*/site-packages/Pros: Works with existing actions/cache
Cons: Complex, fragile, may cause inconsistent conda state
Option 2: Use pip cache only
Cache pip's download cache instead of installed packages:
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ hashFiles('environment.yml') }}Pros: Simple, pip cache is often workspace-accessible
Cons: Only speeds up pip packages, not conda packages
Option 3: GitHub Actions cache mounted volumes
Use Docker volume mounts to expose cache directories to the container:
container:
image: ghcr.io/quantecon/quantecon-build:latest
volumes:
- /home/runner/.cache:/root/.cachePros: Clean solution
Cons: Requires workflow changes, not transparent
Option 4: Pre-bake more packages into container
Include more lecture-specific packages directly in the container image, reducing what needs to be installed at runtime.
Pros: Fastest builds, no caching complexity
Cons: Larger container, harder to customize per-lecture
Recommended Approach
Start with Option 4 - analyze which packages are commonly installed across lectures and add them to quantecon-build container. This provides the best build time improvement with minimal complexity.
If further optimization is needed, investigate Option 3 for pip caching.
Acceptance Criteria
- Profile current build times to establish baseline
- Analyze common packages across lecture repos
- Implement chosen caching strategy
- Document any workflow changes required
- Measure improvement in build times
Related
- PR feat: container-aware setup-environment with lean container variant #17: Container-aware setup-environment