Skip to content

feat: Add caching support for container mode builds #18

@mmcky

Description

@mmcky

Summary

Add caching support for container mode in setup-environment to speed up lecture-specific package installation.

Current State

Container mode currently installs packages fresh each run via conda env update. This takes ~30-60 seconds per build.

Problem with actions/cache

The naive approach of using actions/cache to cache /opt/conda/envs/... doesn't work because:

  1. actions/cache is a JavaScript action - It runs on the host runner, not inside the container
  2. Container filesystem is isolated - Paths like /opt/conda/... exist inside the container but are not visible to the host
  3. Cache restore/save fails silently - The action may appear to succeed but won't actually cache anything

Potential Solutions

Option 1: Workspace-relative cache

Copy installed packages to a workspace directory (e.g., .cache/conda-pkgs/), then restore them on subsequent runs:

- name: Cache packages in workspace
  uses: actions/cache@v4
  with:
    path: .cache/conda-pkgs
    key: lecture-pkgs-${{ hashFiles('environment.yml') }}

- name: Restore packages from workspace cache
  if: steps.cache.outputs.cache-hit == 'true'
  run: cp -r .cache/conda-pkgs/* /opt/conda/envs/quantecon/lib/python*/site-packages/

Pros: Works with existing actions/cache
Cons: Complex, fragile, may cause inconsistent conda state

Option 2: Use pip cache only

Cache pip's download cache instead of installed packages:

- name: Cache pip downloads
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('environment.yml') }}

Pros: Simple, pip cache is often workspace-accessible
Cons: Only speeds up pip packages, not conda packages

Option 3: GitHub Actions cache mounted volumes

Use Docker volume mounts to expose cache directories to the container:

container:
  image: ghcr.io/quantecon/quantecon-build:latest
  volumes:
    - /home/runner/.cache:/root/.cache

Pros: Clean solution
Cons: Requires workflow changes, not transparent

Option 4: Pre-bake more packages into container

Include more lecture-specific packages directly in the container image, reducing what needs to be installed at runtime.

Pros: Fastest builds, no caching complexity
Cons: Larger container, harder to customize per-lecture

Recommended Approach

Start with Option 4 - analyze which packages are commonly installed across lectures and add them to quantecon-build container. This provides the best build time improvement with minimal complexity.

If further optimization is needed, investigate Option 3 for pip caching.

Acceptance Criteria

  • Profile current build times to establish baseline
  • Analyze common packages across lecture repos
  • Implement chosen caching strategy
  • Document any workflow changes required
  • Measure improvement in build times

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions