[RFC] consolidate CI code into shell scripts

[jameslamb]

Is your feature request related to a problem? Please describe.

All of this project's CI configuration currently lives in a YAML file, https://github.com/stitchfix/hamilton/blob/main/.circleci/config.yml.

This introduces some friction to development in the following ways:

• adding a new CI job involves adding a new block of YAML that is mostly the same as the others (see, for example, e2ad136)
• running tests locally involves copying and pasting commands from that YAML file
• duplication of code across jobs makes it a bit more difficult to understand what is different between them

Describe the solution you'd like

I'd like to propose the following refactoring of this project's CI jobs:

• put CI code into one or more shell scripts in a directory like .ci/
  • using environment variables to handle the fact that some jobs are slightly different from others (e.g. the dask jobs don't require installing ray)
• change .circlci/config.yaml so that it runs those shell scripts
• document in CONTRIBUTING.md how to run the tests in Docker locally, with commands that can just directly be copied and run by contributors, like this:

  •  docker run \
         --rm \
         -v $(pwd):/app \
         --workdir /app \
         --env BUILD_TYPE="dask" \
         -it circleci/python:3.7 \
          bash .ci/test.sh

Describe alternatives you've considered

Using a Makefile instead of shell scripts could also work for this purpose, but in my experience shell scripts are understood by a broader range of people and have fewer surprises around quoting, interpolation, and exit codes.

Additional context

A similar pattern to the one I'm proposing has been very very useful for us in LightGBM.

Consider, for example, how many different job configurations the CI for that project's R package uses (https://github.com/microsoft/LightGBM/blob/3ad26a499614cf0af075ce4ea93b880bcc69b6bb/.github/workflows/r_package.yml) and how little repetition there is across jobs.

If you all are interested in trying this out, I'd be happy to propose some PRs.

Thanks for considering it!

[elijahbenizzy]

So, this seems totally reasonable as an approach -- slightly decoupled from circleCI, allowing for easier running of stuff locally (especially outside of a docker image). FWIW, however, using the circleci local client will get the same thing done if you want, for example:

circleci local execute --job dask-py36

Does everything. So, which do you think is cleaner? The fact that similar patterns are used by other succesful open-source projects is a big win.

[skrawcz]

Link to circleci command line - https://circleci.com/docs/2.0/local-cli/

[skrawcz]

but I guess circleci debug loop is slow -- we might need to add caching -- and I'm not sure you can't just stick in a pdb.set_trace() statement either 🤔 ... might need what you suggest.

[elijahbenizzy]

Also this decouples us from circleci which is nice if we want to change up CI tooling. I'm in favor 👍

[jameslamb]

Ok yeah, thanks for considering it! The points @skrawcz made in #114 (comment) also favor this approach...I've found it really powerful to be able to change a pytest to a pytest --pdb (or manually throw in a pdb.set_trace()) and step into a debugger to understand what's going on in tests.

I'll put up some PRs showing more specifically what I'm proposing here.

[elijahbenizzy]

@jameslamb curious -- thoughts on this? Still could be a nice approach, although there's definitely some circleci specific logic in the code... But I think its a nice abstraction.

[jameslamb]

Ha I was literally just looking at this! I'd like to start working on this next, if you're still open to it.

[elijahbenizzy]

@jameslamb yeah! That would be great :) Thanks for all your contributions btw

[jameslamb]

Ok @elijahbenizzy @skrawcz I put up a proposal in #225. Thanks for considering this!