bypass task execution based on hash/timestamp

[deanmalmgren]

I've been looking for an alternative to GNU's make for managing a sequence of data analysis tasks and was delighted to discover invoke this morning!

One nice thing about GNU's make is that it only runs build tasks if the target's timestamp (other tools use a hash of the target's contents) is out of date with the build task's dependencies. This is convenient because if we have a dependency sequence A -> B and task A takes a very long time to run (hours or days), then its nice to bypass step A in subsequent build steps. Example:

make      # runs A and then B
edit B
make      # runs B but does not run A because A hasn't changed since last build

Does invoke have similar functionality or any plans to incorporate that functionality if it doesn't exist?

My understanding is that an equivalent invoke script would look something like this:

@task
def A():
    time.sleep(24*60*60) # a very long running task
    print("A is done")

@task(pre=['A'])
def B():
    print("B is done")

@task(pre=['A', 'B'])
def all():
    print("all is done")

but the equivalent output is not the same

invoke all     # runs A then B
edit B
involke all    # runs A then B again :(

[bitprophet]

I had no concrete plans for this, mostly because while it is a thing Make specifically does, it's not functionality myself or most of the target userbase seems to need often.

However I'm certainly not opposed to adding it, depending on how the implementation needs to work. I don't know exactly how Make does this, I assumed it used the target file paths specified + checked FS timestamp, but your example doesn't involve files so not sure what else you're describing?

Offhand I would imagine Invoke would implement this as some sort of file path (or test/check callback) given to @task() as an e.g. creates= or checks= kwarg. Then the task execution bits would compare timestamps or invoke the callbacks and skip calling the task, as needed.

[deanmalmgren]

I like the idea of adding some kwargs to the @task() decorator. That seems like a really reasonable way to do things. An alternative approach could be to have two separate decorators @creates(*filenames) and @checks(*filenames). Since you're more familiar with the codebase and syntax style of the package, do you have a preference for the kwarg approach?

We're going to be jamming on a project the next few weeks which will give me a good opportunity to mock something up in the wild. I'll be sure to let you know what we come up with and add the PR here for your feedback, etc.

[bitprophet]

I've not settled into a strong preference for "how many decorators is too many?" yet - clearly many/most @task() kwargs could live as distinct decorators, and sometimes using them as such might be a useful thing to do (especially if one wanted to do functional things like combining them).

On the other hand, having too many ways to do things can add mental & API overhead, so I try not to just splat them out willy-nilly.

For now I've stuck with "just @task kwargs for now, can always add syntactic sugar later if necessary." So let's do that for now.

[deanmalmgren]

Thanks for sharing your feedback on this. I admittedly started to think about this and even work on it a bit when we were working on our project in December. Unfortunately, I also realized that I'm pretty sure invoke isn't the right tool for the use cases I have in mind. Having to specify pre + checks (or teach invoke how to automatically infer that relationship) for each task adds a lot of non-information carrying syntax.

I've started to toy around with designing a separate tool that specifically addresses data workflow use cases. Of course, there are other tools that address similar use cases, and it may make more sense to extend invoke to address this use case. Have any thoughts to share before I rip off the band-aid and write something from scratch?

[brechtm]

For what it's worth, I was also expecting this functionality from a make-like tool.

[mohnen]

I know that I am late to the show, but have a look at pyinvokedepends

[gvwilson]

Also late to the party, but need the same functionality ("only update X if Y has changed"). pyinvokedepends would do what I want, but does not seem to play nicely with optional extra per-task arguments: see https://github.com/mohnen/pyinvokedepends/issues/1 for an example. Happy to help work on a fix, but I don't know whether this is a bug in invoke or in pyinvokedepends - tracing through the actions is complex.

[rberger]

@mohnen Seems that pyinvokedepends has disappeared from GitHub

[maxsegl]

This feature is a must have for me. At the moment I use GNU Make for our lengthy, stepwise calculations. But I would like to use rather a all-python solution. This is the last bit, which stops me from using pyinvoke... Is this issue stale or is there any chance to have this implemented?