Explicit support for user-level sharing of data between tasks

[brejoc]

Version 0.7 of invoke passed arguments also to the pre-tasks. Seems like this is not happening with 0.10.1.

Example:

@task
def task2(argument):
    # do something

@task1("task2")
def task1(argument):
    # do something

With version 0.7 invoke task1 --argument foo would execute task2 and then task1. This also happens with v0.10.1, but with v0.7 both tasks would get the argument "foo". This is not happening with v0.10.1.

PS: I know that the pre tasks are no longer passed as a string in the decorator.

[bitprophet]

See #135 which covers most of the where/how/why this changed - the change was done on purpose and isn't a bug per se.

You can tell the pre/post task machinery to give static values to pre-tasks by using the call function, i.e. @task(pre=[call(pretask, kwarg=val)]).

That's obviously not a full solution for the use case you're interested in, jury's still out on the absolute best way to handle it without going back to "all pre/post tasks get same args as main task" (which doesn't work well for a larger set of use cases, thus why it was removed). One way to handle it could be to make it easier to access the raw CLI argument objects from (pre)tasks.

The main problem is simply that the model of "CLI args -> func parameters" doesn't map cleanly for pre/post tasks like it does for the main task. Making it easy for the pretask to just go "ok what WAS given to the main task, I actually care" would let the user handle that mismatch any way they wanted, without forcing pre/post tasks to have the same signature as the main task.

[bitprophet]

I thought there was an open followup ticket for this general issue, but I don't see one. #170 is sort of related. #114 also insofar as context objects are almost definitely the right way to grant that CLI arg access I mentioned (we may need to make sure the arguments actually end up in the context, but that should be reasonably easy).

Leaving this open for now so we are tracking the specific need. Thanks!

[youcandanch]

@bitprophet since we're at a juncture now where contextualized tasks are now the norm, I think it makes sense to resurface context as the proper place to store arguments for later retrieval. It allows the following pattern, which I think I like:

@task(pre=[build])
def run(ctx, no_build=False):
    """Start it all up."""
    ctx.run('run something')


@task
def build(ctx):
    """Build all the things!"""
    if ctx.config.kwargs.get('no_build'):
        print('Skipping building as requested by a parent invocation.')
        return
    ctx.run('build something')

It allows allows for checking kwargs and falling back to context, as well:

@task
def build(ctx, no_static_content=False):
    """Build all the things!"""
    if no_static_content or ctx.config.kwargs.get('no_static_content'):
        print('Skipping building static content')
        return
    else:
        ctx.run('build static content')
    ctx.run('build not static content')

We're running into this with a refactor from make -> invoke that I'm shepherding now, and I've got some discretionary time to put into a patch like this. Is this something you'd generally agree is valuable? If so, I'll take a stab at it and throw a PR in.

[thebjorn]

Any particular reason you want that cumbersome api? Wouldn't ctx.no_build and ctx.no_static_content be much easier to work with?

[youcandanch]

I'm not really tied to any implementation minutae at this point, @thebjorn, and that's a way better idea, thanks! I think you'd have to go at least one level beyond just the root of Context to avoid collision and pollution of a global arena, though. Something like context.options might work there and be fairly explicit.

[thebjorn]

That makes sense (if the suggested naming convention is ctx for the context parameter, then maybe options should be spelled opts?).

Having a well-defined api for using the context as a storage are for runtime state could be very useful and could be used to allow pre-task to store values for later stages too. Eg. pre-tasks could set context.data.foo = 42 (or context.data.taskname.foo = 42) and make it available for all subsequent tasks (really for all subsequent uses of that context).

[youcandanch]

That's a great point. It's powerful and now that everything is contextualized, it's the perfect time to start figuring this stuff out. I'll put together a PR at least for the options bit shortly.

[bitprophet]

Hey folks! Yes, now that we've doubled down on the context, the next step is to make the context more usable/powerful. #309 is strongly related here re: @thebjorn's note about runtime state.

Re: @antinescience's specific q re: access to the main task's CLI args from other tasks handed its context, the place to start is probably in Executor.expand_calls or Executor.config_for, which are what currently drive the Context creation. Might, at least for "get it working enough to try it out" purposes, be as simple as ensuring the Call object gets attached to the Config or Context, since Call is what holds the result of the command-line parsing (as .args and .kwargs).

Depending on needs, we might also want to update the parser so it returns the "raw" argv chunks alongside the parsed result, but for now I'm guessing the parsed/Pythonified args/kwargs are sufficient.

(N.B. For now, all the "data" stuff is in Config, though again per #309 I need to have more of a think on how to possibly split it up in more discrete "from conf files/etc", "from CLI args", "set by the user at runtime via another task" etc. This probably means additional attributes/members on Context alongside config, but again, not sure.)

(Also of note, Context acts as a proxy to its Config so ctx.foo is the same as ctx.config.foo - that should be mentioned in the docs, so if neither of you noticed, please let me know if you've suggestions for making that more visible!)

[youcandanch]

That is all super helpful, thanks. Considering I was introduced to invoke all of, like, a week ago, I've been taking some time just to read through everything and get familiar. Hopping into expand_calls was where I was leaning, but there's going to have to be some modification to ParseMachine (specifically this bit: https://github.com/pyinvoke/invoke/blob/master/invoke/parser/parser.py#L202) in order to make sure that, if you're passing flags through to pre commands that don't exist on the main invocation, you're able to do so without getting that raised error. Does that sound about right?

Going to be locked down pretty hard time-wise with my day job this week, but I'll find some time to get at least a PR for talking point's sake in by week's end.

[thebjorn]

@bitprophet I did notice the Context/config/proxy implementation detail when first reading the docs, but while the Context vs Config probably makes a lot of sense from an implementation point of view, speaking as a user, the distinction doesn't "add" anything. Because of #309 I had stopped using the context for storage, since it made it necessary to do state-storage differently depending on the context ;-) The config-proxy part also makes it much harder to create a new/fresh Context - which will become important now that we're fully contextualized: You'll either have to pass ctx through all functions touching tasks, or figure out how to create a ctx when you need to call a task from a non-task. That Con-text and Con-fig share a prefix doesn't help. Maybe abusing argv, as in Context.argv, would help (i.e. args that may come from the command line, but potentially have defaults/be overridden by...). A natural extension would then be Context.argv.raw_command_line

It might also be an idea to have a distinction between configuration (ctx.argv) and state (ctx.data). An example here is the jQuery distinction between .attr() and .prop(), where attr is the text value written in the html for the attribute of a tag and prop is the data attribute of the DOM object created by the html tag. You can set a prop to the int value 42, and get it back as an int.

In any case it might be good to reserve a set of properties on ctx for potential future use (so you don't have to break backwards compatibility after 1.0). ctx.config and ctx.run should probably already be reserved (now you can do ctx.config = 42...)