Custom binary + default task + missing arguments gives traceback

[SwampFalc]

I'm not 100% certain which part of the code stack is exactly the issue here, but I made a custom binary, gave it a collection of tasks with one marked default (which needs one parameter).

If I just call the binary with its default task and no parameter, I get the standard invoke error message:

$ exportit execute
'execute' did not receive required positional arguments: 'export_type'

However, if I just call the binary and rely on the fact that that one task is defined as default, then I get a traceback error:

$ exportit
Traceback (most recent call last):
  File "/home/ldt/.virtualenvs/matchit/bin/exportit", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 332, in run
    self.execute()
  File "/usr/local/lib/python2.7/dist-packages/invoke/program.py", line 480, in execute
    executor.execute(*self.tasks)
  File "/usr/local/lib/python2.7/dist-packages/invoke/executor.py", line 133, in execute
    result = call.task(*args, **call.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/invoke/tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
TypeError: execute() takes at least 2 arguments (1 given)

Which is of course a bit surprising and annoying, since I would prefer to also see the first behaviour in this case.

Do note that the custom binary was achieved by using the setup.py "entry_points/console_scripts" method.

(Let me know if you also want me to copy/paste the content of the file defining the Collection and Program instances, they're very basic so I sort of assume they're of little use...)

[bitprophet]

To clarify: so the intent here is that you want to just be able to say $ exportit and have that execute a bundled execute task? And that task has an (intentionally) required positional arg? Such that literally calling $ exportit is not actually valid, but say $ exportit some-export-type-here would be?

I'm not sure we've ever actually tested that particular conflux of features, so it's probably a bug, hopefully one that is fixable. But let me know if I've got that right, to start. Thanks!

Wondering if perhaps it's just that top-level default tasks don't play nice with arguments (IOW, it's possible this is not directly related to custom binaries), which would probably be a parser level issue.

[SwampFalc]

Yes to all the questions you asked for clarification.

Now, the fact that you didn't test this was a bit of a surprise since it seems (from my perspective) such an obvious use case, but it did lead me to think on it a bit more and there might be an issue: it would be impossible to have a default task, with a positional argument, whose value might be the same as one of the other defined tasks.

In other words, if I call $ exportit something, intending to call the default task with the argument "something", but there's also a task defined named something... then what interpretation should take precedence?

[bitprophet]

Yea, that sort of parser ambiguity is very common for the use cases this tool is concerned with! We've already had to handle at least 2-3 of them prior to now. This is just one of the less common angles on it.

I will say that I personally haven't done much with top-level default tasks, but that's mostly because I use Invoke as a make-esque tool in repositories where there's rarely/never a single obvious default "thing to do" (compared to, say, make in a builds-some-binary-for-you C repository - clearly, the default should be to Build The Thing). So that's why this area is less fleshed out.

[bitprophet]

To actually answer the question, I don't actually recall what the current intention is as it's been a while, but IIRC if the current parser context has a not-yet-filled required positional argument, the next token is used for it no matter what. That should be how it works for 'regular' tasks, anyway:

• Imagine $ program has-positional other-task, where has-positional is a task requiring a posarg, and other-task is some random other valid task name
• User A was intending to give the literal string value "other-task" to has-positional.
• User B was intending to call has-positional without any arguments (a mistake), followed by calling other-task (also w/o args).
• If we allow has-positional to "eat" the token even though it's also a valid task name, User A is happy, and User B gets confused.
  • For user B, either has-positional can't actually deal with that particular string as its positional arg, and blows up, or it does something potentially embarrassing but nonfatal, the session ends, and the user wonders what happened to the task other-task. Neither's great.
• If we instead decide that other-task forces a parser context switch, we'll get an error once Python tries calling has-positional without its required argument (in real life, the parser itself gets mad first, but let's pretend), meaning that User A is unhappy (things blow up) and User B is unhappy (things blow up).

There's a third option which is to invoke the Zen of Python and raise an exception...but that's the same as the second option at the end, neither user is happy because kaboom.

---

So, as I said earlier: this feels like a bug, or maybe two bugs, depending, and I suspect at least one of them stems from this being at the top level, or for a default-task, instead of after an explicitly named task. I'll dig when I have (more) free time and figure it out.

[SwampFalc]

For whatever it might be worth, here's the use case that has lead to this:

• I have basically a database and a need to export data from it.
• I have 7 possible datasets that could be exported.
• The export process itself requires a few steps (namely, the actual export, then the file gets encrypted, then the file gets securely transmitted to the final destination).

So for possible troubleshooting purposes, each separate step in the process is available as a task, with the default task being the combination of all subtasks in the right order. And because of the different possible datasets, that of course becomes a parameter to most tasks, so it must be a parameter of the default one.

All of that being said, you know your own tool better than anyone, so if you have a better idea of how I could implement some of this, I'd be happy to hear it.

[bitprophet]

That sounds totally reasonable to me, so again to confirm – this sounds like a bug, and what you are attempting to do should absolutely be possible under our design goals.

In terms of workarounds til this bug gets squashed (or if it turns out to be an intractable problem with the parser design, which is possible but hopefully unlikely) I'd first just try to see if 'lifting' everything up one level (so it becomes, say, exportit export sometype, exportit export.step1 sometype, exportit export.step2 sometype etc) makes it work better. (At the very least, knowing whether this works as expected will tell us more about where the bug lies.)

I'd expect that to be unacceptable for a long term workaround given you're clearly intending this to go in front of other users, and thus the extra verbosity is just no good. But it's an option.

[bitprophet]

Also, spitballing, you could experiment with making the export-type argument non-positional (e.g. give it a default None value or something) plus a manual "was it None? raise a stink" check at the top of your task(s). Same as above: not great long term, but might potentially work around the conflux of factors, and/or tell us something.

[marcstreeter]

@SwampFalc are you even able to supply a positional argument? I get the same error with the traceback, however, when I attempt to provide the 1 required positional argument it originally complained about, I only to get another error(No idea what '[whatever-I-typed]' is!). I can however supply the default task and then everything works as expected. I guess it would be somewhat hard to tell if the user is trying to supply an argument to the default task or trying to hit a different task. I only have one sole task (the default task) so it would make sense to default to it.

[marcstreeter]

I'm not sure if this is helpful (it's so easy to reproduce). I'm linking my project that has these symptoms in case you wish to see an example.

https://github.com/marcstreeter/go-multi-build