Stop worker after wrong answers

[giomasce]

See http://www.freelists.org/post/contestms/Stopping-workers-after-wrong-answers. Basically the proposal is to make the workers at least partially score-aware, so that they can stop an evaluation after some testcases have failed, in order to save on resources.

This is difficult to do with our design. This issue is considered in very low priority.

[artikz]

I've made an initial implementation of this feature for our contest. The implementation is here: artikz@d34b6b8

I haven't submitted a pull request because it's not "public-ready", but seems it works: all existing tests passed and my manual testing shows that everything is ok (though, I've tested only batch tasks).

To turn the feature on, one need to add a third parameter "1" to each subgroup which he wants to "fail fast". For example, parameters could look like this: [[10, 5], [40, 10, 0], [50, 5, 1]]. First subgroup will be tested as usually, second subgroup also (but we said this explicitly), and the third one will be tested until a fail occurred.

Here is a short description of what is done:

1. Added "parameters" field to JobGroup, which contains "score_type_name" and "score_type_data" (dictionary of ScoreType fields: parameters and public_tests) to pass scoring data to workers.
2. Added "should_evaluate_testcase" function to ScoreType, which analyzes outcomes that are already known by this moment and decides, whether the test should be evaluated. By default it returns True, so if user does nothing, everything will be as usually.
3. Implemented "should_evaluate_testcase" for group score types.
4. Added "is_outcome_already_known" for group score types, which decides, whether the outcome of a subgroup is already known.
5. Forced workers to run tests in order.
6. Instantiated ScoreType on workers and made worker ask it, whether to run a job or just say that it's failed.

I understand, that this is possibly not the most elegant solution, but it is at least something that works.

Are you interested in this feature and is this way of implementation acceptable for you (in terms of overall system architecture)? If answer is "yes", then I will prepare the code more accurately and submit a pull request.

[artikz]

I've also pushed a commit, that allows to skip whole subtasks basing on the results of other subtasks (artikz@a5fcd7b).

It can be configured in score type parameter as a 4-th option for GroupMin, GroupMul, or 5-th option for GroupThreshold. It should contain a list of 1-based indices of subtasks which shouldn't fail to allow this subtask to be checked. And all these subtasks should have index lower than the current one (though there is no such check in the code yet). Here is an example configuration: [[25, 34, 1],[25,16, 1, [1]], [25,16, 1,[1,2]], [25,16, 1]]. First subtask cannot depend on any. Second subtasks depends on the first, third one depends on the first and second, and the last subtasks doesn't depend on any subtask.

[artikz]

The last commit achieves results similar to ones, which were proposed in this pull request: #238 (One testcase for multiple subtasks)

[lw]

Overall I like your approach. Here are my comments.

First, instead of should_evaluate_testcase I would have introduced the next_testcase_to_evaluate method on ScoreTypes: it takes the results of the testcases that have already been evaluated and the testcases still to evaluate, and returns the codename of one of the latters (or None, to stop the Worker). That's a more flexible interface (and in the future it may also allow more "conscious" ScoreTypes to evaluate testcases from the most likely to fail to the less likely).

This means that there's minimal change in EvaluationService and Worker (some new fields added to Job, some new code to instantiate the ScoreType and determine the evaluation order). It also means that all the burden is on the ScoreTypes. That's what worries me. They will need to become way more complicated. That's not a problem in itself: it's actually a thing we've been wanting for some time (to "fail fast", to include a testcase in multiple subtask without doing two evaluations, to make a subtask fail if a previous one failed, etc.). Yet the configuration of ScoreTypes (their "parameters") would become way more complex, whereas the interface to modify them isn't very admin-friendly: it's a textarea where you modify a JSON list.

I therefore think that before any of these changes can be merged into master we need to provide a more comfortable interface.

[artikz]

Thanks. I agree that next_testcase_to_evaluate is definitely a better approach and that the configuration interface is a bit more complex than it should be. I'm planning to do some work on this in May.

[artikz]

Just a note. I'm currently working on this here: https://github.com/artikz/cms/tree/stop-after-fail-2. It is the version I've already shown adapted to new evaluation mode (separate testcases evaluation). Also it supports both skipping non-important testcases and for lowering their priority (#344).

I had to modify the ScoreType parameter format, - now it is an array of objects instead of array of arrays of integers. The original format is a nightmare if we want some nice and complex features.

If I have time, I will try to implement user interface for editing this JSON in AWS, I already have some ideas for it.