I would like to be able to easily get a somewhat smaller seed from a failing testcase. I've stumbled upon two related issues that make that somewhat cumbersone:
- There is no way to change the default setting for minimize via e.g. environment variables. It would be most convenient to be able to run a testcase with
ARBTEST_SEED, possibly ARBTEST_BUDGET_MS, and e.g. ARBTEST_MINIMIZE set to ask for minimization starting from a particular seed. Currently that requires either a temporary code change (adding .minimize() to the testcase in question) or self-built mechanism of this shape.
- When
minimize() is used, the testcase doesn't actually fail. This is slightly confusing (because the output is not printed at all without --nocapture, so it's natural to think that e.g. the specified seed is no longer failing) and somewhat surprising (because every other type of invocation of ArbTest::run fails when a failing seed is ran.
I would like to be able to easily get a somewhat smaller seed from a failing testcase. I've stumbled upon two related issues that make that somewhat cumbersone:
ARBTEST_SEED, possiblyARBTEST_BUDGET_MS, and e.g.ARBTEST_MINIMIZEset to ask for minimization starting from a particular seed. Currently that requires either a temporary code change (adding.minimize()to the testcase in question) or self-built mechanism of this shape.minimize()is used, the testcase doesn't actually fail. This is slightly confusing (because the output is not printed at all without--nocapture, so it's natural to think that e.g. the specified seed is no longer failing) and somewhat surprising (because every other type of invocation ofArbTest::runfails when a failing seed is ran.