Skip to content

minimize is cumbersome to use #14

@robryk

Description

@robryk

I would like to be able to easily get a somewhat smaller seed from a failing testcase. I've stumbled upon two related issues that make that somewhat cumbersone:

  1. There is no way to change the default setting for minimize via e.g. environment variables. It would be most convenient to be able to run a testcase with ARBTEST_SEED, possibly ARBTEST_BUDGET_MS, and e.g. ARBTEST_MINIMIZE set to ask for minimization starting from a particular seed. Currently that requires either a temporary code change (adding .minimize() to the testcase in question) or self-built mechanism of this shape.
  2. When minimize() is used, the testcase doesn't actually fail. This is slightly confusing (because the output is not printed at all without --nocapture, so it's natural to think that e.g. the specified seed is no longer failing) and somewhat surprising (because every other type of invocation of ArbTest::run fails when a failing seed is ran.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions