run.md

run.sh accepts several options:

Run all tests using the DEFAULT scenario

./run.sh

Run only one file

./run.sh tests/test_waf.py

If the test contains @scenarios.SCENARIO_NAME such as @scenarios.integrations, then the ./run.sh needs to be adjusted to the following:

./run.sh SCENARIO_NAME tests/path_to_test.py

# Example: for @scenarios.integrations in tests/integrations/test_sql.py
./run.sh INTEGRATIONS tests/integrations/test_sql.py

Run only one class, or one method

./run.sh tests/parametric/test_waf.py::Test_WAFAddresses

# and one method:
./run.sh tests/parametric/test_waf.py::Test_WAFAddresses::test_post_json_value

Run a scenario

Without providing a scenario argument, only the tests without any @scenario decorator will be executed. We provide a scenario name to run.sh to run tests decorated with the given scenario.

./run.sh <SCENARIO_NAME>
# e.g, "LIBRARY_CONF_CUSTOM_HEADER_TAGS" to run the LIBRARY_CONF_CUSTOM_HEADER_TAGS scenario

Run a scenario group

You can run a group of scenarios as they are defined in scenario_groups.yml

./run.sh <SCENARIO_GROUP_NAME>
# e.g, "APPSEC_SCENARIOS" to run most of APPSEC scenarios

Force execute disabled tests

Test items are skipped (or not) based on declarations in tests class, using manifest, or @bug/flaky/missing_feature decorators.

You can force a disabled test to execute using the -F option, e.g. ./run.sh MY_SCENARIO -F tests/feature.py::Test_Feature -F tests/feature.py::Test_FeatureEdgeCase

A common use-case is if a feature covered by a test is currently in progress for your library. Once progress is complete, follow the enable-test.md doc to enable the test in CI.

Skip empty scenarios

The --skip-empty-scenario option will deselect all tests if the current scenario contains only disabled tests (see glossary for terminology).

This option can also be activated with the environment variable SYSTEM_TESTS_SKIP_EMPTY_SCENARIO=True

Spawn components, but do nothing

./run.sh <SCENARIO_NAME> --sleep

# play with the weblog, and look inside logs_<scenario_name>/interfaces/ what's happening