Is your feature request related to a problem? Please describe.
The team cannot currently monitor and enforce minimum code coverage (as a proxy for functional behavioral coverage) in automated tests.
Describe the solution you'd like
Suggested milestones
Technical hints / guidance
Top-level components needed for code-coverage support
coverage.proto: Describes the new scripting data structures needed for code coverage.CoverageRunner.kt: A new scripting utility for running code coverage for a selected target.CoverageReporter.kt: A new scripting utility for generating an HTML report of code coverage.RunCoverage.kt: A new script for running code coverage locally.scripts/src/java/org/oppia/android/scripts/testfile/TestFileCheck.kt: Existing utility that needs to be updated to include support for code coverage enforcement.scripts/src/java/org/oppia/android/scripts/proto/script_exemptions.proto: Existing exemptions definition that needs to be updated per the TestFileCheck changes..github/workflows/unit_tests.yml: Existing CI workflow that needs to be updated in order to support code coverage..github/workflows/workflow_canceller.yml: Existing CI workflow that needs to be fixed.- (Other files for wiki changes).
Some key technical notes
- This spec overview does not include details for the Yaml or wiki changes.
- The test file exemptions textproto file will need to be rebuilt. The simplest way to do this would be to update TestFileCheck to include a text proto regenerate argument (as used in other scripts).
- workflow_canceller.yml changes may required manually introducing support for workflow cancellation, or finding an available open source utility that works correctly for dynamic matrix jobs.
- unit_tests.yml will probably be extended to introduce a new job that blocks on the test run (in the same way as check_tests) except it (note that this may require a new script not spec'ed out in this overview):
-
- Intreprets the same matrix data as the main test run but omits the failing tests using the outputs from the test runs themselves.
-
- Starts a new test matrix that runs the
RunCoverage.kt script for each affected file corresponding to each affected test. For example, if StateFragmentTest.kt is one of the affected tests, then RunCoverage would be run for StateFragment.kt (which would then correspond to StateFragmentTest per its behavior).
- Note that the min enforced coverage for now should be something very modest, like 10%. This number will be increased in future changes outside the scope of this project (as the number may not be believable until we understand the limitations of what can be covered by JaCoCo).
BazelClient.kt may need to be updated to include a function for running coverage on a specific target & output the standard output lines from that (which can then be parsed to get the report to generate the CoverageReport proto object).- All new components require new corresponding test files. All updated components will need their tests updated based on the changes to those components.
Suggested files to add/change
coverage.proto:
message CoverageReport {
string bazel_test_target = 1; // The Bazel test target that was run.
repeated CoveredFile covered_file = 2; // Files with some coverage in this report.
}
message CoveredFile {
string file_path = 1; // Relative to the project root.
string file_sha1_hash = 2; // SHA-1 hash of the file content at the time of report (to guard against changes).
repeated CoveredLine covered_line = 3; // Lines of code covered in the report.
}
message CoveredLine {
int32 line_number = 1; // 0-starting line number of the covered line.
Coverage coverage = 2; // Detected coverage.
enum Coverage {
FULL, // This line was fully covered by the test.
PARTIAL, // This line was partially covered by the test, indicating some branches not being covered.
NONE // This line was not executed during the test.
}
}CoverageReporter.kt:
class CoverageReporter {
fun generateRichTextReport(report: CoverageReport, format: ReportFormat): String
fun computeCoverageRatio(file: String, report: CoverageReport): Float // Returns in [0, 1].
enum class ReportFormat {
MARKDOWN,
HTML
}
}CoverageRunner.kt:
class CoverageRunner {
// Uses bazel coverage to run code coverage on the specified test target & interprets the results to generate and return a CoverageReport.
fun runWithCoverageAsync(bazelTestTarget: String): Deferred<CoverageReport>
}RunCoverage.kt:
// Usage:
// bazel run //scripts:run_coverage -- <repo_root> <relative_test_file> <report_output_dir> [min_coverage=<int_perc>] [format=markdown,html]
//
// Examples:
// bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports
// bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20
// bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=markdown
// bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=html
// bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20 format=markdown,html
//
// The script:
// 1. Finds the corresponding test file specific to the file to be checked.
// a. If the file has no test & isn't exempted, the script should fail.
// b. If the file has no test & is exempted, the script should output as such.
// 2. Uses CoverageRunner to run the corresponding test file to generate a report.
// 3. Outputs code coverage percentage based on the report.
// a. If the file has a code coverage percentage exemption, it's also outputted at this point.
// 4. If formats are specified, CoverageReporter should be used to generate reports for each specified format in the destination report_output_dir directory.
// 5. If min_coverage is specified, compares the computed code coverage with the minimum specified. The script should fail iff the file has less than the minimum specified coverage.
fun main(vararg args: String)
class RunCoverage {
fun runCoverage(targetFile: String, outputFormats: List<CoverageReporter.ReportFormat>)
}script_exemptions.proto:
// The old TestFileExemptions proto (for reference).
message TestFileExemptionsOld {
repeated string exempted_file_path = 1;
}
message TestFileExemptions {
repeated TestFileExemption test_file_exemption = 1;
message TestFileExemption {
string exempted_file_path = 1;
oneof exemption_type {
bool test_file_not_required = 2;
int32 override_min_coverage_percent_required = 3;
}
}
}
Describe alternatives you've considered
No response
Additional context
This is the high-level tracking issue corresponding to https://github.com/oppia/oppia/wiki/Google-Summer-of-Code-2024#41-code-coverage-support-and-enforcement.
Note that this project includes work that relate to the following issues: #1497, #1726, #1727, and #1728.
Is your feature request related to a problem? Please describe.
The team cannot currently monitor and enforce minimum code coverage (as a proxy for functional behavioral coverage) in automated tests.
Describe the solution you'd like
Suggested milestones
Milestone 1: Introduce a new script to compute a per-unit code coverage percentage for a single file.
Milestone 2: Integrate code coverage checking.
Technical hints / guidance
Top-level components needed for code-coverage support
coverage.proto: Describes the new scripting data structures needed for code coverage.CoverageRunner.kt: A new scripting utility for running code coverage for a selected target.CoverageReporter.kt: A new scripting utility for generating an HTML report of code coverage.RunCoverage.kt: A new script for running code coverage locally.scripts/src/java/org/oppia/android/scripts/testfile/TestFileCheck.kt: Existing utility that needs to be updated to include support for code coverage enforcement.scripts/src/java/org/oppia/android/scripts/proto/script_exemptions.proto: Existing exemptions definition that needs to be updated per theTestFileCheckchanges..github/workflows/unit_tests.yml: Existing CI workflow that needs to be updated in order to support code coverage..github/workflows/workflow_canceller.yml: Existing CI workflow that needs to be fixed.Some key technical notes
RunCoverage.ktscript for each affected file corresponding to each affected test. For example, if StateFragmentTest.kt is one of the affected tests, then RunCoverage would be run for StateFragment.kt (which would then correspond to StateFragmentTest per its behavior).BazelClient.ktmay need to be updated to include a function for running coverage on a specific target & output the standard output lines from that (which can then be parsed to get the report to generate the CoverageReport proto object).Suggested files to add/change
coverage.proto:
CoverageReporter.kt:
CoverageRunner.kt:
RunCoverage.kt:
script_exemptions.proto:
Describe alternatives you've considered
No response
Additional context
This is the high-level tracking issue corresponding to https://github.com/oppia/oppia/wiki/Google-Summer-of-Code-2024#41-code-coverage-support-and-enforcement.
Note that this project includes work that relate to the following issues: #1497, #1726, #1727, and #1728.