Performance regression tests

[T1mVo]

Description

Add performance regression tests to tytanic to detect unintended slowdowns in Typst compilation and layout.

While tytanic already covers functional correctness well, there is currently no structured way to prevent performance regressions. This feature would make performance a first-class, testable property.

I created a possible solution to this down below although this approach has a problem with machine-dependent variability.

Possible implementation

• Introduce a new test kind: performance
• Run performance tests multiple times to reduce noise (warmup + measured runs)
• Store reference values in ref.json
  • Required: reference compilation time
  • Optional (future): memory usage, layout time, total time, etc.
• Compare measured values against the reference using a configurable tolerance
  • Example: fail if regression exceeds ±5%

Workflow

Create a new performance test

tt new --performance performance-test

Resulting file structure:

.
└── tests
    └── performance-test
        ├── ref.json
        └── test.typ

Initial ref.json:

{
  "compilation_time_ms": 0
}

Implement test and record baseline

tt update performance-test

Output:

  Starting 1 test (run ID: e3c1682e-9cd6-4cba-90f3-aaa53089fc85)
      pass [    900ms] performance-test
──────────
   Summary [    900ms] 1/1 tests run: all passed

This updates ref.json to:

{
  "compilation_time_ms": 240
}

Detect a performance regression

After modifying the test to increase compilation time:

tt run performance-test

Output:

  Starting 1 test (run ID: e3c1682e-9cd6-4cba-90f3-aaa53089fc85)
      fail [    940ms] performance-test
            240ms -> 300ms (+25%)
──────────
   Summary [    940ms] 0/1 tests run: 1 failed

Problems & considerations

• Machine-dependent variability
  Compilation time varies across machines and environments, so performance tests are only meaningful when run under comparable conditions.
  • Possible mitigations:
    • Store a identifier in ref.json for each machine that ran the test and give it its own reference compile time.
    • Just run these tests in on one specific machine.
    • Compare to compiling a previous commit. This requires a vcs though.

[tingerrr]

I think this is generally a good idea, but there are some things to consider:
Like you already mentioned, variability in execution environment is one factor, but perhaps it's not as much of a factor as one might think? Lots of open source projects use CI to test for performance regressions, since CI is fairly consistent this works fine most of the time.

If benchmarking locally is that big of a deal for some people we coudl investigate whether adding some kind of local only reference to compare to is worth it. This reference could be used to extrapolate the result that CI will likely confirm.

Actually saving a per-machine reference sounds like a bad idea. Either every unique machine would have to be benchmarked for each change, requiring some sort of partial reference removal mechanism, or Tytanic would need to handle holes in the references, creating results with wildely differing baselines.

I also don't think VCS integration (comparison to previous commit) is the right choice here, because this massively increases the maintenance surface for VCS support while also making it harder to support users who don't use one of the supported VCSs.

The actual problem would then be, how does decide which machine serves as the baseline and storethat machine's reference time, especially when this is a CI runner?

I also think this presents an interesting case for how Tytanic currently executes tests, all tests share the sme comemo cache, this means the first test that is run will do most of the work when it accesses certain parts of the project that is being tested, subsequent tests benefit from comemo. This is also the reason Tytanic does not follow the cargo-nextest model of one process per test. The runner could purge the cache between these tests (ideally running them at the beginning or end to let the other tests still benefit from the caching) and event test the cold and warm compile times this way.

Finally, I think this also comes with a new way for tests to fail, which is performance improvement, this must be at the very least marked as a good thing in the output, otherwise people could miss this during iterative testing.

---

An idea I briefely entertained is that of measuring performance by using operation metrics that Typst could report, especially machine independent ones such as an "operation count" (from the perspective of the Typst VM) during evaluation. This would require upstream changes, but is unlikely to land without tangible benefit for Typst itself. It would also likely cause a performance regression itself due to the additional overhead of maintaining these metrics during the compilation process, so it would probably have to be opt-in, which still carries maintenance burden.

There's also the issue of some oeprations being more costly than others, the sme operaiton count for example could lead to vastly different compilation times.