Performance test suite

[razzeee]

I'm currently using https://github.com/Razzeee/tree-sitter-elm/blob/master/script/parse-examples-full to parse all of the packages on the elm package website and some hand picked open source applications.

I think it would be nice to have this for elm-format too. I would try to print the timings for each, so that we can see how changes to elm-format impact these and we can reference back to changes as log as the CI logs are still around.

I'm currently using github actions https://github.com/Razzeee/tree-sitter-elm/commit/c8fd1b94446bfaada9b3f39b3b60fcfe2374dbb0/checks?check_suite_id=303385075 which I would recommend as we can run the script on linux and macos then (travis would also work, but feels slower and would be more work for me) , unfortunatly due to it being a bash script windows is not possible.

Let me know what you think and I will start working on it.

[avh4]

I would love to have some way of making performance comparisons of code changes.

I used to print out the runtime for formatting all the example files, as you can see in https://travis-ci.org/avh4/elm-format/builds/565806309:

## good/AllSyntax/0.18/Module.json

real	0m0.067s
user	0m0.046s
sys	0m0.025s
Checksum: fa5a3593293de41bdda084937194bce2  formatted.json

but that got removed when I switched to shake for the builds.

In #646 I just added a way to get profiling info for all the example files under tests/test-files/good/**.elm. Once that merges, the workflow would be:

• checkout your baseline version, and run stack runhaskell Shakefile.hs -- profile
• checkout your new version (and/or make any relevant changes), and run stack runhaskell Shakefile.hs -- profile again
• you now have before and after profiling results under ./_profile/**

I do have a couple real-world example files under tests/test-files/good/Elm-<version>/<username>/<project>, but it would be nice to profile against more than that.

Are there particular projects you think could be most useful to use as benchmarks? Do you think there's particular value in benchmarking all published packages?

I think if something like this is added, I'd like it to integrate with the Shakefile I have for doing performance changes (which could certainly be improved). The could actually currently be done simply by downloading any elm files you want into tests/test-files/good/Elm-0.19/** and then running stack runhaskell Shakefile.hs -- profile.

As I mentioned above, I used to print the execution time of formatting each example file in the CI output, thinking that it would be nice to compare those changes over time, but I never ended up actually doing anything with those logs. My opinion at the moment is that it would be more common to want to locally compare performance of your current work against a specific previous version, and that it's probably best to run the profiling on the same machine. But I'd be curious to hear about your experience from optimizing elm-tree-sitter.

[razzeee]

Well I'm not sure if it will be immediately useful for every code change you do. But if we actually cast a wide net, we might be able to catch some edge cases, that make elm-format slow. So in that stage I would not be comparing old vs new, but see if there are some values that seem weird or just too long.

I'm basically inspired by #643 and knowing that elm-format is too slow for vscode (as you likely need to change the default format timeout).

Even if we'll mostly be running against already formatted code, it might yield some knowledge. We could also run it as an additional CI job, that does not fail a build or not run it at all, but have the file available.

[avh4]

I was looking at this for 0.8.7, but I'm not sure the best way to do it... I think any useful metrics would really need to compare two specific versions of elm-format. The local dev scripts kinda have support for doing that, as running dev/build.sh profile will put results into _profile/ and they are kept per-git-commit` so you can compare between commits. But I'm not really sure what to do that's useful for CI, or to automate comparison.