Skip to content

Commit f3cce97

Browse files
authored
doc: document JavaScript tool for benchmark comparison
This makes it simpler for everyone to do the statistical analysis since it doesn't require to install R and its dependencies. Refs: https://github.com/targos/node-benchmark-compare
1 parent af45be7 commit f3cce97

File tree

1 file changed

+18
-11
lines changed

1 file changed

+18
-11
lines changed

doc/guides/writing-and-running-benchmarks.md

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -62,8 +62,13 @@ from [nghttp2.org][] or built from source.
6262

6363
### Benchmark analysis requirements
6464

65-
To analyze the results, `R` should be installed. Use one of the available
66-
package managers or download it from <https://www.r-project.org/>.
65+
To analyze the results statistically, [node-benchmark-compare][] must be used.
66+
It is a Node.js script that can be installed with
67+
`npm install -g node-benchmark-compare`.
68+
69+
To draw comparison plots when analyzing the results, `R` must be installed.
70+
Use one of the available package managers or download it from
71+
<https://www.r-project.org/>.
6772

6873
The R packages `ggplot2` and `plyr` are also used and can be installed using
6974
the R REPL.
@@ -285,10 +290,10 @@ module, you can use the `--filter` option:*
285290
--no-progress don't show benchmark progress indicator
286291
```
287292

288-
For analysing the benchmark results use the `compare.R` tool.
293+
For analysing the benchmark results, use [node-benchmark-compare][].
289294

290295
```console
291-
$ cat compare-pr-5134.csv | Rscript benchmark/compare.R
296+
$ node-benchmark-compare compare-pr-5134.csv
292297

293298
confidence improvement accuracy (*) (**) (***)
294299
string_decoder/string-decoder.js n=2500000 chunkLen=16 inLen=128 encoding='ascii' *** -3.76 % ±1.36% ±1.82% ±2.40%
@@ -315,17 +320,18 @@ consider at least two stars (`**`) as the threshold, in that case the risk
315320
is 1%. If three stars (`***`) is considered the risk is 0.1%. However this
316321
may require more runs to obtain (can be set with `--runs`).
317322

318-
_For the statistically minded, the R script performs an [independent/unpaired
323+
_For the statistically minded, the script performs an [independent/unpaired
319324
2-group t-test][t-test], with the null hypothesis that the performance is the
320325
same for both versions. The confidence field will show a star if the p-value
321326
is less than `0.05`._
322327

323-
The `compare.R` tool can also produce a box plot by using the `--plot filename`
324-
option. In this case there are 48 different benchmark combinations, and there
325-
may be a need to filter the csv file. This can be done while benchmarking
326-
using the `--set` parameter (e.g. `--set encoding=ascii`) or by filtering
327-
results afterwards using tools such as `sed` or `grep`. In the `sed` case be
328-
sure to keep the first line since that contains the header information.
328+
The `compare.R` tool can additionally produce a box plot by using the
329+
`--plot filename` option. In this case there are 48 different benchmark
330+
combinations, and there may be a need to filter the csv file. This can be done
331+
while benchmarking using the `--set` parameter (e.g. `--set encoding=ascii`) or
332+
by filtering results afterwards using tools such as `sed` or `grep`. In the
333+
`sed` case be sure to keep the first line since that contains the header
334+
information.
329335

330336
```console
331337
$ cat compare-pr-5134.csv | sed '1p;/encoding='"'"ascii"'"'/!d' | Rscript benchmark/compare.R --plot compare-plot.png
@@ -560,5 +566,6 @@ Supported options keys are:
560566
[benchmark-ci]: https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
561567
[git-for-windows]: https://git-scm.com/download/win
562568
[nghttp2.org]: https://nghttp2.org
569+
[node-benchmark-compare]: https://github.com/targos/node-benchmark-compare
563570
[t-test]: https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances
564571
[wrk]: https://github.com/wg/wrk

0 commit comments

Comments
 (0)