@@ -62,8 +62,13 @@ from [nghttp2.org][] or built from source.
6262
6363### Benchmark analysis requirements
6464
65- To analyze the results, ` R ` should be installed. Use one of the available
66- package managers or download it from < https://www.r-project.org/ > .
65+ To analyze the results statistically, [ node-benchmark-compare] [ ] must be used.
66+ It is a Node.js script that can be installed with
67+ ` npm install -g node-benchmark-compare ` .
68+
69+ To draw comparison plots when analyzing the results, ` R ` must be installed.
70+ Use one of the available package managers or download it from
71+ < https://www.r-project.org/ > .
6772
6873The R packages ` ggplot2 ` and ` plyr ` are also used and can be installed using
6974the R REPL.
@@ -285,10 +290,10 @@ module, you can use the `--filter` option:*
285290 --no-progress don't show benchmark progress indicator
286291```
287292
288- For analysing the benchmark results use the ` compare.R ` tool .
293+ For analysing the benchmark results, use [ node-benchmark- compare] [ ] .
289294
290295``` console
291- $ cat compare-pr-5134.csv | Rscript benchmark/compare.R
296+ $ node-benchmark-compare compare-pr-5134.csv
292297
293298 confidence improvement accuracy (*) (**) (***)
294299 string_decoder/string-decoder.js n=2500000 chunkLen=16 inLen=128 encoding='ascii' *** -3.76 % ±1.36% ±1.82% ±2.40%
@@ -315,17 +320,18 @@ consider at least two stars (`**`) as the threshold, in that case the risk
315320is 1%. If three stars (` *** ` ) is considered the risk is 0.1%. However this
316321may require more runs to obtain (can be set with ` --runs ` ).
317322
318- _ For the statistically minded, the R script performs an [ independent/unpaired
323+ _ For the statistically minded, the script performs an [ independent/unpaired
3193242-group t-test] [ t-test ] , with the null hypothesis that the performance is the
320325same for both versions. The confidence field will show a star if the p-value
321326is less than ` 0.05 ` ._
322327
323- The ` compare.R ` tool can also produce a box plot by using the ` --plot filename `
324- option. In this case there are 48 different benchmark combinations, and there
325- may be a need to filter the csv file. This can be done while benchmarking
326- using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or by filtering
327- results afterwards using tools such as ` sed ` or ` grep ` . In the ` sed ` case be
328- sure to keep the first line since that contains the header information.
328+ The ` compare.R ` tool can additionally produce a box plot by using the
329+ ` --plot filename ` option. In this case there are 48 different benchmark
330+ combinations, and there may be a need to filter the csv file. This can be done
331+ while benchmarking using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or
332+ by filtering results afterwards using tools such as ` sed ` or ` grep ` . In the
333+ ` sed ` case be sure to keep the first line since that contains the header
334+ information.
329335
330336``` console
331337$ cat compare-pr-5134.csv | sed ' 1p;/encoding=' " '" ascii" '" ' /!d' | Rscript benchmark/compare.R --plot compare-plot.png
@@ -560,5 +566,6 @@ Supported options keys are:
560566[ benchmark-ci ] : https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
561567[ git-for-windows ] : https://git-scm.com/download/win
562568[ nghttp2.org ] : https://nghttp2.org
569+ [ node-benchmark-compare ] : https://github.com/targos/node-benchmark-compare
563570[ t-test ] : https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances
564571[ wrk ] : https://github.com/wg/wrk
0 commit comments