Default sample_time < clock resolution on Windows, leading to very noisy results

[qwhelan]

When running asv on Windows, it's quite common to see very noisy results like the following:

$ asv dev -b cut_timedelta
· Discovering benchmarks
· Running 4 total benchmarks (1 commits * 1 environments * 4 benchmarks)
[  0.00%] ·· Benchmarking existing-py_home_chris_anaconda3_bin_python
[ 12.50%] ··· reshape.Cut.time_cut_datetime                                                                                                                               ok
[ 12.50%] ··· ====== ==========
               bins
              ------ ----------
                4     15.6±0ms
                10     0±0ns
               1000   46.9±0ms
              ====== ==========

It's obvious from the above that we're using time.time() on Windows, which has a resolution of 15.6 ms. This could be fine if sample_time were large enough, but the default of 10 ms is only ~2/3 the clock resolution, leading to the time quantization to be especially pronounced.

Here's a log-log plot of the error ratio (std/mean) on the pandas asv benchmark suite when run on Windows:

You can see that the first benchmark with an error better than 1% has to run for ~4s, or 400x the default sample_time!

The avenues I see for fixing asv on Windows are:

1. Switch to timeit.default_timer()
2. Increase sample_time by a couple orders of magnitude
3. Dynamically switch between time.time() and time.clock() based on expected runtime

I believe option 1 is by far the best option, but it's not clear to me whether there were wall vs cpu timing issues on Windows in the past that lead to the current selection of timer.

cc @topper-123

[pv]

I agree using the high-resolution timer as default is the best solution, and don't see big problems in making the change (PR accepted). Option 2 is a bit wasteful, and option 3 may cause additional noise due to the switching of timers.

The default of using CPU time is an old decision (pre asv 0.1, before windows support), and IIUC the reasons for using it were not that strong (@mdboom maybe can comment if he can be summoned).

[mdboom]

CPU time will theoretically be less impacted by other tasks on the same CPU than wall time. But I agree, if the resolution of the timer on Windows is so poor, it might be the lesser of bad options.

[qwhelan]

@pv @mdboom Thanks for the input, I'll clean up what I have and submit a PR

[h-vetinari]

@qwhelan
Great investigation, would have never figured that out myself! :)

Any news on that PR?

[qwhelan]

@h-vetinari Please give #780 a shot! Apologies for the delay, mistakenly thought I had already published it before going on vacation.

@pv @mdboom As noted above, I've opened a PR with a first pass at the appropriate changes. If you have a suggestion as to a process-time-sensitive benchmark to check impact upon, I can investigate how much impact this has.