GPU Monitoring bug when step advances faster than train loop steps

[terrykong]

The GPU monitoring advances the wandb step faster than the main training loop causing the metrics to be ignored of past steps. The solution is to use wandb custom log axes.

https://docs.wandb.ai/guides/track/log/customize-logging-axes/

[SahilJain314]

Why does the step advance fast instead of using the last step?

[terrykong]

The step for GPU monitoring was the seconds since the start of monitoring and that logging is async from the training script. When that exceeded the training global step, wandb spit out this warning

 wandb: WARNING Tried to log to step 7 that is less than the current step 421. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
 wandb: WARNING Tried to log to step 7 that is less than the current step 421. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.

So for wandb we need to log these metrics with a different x axis