Hyperparameter Evolution: load dataset every time

[TommyZihao]

when run train.py --evolve, the code will load and analyze datasets every time before training.
On default, it will load 300 times.
As we all know, these are absolutely the same dataset, no need to load and analyze again and again. This will waste a lot of time.

Can we figure out a solution to solve this?

[glenn-jocher]

The train() function within train.py operates independently each run, therefore treating each evolution training identically to a normal training, simplifying the codebase.

Dataset scanning only occurs on first use or modification of a dataset.

Dataset loading occurs constantly throughout training every batch, this is normal unless --cache is used. --cache has already been multithreaded and should not present a time constraint should you opt to use it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.