Set preserve_rng_state to True if there is dropout

[chrisc36]

This is consistent with the documentation which states it should be true when deterministic random behavior is required.

[dirkgr]

LGTM, but I would love to hear from @epwalsh , just in case both @chrisc36 and I are confused about how this works?

[epwalsh]

I think this is correct. The documentation is a little confusing though. The official description of this parameter is worded like it has the opposite effect:

That might explain why we had it the way we did.

[2015aroras]

Can't we just set this to True? Unless this is demonstrably harmful to have enabled when dropout is off, it would be nice to have this set to true so it cannot somehow become a determinism issue later.

[epwalsh]

We might get a little perf back when it's false.

[chrisc36]

Yea that description is definitely confusing, it makes it sound like True causes it to not save the state, but I assume that was not intentional since that contradicts the note and the parameter name.

[dirkgr]

Can't we just set this to True?

The reason this is there in the first place is that it used to be broken when combined with torch.compile(). So it wasn't a perf thing, it was just that a part was broken that we didn't need anyways, so I wanted to disable it.