False negatives in plugin state restoration tests

[phraemer]

I think there is a timing issue with plugin state restoration tests.
We are getting intermittent failures that didn't make any sense to us so I did some investigation by adding logging and more checks to pluginval but I don't have a deep enough understanding of pluginval's own code to fix this, if indeed it is pluginval's issue. Nevertheless I think it's strange enough behaviour to describe it to you and maybe you will more easily identify the cause.

We’ve been getting this with non-bool or choice parameters too. I.e. with regular old float params so it's not some snapping issue.

I spotted the following pattern by adding lots of logging to pluginval but not too much or the issue would disappear hinting at a timing issue:

• Parameter Fuzz tests run and finish and param Foo is last set to say 0.12345
• Next Plugin state restoration tests run and they go basically like this
  • Get original state information from plugin with callGetStateInformationOnMessageThreadIfVST3(instance)
  • Get the value of Foo via parameter->getValue(), turns out it’s still 0.12345
  • Set value of param Foo to random number parameter->setValue(0.6789);
  • Restore original state via callSetStateInformationOnMessageThreadIfVST3(instance, originalState)
  • Get the value of Foo and test if it’s within 0.1 of 0.12345 but it turns out it’s 0.5 so test fails
  • I added code in the case of failure to get and inspect the current state info and in BOTH the original state and current state the value of Foo is 0.5 which is the default value set by the plugin.

I did a bit of digging in the pluginval and JUCE code and it seems like (correct me if I’m wrong) but the cached parameter values accessed via parameter->getValue()are updated not if you get or set state info but only in the processBlock functions. So with that in mind I added a sleep of 200ms at the start of the test and after that I can’t reproduce the failure, which suggests the processBlock had a chance to run and update the cached parameter values so the old fuzz set value is updated with the actual value in the plugin.