Non-deterministic results - certain groups of nodes always missing

[AnnabellaM]

Hi,

I was running soot on 30 sampled configurations * 112 target programs to detect nondeterministic behaviors. I ran each configuration-program combination 5 times and compared the 5 results to detect inconsistent results. All 112 target programs come from the CATS-Microbenchmark. I observed that certain groups of nodes are always missing together.

Below are some examples of these missing groups of nodes.

These groups of nodes are missing on many programs and configurations. I didn't observe any strong patterns they shared in common.

These call graphs are printed out by adding a SceneTransformer in "wjtp". Here is the interface SootInterface I used to invoke soot and print out the call graphs.

To reproduce

A minimum set of options that can reproduce this problem is as follows:
-pp -w -p cg.spark on-fly-cg:false,enabled:true -p cg.spark enabled:true -p cg.spark pre-jimplify:true

Input file

And a target program that can be used to reproduce this problem is CL1.

Could you offer some insights regarding the inconsistency in results?

Thank you in advance for any feedback!

[AnnabellaM]

Hello Soot team, I'm following up on this issue and would greatly appreciate any insights you could provide. Thank you!

[AnnabellaM]

Hi team, any feedback would be really helpful! Thank you so much in advance!

[StevenArzt]

Sorry for the late reply, we're all quite busy at the moment.

Can you elaborate a bit on the problem? If I understand you correctly, you are using the SPARK callgraph algorithm (albeit with a weird command line - why is SPARK enabled twice?). There are some fairly non-standard options with on-the-fly CG being disabled and an explicit setting for pre-jimplification. I wonder whether you observe the same behavior when enabling SPARK but otherwise sticking to the default options.

Further, I'm not sure I understand the issue. You write that some nodes are "always missing together". Does this mean that these nodes are always missing? Or does it mean that sometimes all nodes of a group are there and sometimes all nodes in the group are missing? That would be the difference between inconsistent behavior and simply an incomplete CG.

[AnnabellaM]

Hi Steven, thank you so much for your time and valuable feedback!

We've conducted testing on 30 sampling configurations, encompassing 28 options, which were generated using the 2-way covering array methodology. Consequently, certain option combinations may appear somewhat non-standard. This non-deterministic behavior has been observed in a total of 20 configurations.

Because the original configurations we generated are quite extensive, each comprising 28 options. The configuration I used in this particular example was derived from an original configuration through bisection. Despite this refinement process, it still produces inconsistent results.

I wonder whether you observe the same behavior when enabling SPARK but otherwise sticking to the default options.

I have tried running the same test case while enabling SPARK, not disabling on-the-fly CG, but keeping all other settings at their default values. And the results I obtained with this configuration are deterministic.

Further, I'm not sure I understand the issue. You write that some nodes are "always missing together". Does this mean that these nodes are always missing? Or does it mean that sometimes all nodes of a group are there and sometimes all nodes in the group are missing?

My apologies for any confusion. The statement means that all nodes of a group sometimes are there and sometimes are missing, within a particular group, all nodes occasionally appear/disappear together.

That would be the difference between inconsistent behavior and simply an incomplete CG.

The analysis that led to these inconsistent results all finished successfully. We have excluded the timed-out runs.

@StevenArzt