Creating lists of intermittently failing tests

[adamfarley]

This change lists known unreliable tests in a consistent format.

Resolves #6471

[adamfarley]

Requesting reviews from @ShelleyLambert and @sophia-guo please.

[smlambert]

thanks @adamfarley - a heads up that my active Github tag is @smlambert ... but I did see this review request in any event.

[smlambert]

Thank you for compiling this list @adamfarley. Only openjdk testcase failures are handled by jtreg exclude Problemlist format. For the other test groups that are part of AQAvit, they get excluded via playlist exclusions.

I would have expected the perf and system targets to be listed via playlist files (which do not use a different file for each version as the problem lists do).

I guess it depends on what we intend to do with these lists. If we are adding a feature to exclude unreliable targets on the fly, or post-process failures against a list, we have to consider what format to use, and how to organize them within the directory (as different vendors may eventually have different lists).

[adamfarley]

Thank you for compiling this list @adamfarley. Only openjdk testcase failures are handled by jtreg exclude Problemlist format. For the other test groups that are part of AQAvit, they get excluded via playlist exclusions.

I noticed that. For Perf and Systemtest, I used the jtreg format for consistency, rather than explicit usability (though a consistent format does make any future parsing easier)

I would have expected the perf and system targets to be listed via playlist files (which do not use a different file for each version as the problem lists do).

I considered that, but opted not to because I'm unaware of a mechanism for disabling targets via the playlist files sometimes. As far as I know, for a given vendor, platform, and version, a target is either enabled or disabled. An openjdk test can include an extra exclude file dynamically, so we can choose whether to exclude the test in some runs or not.

I guess it depends on what we intend to do with these lists. If we are adding a feature to exclude unreliable targets on the fly, or post-process failures against a list, we have to consider what format to use, and how to organize them within the directory (as different vendors may eventually have different lists).

My first priority with this task was to provide a unified list of all intermittently failing tests.

• The lists should use a single format to allow parsing and easy-reading.
• The lists should be stored in the repository to allow use at runtime.

Specific uses were a secondary priority. My first thought was that they could be used as exclude files during releases (or during reruns, or reruns during a release), but I didn't want to sidetrack myself looking too much into applications. I figured that if we did pick an application for this data, the best thing I can do right now is to use a consistent format across the board, to make conversion/parsing easier later on.

I'll add a note to the General Retrospective to start discussion into some of our options here.

[smlambert]

openjdk jtreg problemlists are the 'odd man out'... no other test group uses that format

I do not want to store non-openjdk test material in problemlist format as it might lead people to think that is the commonly used format, which it is not.

Agree that we need to have a requirements gathering session and design discussion on this (in terms of formats, and how to structure and where to locate such files). If this is merely a form of documentation, and not a plan for using for debug / temp exclude, then likely all could be in a doc folder.

[adamfarley]

openjdk jtreg problemlists are the 'odd man out'... no other test group uses that format

I do not want to store non-openjdk test material in problemlist format as it might lead people to think that is the commonly used format, which it is not.

Fair. Perhaps I was over-focused on the OpenJDK ProblemList format due to the majority of the intermittent failures being OpenJDK tests.

Agree that we need to have a requirements gathering session and design discussion on this (in terms of formats, and how to structure and where to locate such files). If this is merely a form of documentation, and not a plan for using for debug / temp exclude, then likely all could be in a doc folder.

In my mind, this is a debug/exclude tool first. Mind if I get the ball rolling on Slack?

[smlambert]

Thanks for focussing on the intermittent tests for openjdk @adamfarley ! One last request, can you please put the unreliables dir under the excludes dir (alongside vendors and alpine dirs), thanks! (You'll need to update the link in your README once reorganized).

[adamfarley]

The perf and system tests are a goal for later, so I've split those files out into #6571

The work to exclude these tests at runtime (and the discussion around that) will be in a third PR so we can get this PR merged and ready for reference.

[adamfarley]

Thanks for focussing on the intermittent tests for openjdk @adamfarley ! One last request, can you please put the unreliables dir under the excludes dir (alongside vendors and alpine dirs), thanks! (You'll need to update the link in your README once reorganized).

Sure.

Note: I see this displeases the disableTestsLinter. Looking into this now.

[adamfarley]

@sophia-guo

Heya. I see that the exclude_parser.py specifies that every uncommented line needs to be exactly 3 elements.

In the jtreg FAQ, it allows for the 4+ elements to be comments.

What do you think about me changing that code to be a minimum of 3, rather than exactly 3?

(I've add this change to the PR. Let me know your thoughts.)

[adamfarley]

Our other option, of course, it to strip off the date. I think it's useful information and saves anyone having to access every link to get a sense for the oldest intermittent tests, but perhaps that information doesn't have as much significance as I presume it has.

[sophia-guo]

@adamfarley yes, you can do that. Though personally I think description item is unnecessary as the second element of links( In adoptium we use links as bug information) should already have enough information include description.

[adamfarley]

@adamfarley yes, you can do that. Though personally I think description item is unnecessary as the second element of links( In adoptium we use links as bug information) should already have enough information include description.

Okie dokie. I've spoken with Shelley as well, and her take was that keeping each line to 3 elements is a good way to keep the files clean, so I'll undo the change to the parser and strip out the dates.

[karianna]

@adamfarley will need a small rebase

[adamfarley]

@karianna - The conflict has been resolved. Thanks for pointing it out.

I've also renamed the alpine exclude files and corrected other pieces of code so this doesn't break anything.

[sophia-guo]

I noticed a few problemlist files under unreliable/apline/ are blank. Are those files just place holder? Place holders aren't necessary. See my former comment #6462 (comment)

[karianna]

@adamfarley will need a rebase also

[adamfarley]

I noticed a few problemlist files under unreliable/apline/ are blank. Are those files just place holder? Place holders aren't necessary. See my former comment #6462 (comment)

Good spot. I've removed the placeholder files.

[adamfarley]

Thanks @sophia-guo and @smlambert 😎