[Discuss] Do not fail PRs on description non-compliance

[nknize]

From #10949 (comment)

Mechanisms like this are making PRs more burdensome. I suggest leaning out the GHA gate keeping to make it easier for PRs to merge, not more difficult. Processes like this are an oss turn off and increase contributor attrition.

[reta]

Processes like this are an oss turn off and increase contributor attrition.

This is not an issue but a response to increase awareness that the project has issues. The large amount of flaky tests is the real cause of the statement above, not this check :(

Once the situation improves, this check will be gone.

[msfroh]

I'm not a fan of the checklist verifier action.

We do need to fix flaky tests, but I don't see how the checklist verifier helps us do that. IMO, it's just unnecessary friction and noise.

[peternied]

I think we can agree there needs to be a level of quality of the project, and there are different ways to achieve it.

To me there is balance between clear expectations of pull requests and how long it takes to get feedback. Different maintainers will have different expectations and GHA are a way to consolidate and formalize expectations.

I see how the ./gradlew check triggered by GHA is used to great success to keep the project quality high.

increase[d] contributor attrition

For specifically requiring checking or striking out the items in the pull request description, I would suggest that items impacting contributor attrition should be remove them from the required items list & from the template.

List as of today:

OpenSearch/.github/workflows/pull-request-checks.yml

Lines 22 to 28 in be65f54

	checklist-items: |
	New functionality includes testing.
	All tests pass
	New functionality has been documented.
	New functionality has javadoc added
	Commits are signed per the DCO using --signoff
	Commit changes are listed out in CHANGELOG.md file (See: [Changelog](../blob/main/CONTRIBUTING.md#changelog))

[peternied]

Once the situation improves, this check will be gone.

unnecessary friction

I think its valuable to ask contributors and maintainers alike if the pull request follows certain standards - setting and validating exceptions always has a cost in friction.

@reta @msfroh I don't think this manifestation as a checklist verification on the PR template is the way - I'd be curious if we have other thoughts on how to accomplish standardization between many different people with different backgrounds.

Alternatively, I'm open to suggestions that standardized quality isn't a goal of the project, but I'd be curious how we continue to scale.

[reta]

@reta @msfroh I don't think this manifestation as a checklist verification on the PR template is the way - I'd be curious if we have other thoughts on how to accomplish standardization between many different people with different backgrounds.

To me, it is all about what we are trying to solve with each check (shared here #10970 (comment)). I see it as a temporary measure but not a solution to the problem (sadly).

[nknize]

We do need to fix flaky tests...

Agree. Flakes are a side effect of other poor habits like using draft PRs to test a contribution instead of running them locally before opening a PR. IMHO the checklist was fine as is, a gentle passive reminder of what the maintainers are looking for in a contribution. We shouldn't have a GHA gate on it that fails someone's PR. It comes off a bit punitive.

I see how the ./gradlew check triggered by GHA is used to great success to keep the project quality high.

That trigger was meant as a final check in the chance the developer didn't run it locally to catch qa or packaging errors. It's an expensive and noisy check. For me, I'm unhappy if gradle check doesn't pass on the first or second run because I don't ever open a PR w/o first running it myself locally. Unnecessarily spamming a PR with checks is sloppy work IMO.

[nknize]

Almost a week old.. marked as stalled.