Skip to content

Add explicit dependency to validatePom and generatePom tasks.#12807

Merged
reta merged 3 commits intoopensearch-project:mainfrom
zelinh:add-dependency-pomvalidate
Mar 22, 2024
Merged

Add explicit dependency to validatePom and generatePom tasks.#12807
reta merged 3 commits intoopensearch-project:mainfrom
zelinh:add-dependency-pomvalidate

Conversation

@zelinh
Copy link
Copy Markdown
Member

@zelinh zelinh commented Mar 20, 2024

Description

This PR is to add explicit task dependency on PomGenerate and PomValidate tasks for different publication types.

For plugin with different publication, all generatePom tasks such as generatePomFileForNebulaPublication or generatePomFileForPluginZipPublication are creating the Pom file at the same cache location: build/distributions/*******.pom; and Pom validation tasks are reading the output of this Pom.

We will get error like this in job-scheduler plugin:

> Task :generatePomFileForPluginZipPublication FAILED

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':generatePomFileForPluginZipPublication' (type 'GenerateMavenPom').
  - Gradle detected a problem with the following location: '/Users/zelinhao/workplace/job-scheduler/build/distributions/opensearch-job-scheduler-2.12.0.0-SNAPSHOT.pom'.
    
    Reason: Task ':validateNebulaPom' uses this output of task ':generatePomFileForPluginZipPublication' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
    
    Possible solutions:
      1. Declare task ':generatePomFileForPluginZipPublication' as an input of ':validateNebulaPom'.
      2. Declare an explicit dependency on ':generatePomFileForPluginZipPublication' from ':validateNebulaPom' using Task#dependsOn.
      3. Declare an explicit dependency on ':generatePomFileForPluginZipPublication' from ':validateNebulaPom' using Task#mustRunAfter.
    
    For more information, please refer to https://docs.gradle.org/8.5/userguide/validation_problems.html#implicit_dependency in the Gradle documentation.

Reason
Gradle treats any task that reads a file as implicitly dependent on each task that writes to that file.
In our case, Gradle is assuming each validation task is depending on all the Pom generation tasks as implicit dependency.

In this PR, we are adding these implicit dependencies for generatePomFile and validatePom into its implementation.

We can see same logic is added in few other plugin repos:
In geospatial: validateNebulaPom.dependsOn(generatePomFileForPluginZipPublication)
In skills: tasks.validateNebulaPom.dependsOn ':generatePomFileForPluginZipPublication'

Check List

- [x] New functionality includes testing.
~~ - [x] All tests pass~~
- [x] New functionality has been documented.
~~ - [x] New functionality has javadoc added~~
- [x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)

  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Zelin Hao <zelinhao@amazon.com>
Signed-off-by: Zelin Hao <zelinhao@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 20, 2024

Compatibility status:

Checks if related components are compatible with change 0729327

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

@github-actions
Copy link
Copy Markdown
Contributor

✅ Gradle check result for feea594: SUCCESS

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 21, 2024

Codecov Report

Attention: Patch coverage is 26.66667% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 71.36%. Comparing base (b15cb0c) to head (0729327).
Report is 73 commits behind head on main.

Files Patch % Lines
...gradle/precommit/PomValidationPrecommitPlugin.java 36.36% 7 Missing ⚠️
.../java/org/opensearch/gradle/pluginzip/Publish.java 0.00% 3 Missing ⚠️
...opensearch/gradle/precommit/PomValidationTask.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12807      +/-   ##
============================================
- Coverage     71.42%   71.36%   -0.06%     
- Complexity    59978    60209     +231     
============================================
  Files          4985     5011      +26     
  Lines        282275   283667    +1392     
  Branches      40946    41117     +171     
============================================
+ Hits         201603   202434     +831     
- Misses        63999    64397     +398     
- Partials      16673    16836     +163     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for b6bf6dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❕ Gradle check result for 0729327: UNSTABLE

  • TEST FAILURES:
      3 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@reta
Copy link
Copy Markdown
Contributor

reta commented Mar 22, 2024

Thanks @zelinh , I think it looks good!

@reta reta merged commit c04dad5 into opensearch-project:main Mar 22, 2024
@reta reta added the backport 2.x Backport to 2.x branch label Mar 22, 2024
@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12807-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c04dad58bbc941b8d5f18326d5c5cfb7ac312239
# Push it to GitHub
git push --set-upstream origin backport/backport-12807-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12807-to-2.x.

@reta
Copy link
Copy Markdown
Contributor

reta commented Mar 22, 2024

@zelinh could you please backport to 2.x manually? thank you

@cwperks
Copy link
Copy Markdown
Member

cwperks commented Mar 22, 2024

@reta @zelinh I'm not sure yet, but I think this PR is contributing to failures being seen in security plugin's CI: opensearch-project/security#4156

* Where:
Build file '/home/runner/work/security/security/build.gradle' line: 417

* What went wrong:
A problem occurred evaluating root project 'opensearch-security'.
> NamedDomainObjectContainer#configureEach(Action) on Publication container cannot be executed in the current context.

@reta
Copy link
Copy Markdown
Contributor

reta commented Mar 22, 2024

Ah, that is unexpected, thanks a lot for @cwperks !

reta added a commit to reta/OpenSearch that referenced this pull request Mar 22, 2024
…opensearch-project#12807)"

This reverts commit c04dad5.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
TaskProvider<Task> validatePom = project.getTasks().register("validatePom");
PublishingExtension publishing = project.getExtensions().getByType(PublishingExtension.class);
publishing.getPublications().all(publication -> {
publishing.getPublications().configureEach(publication -> {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line seemed to cause a breaking change in the security plugin:

* Where:
Build file '/home/runner/work/security/security/build.gradle' line: 417

* What went wrong:
A problem occurred evaluating root project 'opensearch-security'.
> NamedDomainObjectContainer#configureEach(Action) on Publication container cannot be executed in the current context.

Is this change from all -> configureEach necessary?

peternied added a commit to peternied/security that referenced this pull request Mar 27, 2024
The underlying issue associated with Pom validation was fixed in OpenSearch [1], this causes the 'hack' that was used to unblock pom generation in security to start causing problems in the distribution builds.

- [1] opensearch-project/OpenSearch#12807

Signed-off-by: Peter Nied <peternied@hotmail.com>
peternied added a commit to peternied/security that referenced this pull request Mar 27, 2024
The underlying issue associated with Pom validation was fixed in OpenSearch [1], this causes the 'hack' that was used to unblock pom generation in security to start causing problems in the distribution builds.

- [1] opensearch-project/OpenSearch#12807

Signed-off-by: Peter Nied <peternied@hotmail.com>
cwperks pushed a commit to opensearch-project/security that referenced this pull request Mar 27, 2024
### Description
The underlying issue associated with Pom validation was fixed in
OpenSearch [1], this causes the 'hack' that was used to unblock pom
generation in security to start causing problems in the distribution
builds.

- [1] opensearch-project/OpenSearch#12807

### Issues Resolved
- Fixes #4160

### Testing
Added new CI test that executes the same job used by the distribution
build and checks the expected artifacts exist

### Check List
- [X] New functionality includes testing
- [ ] ~New functionality has been documented~
- [X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Peter Nied <peternied@hotmail.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…arch-project#12807)

* Add explicit dependency to PomValidation

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

* Update CHANGELOG

Signed-off-by: Zelin Hao <zelinhao@amazon.com>

---------

Signed-off-by: Zelin Hao <zelinhao@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
dlin2028 pushed a commit to dlin2028/security that referenced this pull request May 1, 2024
### Description
The underlying issue associated with Pom validation was fixed in
OpenSearch [1], this causes the 'hack' that was used to unblock pom
generation in security to start causing problems in the distribution
builds.

- [1] opensearch-project/OpenSearch#12807

### Issues Resolved
- Fixes opensearch-project#4160

### Testing
Added new CI test that executes the same job used by the distribution
build and checks the expected artifacts exist

### Check List
- [X] New functionality includes testing
- [ ] ~New functionality has been documented~
- [X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Peter Nied <peternied@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.x Backport to 2.x branch backport-failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants