Improve xpack integration test UX by kaisecheng · Pull Request #18919 · elastic/logstash

kaisecheng · 2026-03-31T18:51:02Z

Summary

Three improvements to the Gradle xpack integration test workflow:

Make ES and Filebeat extraction incremental. copyEs and copyFilebeat now declare inputs / outputs so Gradle skips re-extraction when the tar.gz is unchanged.
Support running a single xpack spec via -PrubyIntegrationSpecs= instead of the full qa/integration suite.

How to test this PR locally

Run ./gradlew :logstash-xpack:rubyIntegrationTests -PrubyIntegrationSpecs=qa/integration/monitoring/monitoring_is_disabled_spec.rb. Only that spec runs.
Run the second time ./gradlew :logstash-xpack:rubyIntegrationTests -PrubyIntegrationSpecs=qa/integration/monitoring/monitoring_is_disabled_spec.rb. ES is NOT re-downloaded on second run.
Run ./gradlew :logstash-xpack:rubyIntegrationTests. All specs run. ES and Filebeat extracted correctly.

github-actions · 2026-03-31T18:51:16Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

mergify · 2026-03-31T19:34:40Z

This pull request does not have a backport label. Could you fix it @kaisecheng? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

copyEs and copyFilebeat had no inputs/outputs declared and depended on deleteLocalEs/deleteLocalFilebeat, which wiped the extracted directories unconditionally, forcing re-extraction of multi-gigabyte archives on every integration test run. Declare inputs.files and outputs.dir on both tasks so Gradle can skip extraction when the tar.gz is unchanged. Move the delete inside doLast so it only runs when the task actually executes. Add mustRunAfter unpackTarDistribution to satisfy Gradle implicit dependency validation for the shared build/ output directory. Use inputs.files(tasks.named(...)) instead of a closure reading project.ext to avoid configuration-phase property access errors.

…rationSpecs Pass -PrubyIntegrationSpecs=<spec-path> to target a single spec file instead of the entire qa/integration suite: ./gradlew :logstash-xpack:rubyIntegrationTests \ -PrubyIntegrationSpecs=qa/integration/management/multiple_pipelines_spec.rb RSpecIntegrationTests reads org.logstash.xpack.integration.specs system property with qa/integration as default. x-pack/build.gradle forwards the Gradle property to the JVM system property when present. Also adds outputs.upToDateWhen { false } so rubyIntegrationTests always executes when invoked, regardless of Gradle UP-TO-DATE state.

donoghuc · 2026-04-01T18:56:37Z

After each test run, no leftover ES or Logstash processes remain.

I'm having trouble understanding or reproducing this. Using ruby to do process managment seems like something is going really wrong, i'de like to understand what the actual issue is to try to wrap my head around why we would need to do that.

I tried:

 4804  ./gradlew :logstash-xpack:rubyIntegrationTests
 4805  ps aux | grep elasticsearch
 4806  ps aux | grep filebeat

but I dont see any left over processes.

kaisecheng · 2026-04-02T00:12:30Z

@donoghuc Sorry for the confusion, I removed the point in summary but forgot to clean up in test plan.
I thought there was leftover but actually, Elasticsearch process was just a bit slow to fully shut down.

andsel

I have a couple of doubts:

what happens when there is already an unpacked ES distribution (in build/elasticsearch) an the a new version (different remote SHA) has been published in the artifacts API? I think it unpacks on the existing, with the risk to keep stale files. I think in that case, when a new version is discovered, downloaded and unpacked it has to cleanup the target folder.
the tasks copyFilebeat depends on downloadFilebeat which depends on prepareFilebeatDownload. prepareFilebeatDownload prepares the URL for download in the project extension properties but doesn't check any SHA difference between local and remote.
The downloadEs task isntead, depends on both prepareEsDownload and checkEsSHA. The checkEsSHA is responsible to clean the local extracted copy of ES. I think that also the downloadFilebeat should to the same thing, else we risk that once downloaded a filebeat, we don't update with sequent versions (for the same version), if I'm not wrong.

Add checkFilebeatSHA task mirroring checkEsSHA. Without it, daily snapshot rebuilds of the same version go undetected because Gradle skips downloadFilebeat when versions.yml is unchanged.

…nd checkFilebeatSHA Both tasks share identical logic: fetch remote SHA, compare with local archive, delete local if different. Extract into a single method.

kaisecheng · 2026-04-02T10:20:29Z

This is not a concern. The full chain when a new remote version is published: checkEsSHA detects the SHA mismatch > deletes the local tar.gz > downloadEs sees its output is missing and re-downloads > copyEs sees its declared input changed and re-runs > delete('./build/elasticsearch') + fresh extraction. So, it won't unpack existing and won't keep the stale files.
Before this PR, downloadFilebeat had no SHA check. As no new daily snapshot is downloaded, copyFilebeat is not triggered. This is existing behaviour. But I am happy to add SHA verification for filebeat task

Use artifactProject as the log label instead.

elasticmachine · 2026-04-02T10:54:40Z

💚 Build Succeeded

Buildkite Build
Commit: b6fcf7e

History

💛 Build #4564 was flaky 33c227e
💚 Build #4561 succeeded e9b4c24
💔 Build #4560 failed bcdf73e
💚 Build #4556 succeeded 33c227e
💚 Build #4553 succeeded a957d1e
💛 Build #4545 was flaky 02b6c21

andsel

LGTM

Nice fix! And thanks to have improved also the filebeat case.

github-actions · 2026-04-02T13:52:17Z

@Mergifyio backport 8.19 9.2 9.3

mergify · 2026-04-02T13:52:41Z

backport 8.19 9.2 9.3

✅ Backports have been created

Details

#18932 [8.19] (backport #18919) Improve xpack integration test UX has been created for branch 8.19 but encountered conflicts

Cherry-pick of fcfcea0 has failed:

On branch mergify/bp/8.19/pr-18919
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit fcfcea0c3.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/build.gradle
	modified:   x-pack/src/test/java/org/logstash/xpack/test/RSpecIntegrationTests.java

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   build.gradle
	deleted by us:   x-pack/AGENTS.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

#18933 [9.2] (backport #18919) Improve xpack integration test UX has been created for branch 9.2 but encountered conflicts

Cherry-pick of fcfcea0 has failed:

On branch mergify/bp/9.2/pr-18919
Your branch is up to date with 'origin/9.2'.

You are currently cherry-picking commit fcfcea0c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   build.gradle
	modified:   x-pack/build.gradle
	modified:   x-pack/src/test/java/org/logstash/xpack/test/RSpecIntegrationTests.java

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   x-pack/AGENTS.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

#18934 [9.3] (backport #18919) Improve xpack integration test UX has been created for branch 9.3 but encountered conflicts

Cherry-pick of fcfcea0 has failed:

On branch mergify/bp/9.3/pr-18919
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit fcfcea0c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   build.gradle
	modified:   x-pack/build.gradle
	modified:   x-pack/src/test/java/org/logstash/xpack/test/RSpecIntegrationTests.java

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   x-pack/AGENTS.md

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

- Make ES and Filebeat extraction incremental. copyEs and copyFilebeat now declare inputs / outputs so Gradle skips re-extraction when the tar.gz is unchanged. - Support running a single xpack spec via `-PrubyIntegrationSpecs` to align with core. Example: `./gradlew :logstash-xpack:rubyIntegrationTests -PrubyIntegrationSpecs=qa/integration/monitoring/monitoring_is_disabled_spec.rb` (cherry picked from commit fcfcea0) # Conflicts: # build.gradle # x-pack/AGENTS.md

- Make ES and Filebeat extraction incremental. copyEs and copyFilebeat now declare inputs / outputs so Gradle skips re-extraction when the tar.gz is unchanged. - Support running a single xpack spec via `-PrubyIntegrationSpecs` to align with core. Example: `./gradlew :logstash-xpack:rubyIntegrationTests -PrubyIntegrationSpecs=qa/integration/monitoring/monitoring_is_disabled_spec.rb` (cherry picked from commit fcfcea0) # Conflicts: # x-pack/AGENTS.md

kaisecheng force-pushed the fix-gradle-copyEs branch from fede1e9 to 1db27b2 Compare April 1, 2026 00:05

kaisecheng changed the title ~~Avoid redundant download when running integration test~~ Improve Gradle xpack test UX Apr 1, 2026

kaisecheng force-pushed the fix-gradle-copyEs branch from 1db27b2 to 3dae1c4 Compare April 1, 2026 00:30

kaisecheng added 3 commits April 1, 2026 11:52

docs: document single xpack integration spec command in x-pack/AGENTS.md

33c227e

kaisecheng force-pushed the fix-gradle-copyEs branch from a957d1e to 33c227e Compare April 1, 2026 10:53

kaisecheng changed the title ~~Improve Gradle xpack test UX~~ Improve xpack integration test UX Apr 1, 2026

kaisecheng marked this pull request as ready for review April 1, 2026 15:18

kaisecheng added the backport-active-all Automated backport with mergify to all the active branches label Apr 1, 2026

kaisecheng requested a review from andsel April 1, 2026 15:36

kaisecheng force-pushed the fix-gradle-copyEs branch from e9b4c24 to 33c227e Compare April 1, 2026 16:44

andsel requested changes Apr 2, 2026

View reviewed changes

kaisecheng added 2 commits April 2, 2026 10:50

feat: add SHA verification for Filebeat download

cb24547

Add checkFilebeatSHA task mirroring checkEsSHA. Without it, daily snapshot rebuilds of the same version go undetected because Gradle skips downloadFilebeat when versions.yml is unchanged.

refactor: extract verifyPackageSHA helper to deduplicate checkEsSHA a…

4b10ed8

…nd checkFilebeatSHA Both tasks share identical logic: fetch remote SHA, compare with local archive, delete local if different. Extract into a single method.

refactor: remove redundant label param from verifyPackageSHA

b6fcf7e

Use artifactProject as the log label instead.

kaisecheng requested a review from andsel April 2, 2026 11:34

andsel approved these changes Apr 2, 2026

View reviewed changes

kaisecheng merged commit fcfcea0 into main Apr 2, 2026
11 checks passed

kaisecheng deleted the fix-gradle-copyEs branch April 2, 2026 13:51

mergify bot mentioned this pull request Apr 2, 2026

[8.19] (backport #18919) Improve xpack integration test UX #18932

Open

3 tasks

mergify bot mentioned this pull request Apr 2, 2026

[9.2] (backport #18919) Improve xpack integration test UX #18933

Open

3 tasks

mergify bot mentioned this pull request Apr 2, 2026

[9.3] (backport #18919) Improve xpack integration test UX #18934

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve xpack integration test UX#18919

Improve xpack integration test UX#18919
kaisecheng merged 6 commits intomainfrom
fix-gradle-copyEs

kaisecheng commented Mar 31, 2026 •

edited by andsel

Loading

Uh oh!

github-actions bot commented Mar 31, 2026

Uh oh!

mergify bot commented Mar 31, 2026

Uh oh!

donoghuc commented Apr 1, 2026

Uh oh!

kaisecheng commented Apr 2, 2026

Uh oh!

andsel left a comment

Uh oh!

kaisecheng commented Apr 2, 2026

Uh oh!

elasticmachine commented Apr 2, 2026

Uh oh!

andsel left a comment

Uh oh!

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

mergify bot commented Apr 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kaisecheng commented Mar 31, 2026 • edited by andsel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test this PR locally

Uh oh!

github-actions bot commented Mar 31, 2026

🤖 GitHub comments

Uh oh!

mergify bot commented Mar 31, 2026

Uh oh!

donoghuc commented Apr 1, 2026

Uh oh!

kaisecheng commented Apr 2, 2026

Uh oh!

andsel left a comment

Choose a reason for hiding this comment

Uh oh!

kaisecheng commented Apr 2, 2026

Uh oh!

elasticmachine commented Apr 2, 2026

💚 Build Succeeded

History

Uh oh!

andsel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

mergify bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaisecheng commented Mar 31, 2026 •

edited by andsel

Loading

mergify bot commented Apr 2, 2026 •

edited

Loading