Skip building plugins moved to modules

[b-deam]

Since 8.0, we've converted the repository-azure, repository-gcs and repository-s3 plugins into Elasticsearch modules, so that they are always included. Whilst adding or removing these plugins still succeeds, it is now a no-op.

However, if you try and build these plugins with something like:

esrally install --http-port=29200 --node=rally-node --master-nodes=rally-node --car=4gheap,trial-license --seed-hosts="127.0.0.1:29300" --revision "@2022-02-04" --elasticsearch-plugins repository-s3,repository-gcs,repository-azure --plugin-params="s3_client_name:mys3client,s3_access_key:XXXXX,s3_secret_key:YYYYY,s3_session_token:ZZZZZ" --source-build-method=docker

esrally build --revision "@2022-02-04" --elasticsearch-plugins repository-s3,repository-gcs,repository-azure --plugin-params="s3_client_name:mys3client,s3_access_key:XXXXX,s3_secret_key:YYYYY,s3_session_token:ZZZZZ"

It will fail, with this error in Rally's ~/.rally/logs/build.log:

FAILURE: Build failed with an exception.

* What went wrong:
Project 'repository-s3' not found in project ':plugins'.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 6s

For now we still need to retain the existing post-install hooks (see elastic/rally-teams#76), so this is a first step in gracefully skipping any build steps.

Must be merged with:

• Remove plugins that have been converted to modules rally-teams#76

Relates:

• Clean up handling of repository-s3/gcs/azure for 8.0+ #1622

[dliappis]

Works great. Do you think we should add a note in the migrate.rst doc documenting this (and the change in https://github.com/elastic/rally-teams/pull/76/files)?

[dliappis]

I also hit a weird issue when specifying a build revision as a SHA:

If I use a revision with a timestamp like the reproductions you provided in the description it works fine (using either build or install) e.g.

$ esrally build --revision "@2022-11-17" --elasticsearch-plugins repository-azure --plugin-params="azure_client_name:default,azure_account:SOMEAZUREACCOUNT,azure_key:SOMEAUZREKEY" --team-path=../rally-teams

[INFO] Creating installable binary from source files
[INFO] Creating installable binary from source files
{
  "elasticsearch": "/home/dl/.rally/benchmarks/distributions/src/elasticsearch-e68e28e4cb9-linux-x86_64.tar.gz"
}

(Note that I have checked out elastic/rally-teams#76 in a sibling director and referenced with --team-path)

However, when I specify the same revision using a commit SHA (again with build or install), things start to fail:

$ esrally install --http-port=29200 --node=rally-node --master-nodes=rally-node --car=4gheap,trial-license --seed-hosts="127.0.0.1:29300" --revision "e68e28e4cb983cf06be5612b38078bc7d9e02f48" --elasticsearch-plugins repository-azure --plugin-params="azure_client_name:default,azure_account:SOMEAZUREACCOUNT,azure_key:SOMEAUZREKEY" --team-path=../rally-teams
 
[ERROR] Cannot install. No revision specified for plugin [repository-azure].

[b-deam]

I also hit a weird issue when specifying a build revision as a SHA:

If I use a revision with a timestamp like the reproductions you provided in the description it works fine (using either build or install) e.g.

$ esrally build --revision "@2022-11-17" --elasticsearch-plugins repository-azure --plugin-params="azure_client_name:default,azure_account:SOMEAZUREACCOUNT,azure_key:SOMEAUZREKEY" --team-path=../rally-teams

[INFO] Creating installable binary from source files
[INFO] Creating installable binary from source files
{
  "elasticsearch": "/home/dl/.rally/benchmarks/distributions/src/elasticsearch-e68e28e4cb9-linux-x86_64.tar.gz"
}

(Note that I have checked out elastic/rally-teams#76 in a sibling director and referenced with --team-path)

However, when I specify the same revision using a commit SHA (again with build or install), things start to fail:

$ esrally install --http-port=29200 --node=rally-node --master-nodes=rally-node --car=4gheap,trial-license --seed-hosts="127.0.0.1:29300" --revision "e68e28e4cb983cf06be5612b38078bc7d9e02f48" --elasticsearch-plugins repository-azure --plugin-params="azure_client_name:default,azure_account:SOMEAZUREACCOUNT,azure_key:SOMEAUZREKEY" --team-path=../rally-teams
 
[ERROR] Cannot install. No revision specified for plugin [repository-azure].

Ah, I missed that spot. Fixed in 010e08b.

[b-deam]

Works great. Do you think we should add a note in the migrate.rst doc documenting this (and the change in https://github.com/elastic/rally-teams/pull/76/files)?

Good suggestion - addressed in 10ad4fa

[dliappis]

Works great. Left a few small comments, but no need for another review cycle.

[dliappis]

I think this is a pass here, rather than a continue? (I mean there is no loop)

also a nit: shouldn't the line below be converted to an elif?

[b-deam]

I think this is a pass here, rather than a continue? (I mean there is no loop)

I think continue is the correct choice here because it is inside a for loop.

also a nit: shouldn't the line below be converted to an elif?

Addressed in f6d36e0

[dliappis]

I think continue is the correct choice here because it is inside a for loop.

Ah ofc missed that 👍