fix and enable repository-hdfs secure tests#44044
Merged
bizybot merged 13 commits intoelastic:masterfrom Jul 10, 2019
Merged
Conversation
bizybot
added
>test
Collaborator
|
Pinging @elastic/es-distributed |
Due to recent changes done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of fixture to the task. The `secureHdfsFixture` is an AntFixture which is spawned process. Internally it waits for 30 seconds for the resources to be made available. For my local machine it took almost 45 seconds to be available so I have added the wait time as an input to the AntFixture defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. Another problem while running the `secureHdfsFixture` where it would fail due to port not being privileged port (i.e system port, port < 1024). By default datanode address key tries to find a free port and this would later on fail in case it is running in a secure mode. To address this in case of secure mode we find free port below 1024 and set it in the config. The config `DFSConfigKeys.IGNORE_SECURE_PORTS_FOR_TESTING_KEY` is set to `true` but it did not help. https://fisheye.apache.org/browse/~br=branch-2.8.1/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java?hb=true#to140 The integ test for secure hdfs were disabled for long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
force-pushed
the
fix-hdfs-repository-tests
branch
from
79d5f52 to
f4bf100
Compare
Contributor
Author
|
@elasticmachine run elasticsearch-ci/1 |
1 similar comment
Contributor
Author
|
@elasticmachine run elasticsearch-ci/1 |
alpar-t
approved these changes
Jul 8, 2019
Contributor
alpar-t
left a comment
alpar-t
left a comment
There was a problem hiding this comment.
LGTM
We have the issue you describe with all ant fixtures,
but we haven't gotten around to address it yet.
Long term I think we should have all fixtures dockerized and then make sure
we don't leave containers running, like we do for test clusters.
added 7 commits
July 9, 2019 00:19
…nd processes of MiniHDFS not killed properly, reverting unrequired changes
Contributor
Author
|
failed because of existing issue #43889 |
Contributor
Author
alpar-t
approved these changes
Jul 9, 2019
Contributor
alpar-t
left a comment
alpar-t
left a comment
There was a problem hiding this comment.
Thanks for taking this on!
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (elastic#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in elastic#42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
bizybot
added a commit
that referenced
this pull request
Jul 12, 2019
Due to recent changes are done for converting `repository-hdfs` to test clusters (#41252), the `integTestSecure*` tasks did not depend on `secureHdfsFixture` which when running would fail as the fixture would not be available. This commit adds the dependency of the fixture to the task. The `secureHdfsFixture` is a `AntFixture` which is spawned a process. Internally it waits for 30 seconds for the resources to be made available. For my local machine, it took almost 45 seconds to be available so I have added the wait time as an input to the `AntFixture` defaults to 30 seconds and set it to 60 seconds in case of secure hdfs fixture. The integ test for secure hdfs was disabled for a long time and so the changes done in #42090 to fix the tests are also done in this commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Due to recent changes are done for converting
repository-hdfsto testclusters (#41252), the
integTestSecure*tasks did not depend onsecureHdfsFixturewhich when running would fail as the fixturewould not be available. This commit adds the dependency of the fixture
to the task.
The
secureHdfsFixtureis aAntFixturewhich is spawned a process.Internally it waits for 30 seconds for the resources to be made available.
For my local machine, it took almost 45 seconds to be available so I have
added the wait time as an input to the
AntFixturedefaults to 30 secondsand set it to 60 seconds in case of secure hdfs fixture.
The integ test for secure hdfs was disabled for a long time and so
the changes done in #42090 to fix the tests are also done in this commit.
@atorok
As we spawn a process for secure hdfs fixture, in case the task is interrupted/killed the spawned
process would keep running. Though I faced this problem locally when I used ctrl + c, there is a possibility that this happens on CI and then these process will keep running. Is there a standard way I could handle this? Thank you.