[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) by reta · Pull Request #16731 · opensearch-project/OpenSearch

reta · 2024-11-27T11:57:07Z

Description

Explore the the native Java Agent (dynamic code rewriting, must be low overhead).

How does it work:

the application (OpenSearch) and agent use common module bootstrap
the application (OpenSearch) is run with the agent
the application (OpenSearch) uses bootstrap module apply security policies

Example:

The sample security.policy (stays the same as before):

grant codeBase "${codebase.opensearch-core}" {
   permission  java.net.SocketPermission "localhost", "connect";
};

The application (OpenSearch) is run with the agent:

-javaagent:agent-3.0.0-SNAPSHOT.jar

The application (OpenSearch) is applies security policy to the agent:

final Policy policy =  new PolicyFile("/security.policy");
AgentPolicy.setPolicy(policy);

Running with 24-ea+31-3600:

[2025-01-22T11:58:11,913][INFO ][o.o.n.Node               ] [host] version[3.0.0-SNAPSHOT], pid[101497], build[tar/7cf6a66e74d8352cf42d60c50b97e46a2aa8866c/2025-01-21T18:11:00.731851515Z], OS[Linux/6.11.0-13-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/24-ea/24-ea+31-3600]                             
[2025-01-22T11:58:11,916][INFO ][o.o.n.Node               ] [host] JVM home [/home/user/jdk-24], using bundled JDK/JRE [false]
[2025-01-22T11:58:11,916][INFO ][o.o.n.Node               ] [host] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:
+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:In
itiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12632241661790883371, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoin
Pool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -javaagent:agent/opensearch-agent-3.0.0-SNAPSHOT.jar, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/home/user/opensearch-3.0.0-jdk24, -Dopensearch.path.conf=/home/user/opensearch-3.0.0-jdk24/config, -Do
pensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]                                                                                                                                                                                                                                                                                       
[2025-01-22T11:58:11,916][WARN ][o.o.n.Node               ] [host] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production                              
[2025-01-22T11:58:11,967][WARN ][o.a.l.i.v.VectorizationProvider] [host] You are running with Java 23 or later. To make full use of the Vector API, please update Apache Lucene.                                                                                                                                                 
[2025-01-22T11:58:12,347][INFO ][o.o.i.r.ReindexModulePlugin] [host] ReindexPlugin reloadSPI called                                                                                                                                                                                                                              
[2025-01-22T11:58:12,348][INFO ][o.o.i.r.ReindexModulePlugin] [host] Unable to find any implementation for RemoteReindexExtension

Related Issues

Closes #16633

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-11-27T12:07:58Z

❌ Gradle check result for 6b73ddf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2024-11-27T13:59:05Z

thanks @reta this is really interesting and such a quick progress.

On a side note, it would be useful to add a small intro snippet how the agent would work overall.

reta · 2024-11-27T14:35:24Z

thanks @reta this is really interesting and such a quick progress.

Thanks @kumargu

On a side note, it would be useful to add a small intro snippet how the agent would work overall.

Absolutely, I have updated the description (but will push it a bit once we get JDK-21 baseline with #16366, it would simplify a lot the APIs usage)

reta · 2024-12-16T19:03:02Z

libs/agent-sm/agent/build.gradle

+      "Can-Retransform-Classes": "true",
+      "Agent-Class": "org.opensearch.javaagent.Agent",
+      "Premain-Class": "org.opensearch.javaagent.Agent",
+      "Boot-Class-Path": 'byte-buddy-1.15.10.jar opensearch-agent-bootstrap-3.0.0-SNAPSHOT.jar'


opensearch-agent-bootstrap is shared between the OpenSearch service and the agent (so the Policy instance could be propagated)

github-actions · 2024-12-16T19:04:16Z

❌ Gradle check result for ea045b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2025-01-16T15:22:09Z

This PR is stalled because it has been open for 30 days with no activity.

github-actions · 2025-01-16T16:04:30Z

❌ Gradle check result for 58a227c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-21T18:49:20Z

❌ Gradle check result for 5e20fde: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-22T17:10:30Z

❌ Gradle check result for 4688fd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-22T21:51:49Z

❌ Gradle check result for 930e6ef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-01-28T03:45:05Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

reta · 2025-01-28T13:45:39Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

@kumargu I think it is feasible in theory but should not be necessary in practice, could you share your thoughts why we may need that?

kumargu · 2025-01-28T15:45:51Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

@kumargu I think it is feasible in theory but should not be necessary in practice, could you share your thoughts why we may need that?

I was thinking we could bring in replacements of JSM in 3.0 while JSM remains enabled in 3.0 (because we'd be still on JDK-21 in 3.0). Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

(note JDK-24 LTS will be available in Sep 2025)

reta · 2025-01-28T16:01:21Z

Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

I think we would only target a most critical APIs by Java Agent (we just cannot much it to SM), however we should be able to run Java Agent on JDK-21 at least.

kumargu · 2025-01-28T16:47:07Z

Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

I think we would only target a most critical APIs by Java Agent (we just cannot much it to SM), however we should be able to run Java Agent on JDK-21 at least.

100% agree. Maybe just the Socket interceptor for now since we see the problems with defining the port ranges in the PR #17107

github-actions · 2025-04-07T17:24:21Z

❌ Gradle check result for c7e3022: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-04-07T19:50:49Z

fyi @reta the policy parser PR #17753 is merged.

reta · 2025-04-07T19:53:05Z

fyi @reta the policy parser PR #17753 is merged.

Sorry @kumargu , was late on that one, we have an issue with it https://github.com/opensearch-project/OpenSearch/pull/17753/files#r2031916820

kumargu · 2025-04-07T19:56:49Z

fyi @reta the policy parser PR #17753 is merged.

Sorry @kumargu , was late on that one, we have an issue with it https://github.com/opensearch-project/OpenSearch/pull/17753/files#r2031916820

ack. I'll raise a fix in a new PR tomorrow.

cwperks · 2025-04-07T20:50:20Z

@kumargu @reta I raised a PR to replace the dependency on :test:framework and to use junit directly: #17821

kumargu · 2025-04-07T20:51:19Z

Oh thanks. I will cancel mine #17820.

github-actions · 2025-04-08T02:27:03Z

❌ Gradle check result for 1ecaa7a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T18:09:18Z

❌ Gradle check result for c836a03: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T19:27:29Z

❌ Gradle check result for 689934e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-04-08T20:07:20Z

❌ Gradle check result for 689934e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Hopefully these are not failing due to the new parser.

Alternately should we try to simplify the FileInterceptor to not use newByteChannel. we can add back the newByteChannel iteratively.

reta · 2025-04-08T22:30:58Z

Hopefully these are not failing due to the new parser.

Noo, see please #17852

Alternately should we try to simplify the FileInterceptor to not use newByteChannel. we can add back the newByteChannel iteratively.

Is there an issue with newByteChannel?

github-actions · 2025-04-08T23:58:22Z

❕ Gradle check result for 8739577: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

andrross · 2025-04-09T00:01:31Z

build.gradle

      if (task != null) {
-        if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
+        if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17 && BuildParams.runtimeJavaVersion <= JavaVersion.VERSION_23) {
          task.jvmArgs += ["-Djava.security.manager=allow"]


Is -Djava.security.manager=allow needed anymore now that the System.setSecurityManager calls have been removed?

No, not anymore :-) I keep this pull request as POC to proof check any other work

@andrross: @cwperks started taking a look at clean-ups of the -Djava.security.manager across all code bases in core. We think its not a lot of changes and we can do the clean-up (in a day or two) once the final PR from this POC is merged.

@andrross: @cwperks started taking a look at clean-ups of the -Djava.security.manager across all code bases in core. We think its not a lot of changes and we can do the clean-up (in a day or two) once the final PR from this POC is merged.

Thanks @kumargu, I should have the clean pull request to core tomorrow, doing some final tests with JDK-24

…ing, must be low overhead) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <drreta@gmail.com>

github-actions · 2025-04-09T01:10:52Z

❌ Gradle check result for ffaf1cd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-09T02:16:43Z

❌ Gradle check result for ffaf1cd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2025-04-09T12:07:54Z

Closing POC in favor of #17861

github-actions bot added the enhancement Enhancement or improvement to existing feature or request label Nov 27, 2024

reta added the skip-changelog label Nov 27, 2024

reta force-pushed the issue-16633 branch from 7088bf8 to 6b73ddf Compare November 27, 2024 11:58

reta mentioned this pull request Nov 27, 2024

[RFC] Replace Java Security Manager (JSM) #1687

Closed

1 task

reta force-pushed the issue-16633 branch 2 times, most recently from 9858717 to ea045b0 Compare December 16, 2024 18:58

reta commented Dec 16, 2024

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jan 16, 2025

reta force-pushed the issue-16633 branch from ea045b0 to 58a227c Compare January 16, 2025 15:59

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Jan 17, 2025

reta force-pushed the issue-16633 branch from 58a227c to 5e20fde Compare January 21, 2025 18:43

reta force-pushed the issue-16633 branch from 5e20fde to 4688fd1 Compare January 22, 2025 16:57

reta force-pushed the issue-16633 branch from 4688fd1 to 930e6ef Compare January 22, 2025 21:42

kumargu mentioned this pull request Jan 26, 2025

Add systemd configurations to strengthen OS core security #17107

Merged

3 tasks

kumargu mentioned this pull request Jan 29, 2025

Permanently turn-off Security Manager (JSM) starting 3.0 #17181

Closed

reta force-pushed the issue-16633 branch from 930e6ef to 82d4eb8 Compare February 4, 2025 18:00

andrross reviewed Apr 9, 2025

View reviewed changes

[Security Manager Replacement] Native Java Agent (dynamic code rewrit…

ffaf1cd

…ing, must be low overhead) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <drreta@gmail.com>

opensearch-ci-bot mentioned this pull request Jul 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ReindexBasicTests #15826

Open

opensearch-ci-bot mentioned this pull request Aug 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for LocalTranslogTests #19168

Closed

opensearch-ci-bot mentioned this pull request Oct 22, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FlightClientManagerTests #19728

Closed

opensearch-ci-bot mentioned this pull request Feb 18, 2026

[AUTOCUT] Gradle Check Flaky Test Report for AzureDiscoveryClusterFormationTests #20658

Open

Conversation

reta commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

kumargu commented Nov 27, 2024

Uh oh!

reta commented Nov 27, 2024

Uh oh!

reta Dec 16, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 16, 2024

Uh oh!

opensearch-trigger-bot bot commented Jan 16, 2025

Uh oh!

github-actions bot commented Jan 16, 2025

Uh oh!

github-actions bot commented Jan 21, 2025

Uh oh!

github-actions bot commented Jan 22, 2025

Uh oh!

github-actions bot commented Jan 22, 2025

Uh oh!

kumargu commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reta commented Jan 28, 2025

Uh oh!

kumargu commented Jan 28, 2025

Uh oh!

reta commented Jan 28, 2025

Uh oh!

kumargu commented Jan 28, 2025

Uh oh!

github-actions bot commented Apr 7, 2025

Uh oh!

kumargu commented Apr 7, 2025

Uh oh!

reta commented Apr 7, 2025

Uh oh!

kumargu commented Apr 7, 2025

Uh oh!

cwperks commented Apr 7, 2025

Uh oh!

kumargu commented Apr 7, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

kumargu commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reta commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

andrross Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

reta Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kumargu Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

reta Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

reta commented Nov 27, 2024 •

edited

Loading

kumargu commented Jan 28, 2025 •

edited

Loading

kumargu commented Apr 8, 2025 •

edited

Loading

reta Apr 9, 2025 •

edited

Loading