[Feature Request] OSB based Performance benchmark with Java Agent

[kumargu]

Is your feature request related to a problem? Please describe

The goal is to track the cost of stack walking with Java agent enabled.

Describe the solution you'd like

The goal would be to run standard OSB test (http logs & nyc taxis) with Java agent enabled and security manager disabled. If we see any regression, we will apply for performance improvements like caching (tbd).

Related component

No response

Describe alternatives you've considered

No response

Additional context

No response

[kumargu]

The first benchmark would be against the SocketInterceptor followed by FileInterceptor.

[kumargu]

@pranu2502 assigning this to you.

[andrross]

FYI, you should be able to do this directly from the pull request that adds the Java agent functionality using the framework described here: https://github.com/opensearch-project/OpenSearch/blob/main/PERFORMANCE_BENCHMARKS.md

[kumargu]

I did try it in my other PR #17517 (comment), but it seems that the OSB with the Java agent now needs socket permission https://build.ci.opensearch.org/job/benchmark-pull-request/2615/console. We'll fix the OSB repo when we get closer to merging, for now; we are running OSB on dedicated EC2 instances.

[pranu2502]

Performance Benchmarks

Environments Used

• For each test, a seperate EC2 instance of type r5.8xlarge with 500GB GP3 EBS volume was used.
• All the benchmarks are performed using Opensearch Benchmarks on HTTP_LOGS workload
• The 3 cases that were tested and reported are
  • jsm: Latest Fork of Opensearch from the main branch
  • java-agent: Latest Pull of the [Security Manager Replacement] which has JSM Disabled, Java Agent with StackWalking enabled
  • java-agent-Optimized: Optimizations to the Java-Agent code
    • All the permissions except SocketPermission in Security.Policy have been removed.
    • All the permissions except SocketPermission in plugins-security.Policy have been removed.
    • A Local Cache has been created to make sure that the visited Protection Domains are not visited again.
    • Pre-Emptive Exit from Stack Walking to exit as soon as the first protection domain with insufficient permission is found

Introduction of Socket Call

A Socket call has been introduced in the Security Plugin within Privileges Evaluator, resulting in the stack walker recieving stack frames with a depth of 130 for a socket call.

Stalk Walking code used for Optimization

SocketChannelInterceptor

final Policy policy = AgentPolicy.getPolicy();
        if (policy == null) {
            return; /* noop */
        }

        java.security.Permission permission = null;

        if (args[0] instanceof InetSocketAddress address) {
            if (!AgentPolicy.isTrustedHost(address.getHostString())) {
                final String host = address.getHostString() + ":" + address.getPort();
                permission = new SocketPermission(host, "connect,resolve");
            }
        } else if (args[0] instanceof UnixDomainSocketAddress) {
            permission = new NetPermission("accessUnixDomainSocket");
        }

        final StackWalker walker = StackWalker.getInstance(Option.RETAIN_CLASS_REFERENCE);


        if (permission != null) {
            final java.security.Permission permissionToCheck = permission;
            walker.walk(new StackCallerChainExtractor(policy, permissionToCheck));// evaluate only once. Actually can be merged later for FileInterceptor.
        }

StackCallerChainExtractor

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

package org.opensearch.javaagent;

import java.lang.StackWalker.StackFrame;
import java.security.ProtectionDomain;
import java.util.List;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.HashSet;
import java.util.Set;
import java.security.Policy;
import java.security.ProtectionDomain;
import java.security.Permission;

/**
 * Stack Caller Chain Extractor
 */
@SuppressWarnings("removal")
public final class StackCallerChainExtractor implements Function<Stream<StackFrame>, List<ProtectionDomain>> {
    private Policy policy;
    private Permission permission;
    /**
     * Constructor
     * @param policy policy
     * @param permission permission
     */
    public StackCallerChainExtractor(Policy policy, Permission permission) {
        this.policy = policy;
        this.permission = permission;

    }

    /**
     * Folds the stack
     * @param frames stack frames
     */
    @Override
    public List<ProtectionDomain> apply(Stream<StackFrame> frames) {
        Set<ProtectionDomain> visitedDomains = new HashSet<>();
         frames.map(StackFrame::getDeclaringClass)
                  .map(Class::getProtectionDomain)
                  .filter(pd -> pd.getCodeSource() != null)
                  .filter(visitedDomains::add)  // Only process domains we haven't seen yet in the current walk
                  .forEach(pd -> {
                      if (!policy.implies(pd, permission)) {
                          throw new SecurityException("Denied access, domain " + pd); // early exit...
                      }
                  });
            return null;
    }
}

Results

Task	Operation	Metric_Variation	jsm-logs (Basline)	java-agent	%diff from Baseline	java-agent-Optimized	%diff from Baseline
index-append	index-append	throughput_min	220138.364	219952.7616	-0.08431172	217915.0217	-1.00997496
index-append	index-append	throughput_mean	237962.6378	236598.9075	-0.573085903	239424.386	0.614276368
index-append	index-append	throughput_median	239642.7141	238150.925	-0.622505505	243121.7265	1.451749684
index-append	index-append	throughput_max	259896.8745	256219.0367	-1.415114288	259830.3694	-0.025589005
index-append	index-append	latency_50_0	128.314324	126.989604	-1.032402275	130.6033605	1.783929049
index-append	index-append	latency_90_0	238.3328975	244.1993996	2.461473914	232.826657	-2.310314924
index-append	index-append	latency_99_0	369.9548941	400.2235777	8.181722708	379.8893373	2.685312009
index-append	index-append	latency_99_9	518.3756169	573.390576	10.61295272	526.6846291	1.602894122
index-append	index-append	latency_99_99	670.6128786	710.751803	5.985409114	674.5339184	0.584694967
index-append	index-append	latency_100_0	1177.778467	1133.905779	-3.725037367	1248.812203	6.031162735
index-append	index-append	latency_mean	144.1755917	146.8731204	1.871002347	144.2799422	0.072377364
status-200s-in-range	200s-in-range	throughput_min	32.87301416	32.85289796	-0.061193663	32.84425626	-0.087481807
status-200s-in-range	200s-in-range	throughput_mean	32.88057872	32.86144974	-0.058177127	32.85287285	-0.08426211
status-200s-in-range	200s-in-range	throughput_median	32.88062506	32.86178459	-0.057299597	32.85293037	-0.084227974
status-200s-in-range	200s-in-range	throughput_max	32.88809693	32.86966667	-0.056039304	32.86143192	-0.08107802
status-200s-in-range	200s-in-range	latency_50_0	6.911123712	7.239671424	4.753897138	6.143590364	-11.10576775
status-200s-in-range	200s-in-range	latency_90_0	7.435561033	7.809523949	5.029383988	6.585301887	-11.43503687
status-200s-in-range	200s-in-range	latency_99_0	13.9696254	8.565088503	-38.68777252	7.026202411	-49.70371639
status-200s-in-range	200s-in-range	latency_100_0	21.00120306	11.58588333	-44.83228746	10.72047015	-48.95306654
status-200s-in-range	200s-in-range	latency_mean	7.127409165	7.284259615	2.20066572	6.183039704	-13.24982807
status-400s-in-range	400s-in-range	throughput_min	49.92848042	49.94394027	0.030963995	49.93326156	0.009575963
status-400s-in-range	400s-in-range	throughput_mean	49.93128162	49.94518225	0.02783951	49.93565294	0.00875467
status-400s-in-range	400s-in-range	throughput_median	49.93128162	49.94518225	0.02783951	49.93565294	0.00875467
status-400s-in-range	400s-in-range	throughput_max	49.93408282	49.94642422	0.024715377	49.93804432	0.00793347
status-400s-in-range	400s-in-range	latency_50_0	4.966901	4.881262	-1.724193804	4.369709	-12.02343272
status-400s-in-range	400s-in-range	latency_90_0	5.459654299	5.4766044	0.310461066	4.938724801	-9.541437431
status-400s-in-range	400s-in-range	latency_99_0	5.88642437	6.03897947	2.591642913	8.86756837	50.64439485
status-400s-in-range	400s-in-range	latency_100_0	5.9717	8.898739999	49.01518829	14.043325	135.1646097
status-400s-in-range	400s-in-range	latency_mean	4.9788779	4.94374882	-0.705562186	4.57933213	-8.02481559

[kumargu]

thanks @pranu2502

JUST to surface it again: the number are collected by introducing a socket call (at 130 depth stack frame -- which is pretty deep).

Next, we will run a nyc-taxi benchmark followed by benchmark with Fileinterceptor.

IMO: i don't see a considerable impact due to the agent, in some cases we do see a better performance (maybe due to removal of the JSM overhead). So, i'd vote for going ahead with the plan of agent introduction.

[andrross]

JUST to surface it again: the number are collected by introducing a socket call (at 130 depth stack frame -- which is pretty deep)

@kumargu Just curious, can you share the snippet of code that you added to do this?

[pranu2502]

Hi @andrross , this was the snippet of code introduced in the Security Plugin at https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java#L336

    try (SocketChannel socketChannel = SocketChannel.open()) {
            socketChannel.connect(new InetSocketAddress("google.com", 443));
            // log.info("Successfully connected to google.com:443, sufficient permissions in Security Interceptor");
        } catch (IOException e) {
            // log.info("Could not connect to google.com:443, sufficient permissions in Security Interceptor");
            e.printStackTrace();
            // throw e;
        }