stirling support for boringcrypto as used by the 'boringcrypto' version of Golang

[kay-ge]

It would be great if stirling were able to decode TLS flows from the 'boringcrypto' versions of Golang.

Stirling would need to be modified to have uprobes for this version of Golang: https://go.googlesource.com/go/+/dev.boringcrypto

This version of Golang is used in order to have FIPS compliant Go programs. We at VMware use it extensively.

Alternatives:
Don't use the boringcrypto version of Golang. This is not an option in cases where one wants to claim FIPS compliance for their Go program.

[ddelnano]

@kay-ge while working on #692, I realized that our existing Go TLS tracing already covers Go applications using boringcrypto (as long as DWARF debug information is available).

Can you confirm if your boringcrypto applications had missing traffic or was it because our documentation only states Go TLS and OpenSSL support?

I was able to update Pixie's go tls tracing tests to use binaries opted into GOEXPERIMENT=boringcrypto and the tests pass (commit). This is because the functions we hook into (crypto/tls.(*Conn) Write and Read) are triggered for both the standard crypto and boringcrypto case. I'm going to add a boringcrypto acceptance test to the Pixie repo so we are regularly exercising this.

Test output

# Verify that boringcrypto is active

ddelnano@vigenere:~/code/pixie (main) $ bazel build //src/stirling/testing/demo_apps/go_grpc_tls_pl/server:golang_1_19_grpc_tls_server_binary
ddelnano@vigenere:~/code/pixie (main) $ bazel build //src/stirling/testing/demo_apps/go_grpc_tls_pl/server:golang_1_20_grpc_tls_server_binary

ddelnano@vigenere:~/code/pixie (main) $ ~/go/bin/goversion -crypto bazel-bin/src/stirling/testing/demo_apps/go_grpc_tls_pl/server/golang_1_19_grpc_tls_server_binary
bazel-bin/src/stirling/testing/demo_apps/go_grpc_tls_pl/server/golang_1_19_grpc_tls_server_binary go1.19.9 X:boringcrypto (boring crypto)

ddelnano@vigenere:~/code/pixie (main) $ ~/go/bin/goversion -crypto bazel-bin/src/stirling/testing/demo_apps/go_grpc_tls_pl/server/golang_1_20_grpc_tls_server_binary
bazel-bin/src/stirling/testing/demo_apps/go_grpc_tls_pl/server/golang_1_20_grpc_tls_server_binary go1.20.4 X:boringcrypto,nocoverageredesign (boring crypto)

# Verify that the testing is working
ddelnano@vigenere:~/code/pixie (main) $ ./scripts/sudo_bazel_run.sh src/stirling/source_connectors/socket_tracer:go_tls_trace_bpf_test
INFO: Invocation ID: 6d5c214e-68fc-4ff2-bdb2-cee3030a7c35
INFO: Streaming build results to: https://bb.corp.pixielabs.ai/invocation/6d5c214e-68fc-4ff2-bdb2-cee3030a7c35
INFO: Build option --run_under has changed, discarding analysis cache.
INFO: Analyzed target //src/stirling/source_connectors/socket_tracer:go_tls_trace_bpf_test (365 packages loaded, 68206 targets configured).
INFO: Found 1 target...
Target //src/stirling/source_connectors/socket_tracer:go_tls_trace_bpf_test up-to-date:
  bazel-bin/src/stirling/source_connectors/socket_tracer/go_tls_trace_bpf_test
INFO: Elapsed time: 11.184s, Critical Path: 3.90s
INFO: 232 processes: 219 remote cache hit, 3 internal, 10 linux-sandbox.
INFO: Running command line: external/bazel_tools/tools/test/test-setup.sh /bin/bash -c '"$@"' /bin/bash sudo src/stirling/source_connectors/socket_tracer/go_tls_trace_bpf_test
INFO: Streaming build results to: https://bb.corp.pixielabs.ai/invocation/6d5c214e-68fc-4ff2-bdb2-cee3030a7c35
INFO: Build completed successfully, 232 total actions
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //src/stirling/source_connectors/socket_tracer:go_tls_trace_bpf_test
-----------------------------------------------------------------------------
I20230606 21:36:17.353314 3753491 env.cc:47] Started: src/stirling/source_connectors/socket_tracer/go_tls_trace_bpf_test
[==========] Running 8 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from GoTLSTraceTest/0, where TypeParam = px::stirling::Go1_17TLSClientServerContainers
[ RUN      ] GoTLSTraceTest/0.BasicHTTP
I20230606 21:36:17.477846 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_17_https_server
I20230606 21:36:17.579900 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_17_https_client
I20230606 21:36:17.579963 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300236325333127 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_17_https_server
I20230606 21:36:17.833567 3753491 container_runner.cc:144] Container https_server_1300236325333127 status: running
I20230606 21:36:17.866173 3753491 container_runner.cc:175] Container https_server_1300236325333127 process PID: 3753670
I20230606 21:36:17.866202 3753491 container_runner.cc:177] Container https_server_1300236325333127 waiting for log message: Starting HTTPS service
I20230606 21:36:17.898833 3753491 container_runner.cc:189] Container https_server_1300236325333127 status: running
I20230606 21:36:17.898867 3753491 container_runner.cc:225] Container https_server_1300236325333127 is ready.
I20230606 21:36:17.899278 3753491 linux_headers.cc:211] Found Linux kernel version using .note section.
I20230606 21:36:17.899297 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:36:17.899320 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:36:17.899329 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:36:17.899365 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:36:17.899418 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:36:26.323107 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.42 s
I20230606 21:36:27.136358 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:36:27.136401 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:36:27.136457 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:36:27.136494 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:36:27.136504 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:36:27.241335 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:36:27.654106 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:28.308135 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:28.933756 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:29.316942 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:29.712752 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:30.110132 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:30.493902 3753722 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:36:36.479471 3753722 uprobe_manager.cc:908] Number of uprobes deployed = 10283
I20230606 21:36:36.543285 3753762 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:36:36.580655 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300236325333127 --name=https_client_1300255326022761 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_17_https_client --http2=false --iters=2 --sub_iters=5
I20230606 21:36:36.814766 3753491 container_runner.cc:144] Container https_client_1300255326022761 status: running
I20230606 21:36:36.849220 3753491 container_runner.cc:175] Container https_client_1300255326022761 process PID: 3753827
I20230606 21:36:36.849249 3753491 container_runner.cc:177] Container https_client_1300255326022761 waiting for log message: {"status":"ok"}
I20230606 21:36:36.882223 3753491 container_runner.cc:189] Container https_client_1300255326022761 status: running
I20230606 21:36:36.882259 3753491 container_runner.cc:225] Container https_client_1300255326022761 is ready.
I20230606 21:36:38.097079 3753821 uprobe_manager.cc:908] Number of uprobes deployed = 37
I20230606 21:36:38.367131 3753884 uprobe_manager.cc:908] Number of uprobes deployed = 59
I20230606 21:36:43.551957 3753959 uprobe_manager.cc:908] Number of uprobes deployed = 18
I20230606 21:36:55.013135 3753491 container_runner.cc:53] podman rm -f https_client_1300255326022761 &>/dev/null
I20230606 21:36:55.045686 3753491 container_runner.cc:53] podman rm -f https_server_1300236325333127 &>/dev/null
[       OK ] GoTLSTraceTest/0.BasicHTTP (39063 ms)
[ RUN      ] GoTLSTraceTest/0.BasicHTTP2
I20230606 21:36:56.512127 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_17_https_server
I20230606 21:36:56.609123 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_17_https_client
I20230606 21:36:56.609201 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300275354575322 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_17_https_server
I20230606 21:36:56.855082 3753491 container_runner.cc:144] Container https_server_1300275354575322 status: running
I20230606 21:36:56.886946 3753491 container_runner.cc:175] Container https_server_1300275354575322 process PID: 3754584
I20230606 21:36:56.886977 3753491 container_runner.cc:177] Container https_server_1300275354575322 waiting for log message: Starting HTTPS service
I20230606 21:36:56.918601 3753491 container_runner.cc:189] Container https_server_1300275354575322 status: running
I20230606 21:36:56.918632 3753491 container_runner.cc:225] Container https_server_1300275354575322 is ready.
I20230606 21:36:56.918870 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:36:56.918912 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:36:56.918921 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:36:56.918948 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:36:56.919004 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:37:05.421617 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.50 s
I20230606 21:37:06.229264 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:37:06.229305 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:37:06.229362 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:37:06.229398 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:37:06.229406 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:37:06.331228 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:37:06.893393 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:07.358516 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:07.693328 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:08.121908 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:08.491451 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:08.888175 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:09.498785 3754788 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:15.461755 3754788 uprobe_manager.cc:908] Number of uprobes deployed = 10535
I20230606 21:37:15.526285 3754804 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:37:15.563195 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300275354575322 --name=https_client_1300294308565042 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_17_https_client --http2=true --iters=2 --sub_iters=5
I20230606 21:37:15.720250 3753491 container_runner.cc:144] Container https_client_1300294308565042 status: running
I20230606 21:37:15.753818 3753491 container_runner.cc:175] Container https_client_1300294308565042 process PID: 3754861
I20230606 21:37:15.753849 3753491 container_runner.cc:177] Container https_client_1300294308565042 waiting for log message: {"status":"ok"}
I20230606 21:37:15.787252 3753491 container_runner.cc:189] Container https_client_1300294308565042 status: running
I20230606 21:37:15.787283 3753491 container_runner.cc:225] Container https_client_1300294308565042 is ready.
I20230606 21:37:17.074206 3754864 uprobe_manager.cc:908] Number of uprobes deployed = 37
I20230606 21:37:17.330798 3754926 uprobe_manager.cc:908] Number of uprobes deployed = 41
I20230606 21:37:17.615985 3754930 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:37:34.425158 3753491 container_runner.cc:53] podman rm -f https_client_1300294308565042 &>/dev/null
I20230606 21:37:34.457953 3753491 container_runner.cc:53] podman rm -f https_server_1300275354575322 &>/dev/null
[       OK ] GoTLSTraceTest/0.BasicHTTP2 (39468 ms)
[----------] 2 tests from GoTLSTraceTest/0 (78531 ms total)

[----------] 2 tests from GoTLSTraceTest/1, where TypeParam = px::stirling::Go1_18TLSClientServerContainers
[ RUN      ] GoTLSTraceTest/1.BasicHTTP
I20230606 21:37:35.982336 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_18_https_server
I20230606 21:37:36.081152 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_18_https_client
I20230606 21:37:36.081214 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300314826587724 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_18_https_server
I20230606 21:37:36.307430 3753491 container_runner.cc:144] Container https_server_1300314826587724 status: running
I20230606 21:37:36.339200 3753491 container_runner.cc:175] Container https_server_1300314826587724 process PID: 3755337
I20230606 21:37:36.339228 3753491 container_runner.cc:177] Container https_server_1300314826587724 waiting for log message: Starting HTTPS service
I20230606 21:37:36.369964 3753491 container_runner.cc:189] Container https_server_1300314826587724 status: running
I20230606 21:37:36.369990 3753491 container_runner.cc:225] Container https_server_1300314826587724 is ready.
I20230606 21:37:36.370245 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:37:36.370290 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:37:36.370299 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:37:36.370324 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:37:36.370373 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:37:44.779799 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.41 s
I20230606 21:37:45.587615 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:37:45.587656 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:37:45.587711 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:37:45.587749 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:37:45.587757 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:37:45.688004 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:37:46.144762 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:46.672156 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:47.098901 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:47.371347 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:47.732771 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:48.059239 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:48.672499 3755386 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:37:54.677088 3755386 uprobe_manager.cc:908] Number of uprobes deployed = 10742
I20230606 21:37:54.742759 3755398 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:37:54.778105 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300314826587724 --name=https_client_1300333523475675 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_18_https_client --http2=false --iters=2 --sub_iters=5
I20230606 21:37:54.970698 3753491 container_runner.cc:144] Container https_client_1300333523475675 status: running
I20230606 21:37:55.003922 3753491 container_runner.cc:175] Container https_client_1300333523475675 process PID: 3755445
I20230606 21:37:55.003962 3753491 container_runner.cc:177] Container https_client_1300333523475675 waiting for log message: {"status":"ok"}
I20230606 21:37:55.037881 3753491 container_runner.cc:189] Container https_client_1300333523475675 status: running
I20230606 21:37:55.037921 3753491 container_runner.cc:225] Container https_client_1300333523475675 is ready.
W20230606 21:37:56.301640 3755462 uprobe_manager.cc:760] Cannot analyze binary /proc/3755446/root/usr/bin/dash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3755446/root/usr/bin/dash
I20230606 21:37:56.301682 3755462 uprobe_manager.cc:908] Number of uprobes deployed = 64
I20230606 21:37:57.803908 3755523 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:38:02.934703 3755694 uprobe_manager.cc:908] Number of uprobes deployed = 18
I20230606 21:38:13.373070 3753491 container_runner.cc:53] podman rm -f https_client_1300333523475675 &>/dev/null
I20230606 21:38:13.404954 3753491 container_runner.cc:53] podman rm -f https_server_1300314826587724 &>/dev/null
[       OK ] GoTLSTraceTest/1.BasicHTTP (38447 ms)
[ RUN      ] GoTLSTraceTest/1.BasicHTTP2
I20230606 21:38:14.423771 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_18_https_server
I20230606 21:38:14.519914 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_18_https_client
I20230606 21:38:14.519966 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300353265340468 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_18_https_server
I20230606 21:38:14.740466 3753491 container_runner.cc:144] Container https_server_1300353265340468 status: running
I20230606 21:38:14.772504 3753491 container_runner.cc:175] Container https_server_1300353265340468 process PID: 3756360
I20230606 21:38:14.772527 3753491 container_runner.cc:177] Container https_server_1300353265340468 waiting for log message: Starting HTTPS service
I20230606 21:38:14.802927 3753491 container_runner.cc:189] Container https_server_1300353265340468 status: running
I20230606 21:38:14.802945 3753491 container_runner.cc:225] Container https_server_1300353265340468 is ready.
I20230606 21:38:14.803196 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:38:14.803244 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:38:14.803253 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:38:14.803279 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:38:14.803329 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:38:23.275279 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.47 s
I20230606 21:38:24.087327 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:38:24.087368 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:38:24.087424 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:38:24.087460 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:38:24.087469 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:38:24.189576 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:38:24.435456 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:25.073774 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:25.551555 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:25.941008 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:26.375900 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:26.770718 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:27.219740 3756587 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:38:33.561997 3756587 uprobe_manager.cc:908] Number of uprobes deployed = 10958
I20230606 21:38:33.628572 3756600 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:38:33.663100 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300353265340468 --name=https_client_1300372408469798 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_18_https_client --http2=true --iters=2 --sub_iters=5
I20230606 21:38:33.856223 3753491 container_runner.cc:144] Container https_client_1300372408469798 status: running
I20230606 21:38:33.888442 3753491 container_runner.cc:175] Container https_client_1300372408469798 process PID: 3756649
I20230606 21:38:33.888470 3753491 container_runner.cc:177] Container https_client_1300372408469798 waiting for log message: {"status":"ok"}
I20230606 21:38:33.920763 3753491 container_runner.cc:189] Container https_client_1300372408469798 status: running
I20230606 21:38:33.920789 3753491 container_runner.cc:225] Container https_client_1300372408469798 is ready.
W20230606 21:38:35.195266 3756665 uprobe_manager.cc:760] Cannot analyze binary /proc/3756647/root/usr/bin/dash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3756647/root/usr/bin/dash
W20230606 21:38:35.195317 3756665 uprobe_manager.cc:760] Cannot analyze binary /proc/3756648/root/usr/bin/podman for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3756648/root/usr/bin/podman
I20230606 21:38:35.426390 3756665 uprobe_manager.cc:908] Number of uprobes deployed = 96
I20230606 21:38:37.849357 3756732 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:38:52.301106 3753491 container_runner.cc:53] podman rm -f https_client_1300372408469798 &>/dev/null
I20230606 21:38:52.333333 3753491 container_runner.cc:53] podman rm -f https_server_1300353265340468 &>/dev/null
[       OK ] GoTLSTraceTest/1.BasicHTTP2 (39306 ms)
[----------] 2 tests from GoTLSTraceTest/1 (77753 ms total)

[----------] 2 tests from GoTLSTraceTest/2, where TypeParam = px::stirling::Go1_19TLSClientServerContainers
[ RUN      ] GoTLSTraceTest/2.BasicHTTP
I20230606 21:38:53.742610 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_19_https_server
I20230606 21:38:53.860426 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_19_https_client
I20230606 21:38:53.860493 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300392605866847 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_19_https_server
I20230606 21:38:54.112104 3753491 container_runner.cc:144] Container https_server_1300392605866847 status: running
I20230606 21:38:54.143815 3753491 container_runner.cc:175] Container https_server_1300392605866847 process PID: 3757135
I20230606 21:38:54.143842 3753491 container_runner.cc:177] Container https_server_1300392605866847 waiting for log message: Starting HTTPS service
I20230606 21:38:54.174379 3753491 container_runner.cc:189] Container https_server_1300392605866847 status: running
I20230606 21:38:54.174396 3753491 container_runner.cc:225] Container https_server_1300392605866847 is ready.
I20230606 21:38:54.174649 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:38:54.174698 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:38:54.174707 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:38:54.174733 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:38:54.174782 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:39:02.647487 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.47 s
I20230606 21:39:03.459894 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:39:03.459937 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:39:03.459993 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:39:03.460036 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:39:03.460047 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:39:03.562886 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:39:03.827862 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:04.302207 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:04.689600 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:05.074455 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:05.456552 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:05.915161 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:39:06.506151 3757186 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
W20230606 21:39:11.220996 3757186 uprobe_manager.cc:760] Cannot analyze binary /proc/3757182/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3757182/root/usr/bin/sleep
I20230606 21:39:12.720753 3757186 uprobe_manager.cc:908] Number of uprobes deployed = 11174
I20230606 21:39:12.789543 3757316 uprobe_manager.cc:908] Number of uprobes deployed = 18
I20230606 21:39:12.822078 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300392605866847 --name=https_client_1300411567444612 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_19_https_client --http2=false --iters=2 --sub_iters=5
I20230606 21:39:13.015336 3753491 container_runner.cc:144] Container https_client_1300411567444612 status: running
I20230606 21:39:13.047950 3753491 container_runner.cc:175] Container https_client_1300411567444612 process PID: 3757368
I20230606 21:39:13.047983 3753491 container_runner.cc:177] Container https_client_1300411567444612 waiting for log message: {"status":"ok"}
I20230606 21:39:13.079881 3753491 container_runner.cc:189] Container https_client_1300411567444612 status: running
I20230606 21:39:13.079913 3753491 container_runner.cc:225] Container https_client_1300411567444612 is ready.
W20230606 21:39:14.333946 3757385 uprobe_manager.cc:760] Cannot analyze binary /proc/3757366/root/usr/bin/dash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3757366/root/usr/bin/dash
W20230606 21:39:14.333994 3757385 uprobe_manager.cc:760] Cannot analyze binary /proc/3757367/root/usr/bin/podman for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3757367/root/usr/bin/podman
I20230606 21:39:14.565354 3757385 uprobe_manager.cc:908] Number of uprobes deployed = 96
I20230606 21:39:15.922740 3757450 uprobe_manager.cc:908] Number of uprobes deployed = 18
W20230606 21:39:17.236884 3758643 uprobe_manager.cc:760] Cannot analyze binary /proc/3758554/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758554/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.236941 3758643 uprobe_manager.cc:760] Cannot analyze binary /proc/3758555/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758555/root/usr/bin/bash
W20230606 21:39:17.238430 3758643 uprobe_manager.cc:760] Cannot analyze binary /proc/3758579/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758579/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.239379 3758643 uprobe_manager.cc:760] Cannot analyze binary /proc/3758584/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758584/root/usr/bin/bash
I20230606 21:39:17.307154 3758643 uprobe_manager.cc:908] Number of uprobes deployed = 108
W20230606 21:39:17.465656 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758663/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758663/root/usr/bin/bash
W20230606 21:39:17.465718 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758667/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758667/root/usr/bin/sleep
W20230606 21:39:17.465739 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758668/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758668/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.465756 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758669/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758669/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.465775 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758670/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758670/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.465792 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758671/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758671/root/usr/bin/bash
W20230606 21:39:17.465808 3758706 uprobe_manager.cc:760] Cannot analyze binary /proc/3758672/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758672/root/usr/bin/bash
I20230606 21:39:17.465826 3758706 uprobe_manager.cc:908] Number of uprobes deployed = 108
I20230606 21:39:17.600340 3758777 uprobe_manager.cc:908] Number of uprobes deployed = 63
W20230606 21:39:17.890043 3758869 uprobe_manager.cc:760] Cannot analyze binary /proc/3758839/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758839/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.890985 3758869 uprobe_manager.cc:760] Cannot analyze binary /proc/3758843/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758843/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:39:17.891009 3758869 uprobe_manager.cc:760] Cannot analyze binary /proc/3758844/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758844/root/usr/bin/bash
I20230606 21:39:17.891027 3758869 uprobe_manager.cc:908] Number of uprobes deployed = 99
W20230606 21:39:18.003372 3758939 uprobe_manager.cc:760] Cannot analyze binary /proc/3758916/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758916/root/usr/bin/bash
W20230606 21:39:18.017271 3758939 uprobe_manager.cc:760] Cannot analyze binary /proc/3758929/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758929/root/usr/bin/sleep
I20230606 21:39:18.042614 3758939 uprobe_manager.cc:908] Number of uprobes deployed = 135
I20230606 21:39:18.155433 3758961 uprobe_manager.cc:908] Number of uprobes deployed = 45
W20230606 21:39:18.452198 3759000 uprobe_manager.cc:760] Cannot analyze binary /proc/3758989/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/udf/registry_test for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758989/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/udf/registry_test
W20230606 21:39:18.452253 3759000 uprobe_manager.cc:760] Cannot analyze binary /proc/3758990/root/usr/bin/python3.10 for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3758990/root/usr/bin/python3.10
I20230606 21:39:18.452275 3759000 uprobe_manager.cc:908] Number of uprobes deployed = 90
W20230606 21:39:18.743194 3759006 uprobe_manager.cc:760] Cannot analyze binary /proc/3759004/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/udf/registry_test for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3759004/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/udf/registry_test
W20230606 21:39:18.743256 3759006 uprobe_manager.cc:760] Cannot analyze binary /proc/3759005/root/usr/bin/python3.10 for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3759005/root/usr/bin/python3.10
I20230606 21:39:18.743271 3759006 uprobe_manager.cc:908] Number of uprobes deployed = 36
I20230606 21:39:19.188568 3759019 uprobe_manager.cc:908] Number of uprobes deployed = 27
I20230606 21:39:19.367743 3759023 uprobe_manager.cc:908] Number of uprobes deployed = 27
I20230606 21:39:20.391285 3759041 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:39:20.694757 3759042 uprobe_manager.cc:908] Number of uprobes deployed = 72
I20230606 21:39:20.860173 3759055 uprobe_manager.cc:908] Number of uprobes deployed = 54
I20230606 21:39:21.086771 3759060 uprobe_manager.cc:908] Number of uprobes deployed = 54
I20230606 21:39:21.584852 3759071 uprobe_manager.cc:908] Number of uprobes deployed = 54
I20230606 21:39:22.663909 3759085 uprobe_manager.cc:908] Number of uprobes deployed = 54
I20230606 21:39:22.806337 3759090 uprobe_manager.cc:908] Number of uprobes deployed = 27
I20230606 21:39:22.989348 3759095 uprobe_manager.cc:908] Number of uprobes deployed = 27
I20230606 21:39:24.359383 3759128 go_syms.cc:66] Falling back to the runtime.buildVersion symbol for go version detection
W20230606 21:39:24.418660 3759128 uprobe_manager.cc:813] Failed to attach HTTP2 Uprobes to /proc/3759100/root/usr/bin/podman: Internal : Unable to find offset for binary /proc/3759100/root/usr/bin/podman symbol github.com/containers/podman/vendor/google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders address 0
I20230606 21:39:24.572408 3759128 uprobe_manager.cc:908] Number of uprobes deployed = 90
I20230606 21:39:54.677610 3753491 container_runner.cc:53] podman rm -f https_client_1300411567444612 &>/dev/null
I20230606 21:39:54.776916 3753491 container_runner.cc:53] podman rm -f https_server_1300392605866847 &>/dev/null
[       OK ] GoTLSTraceTest/2.BasicHTTP (64692 ms)
[ RUN      ] GoTLSTraceTest/2.BasicHTTP2
I20230606 21:39:58.507894 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_19_https_server
I20230606 21:39:58.662302 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_19_https_client
I20230606 21:39:58.662384 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300457407755886 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_19_https_server
I20230606 21:39:58.974414 3753491 container_runner.cc:144] Container https_server_1300457407755886 status: running
I20230606 21:39:59.012107 3753491 container_runner.cc:175] Container https_server_1300457407755886 process PID: 3770606
I20230606 21:39:59.012161 3753491 container_runner.cc:177] Container https_server_1300457407755886 waiting for log message: Starting HTTPS service
I20230606 21:39:59.059190 3753491 container_runner.cc:189] Container https_server_1300457407755886 status: running
I20230606 21:39:59.059244 3753491 container_runner.cc:225] Container https_server_1300457407755886 is ready.
I20230606 21:39:59.059557 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:39:59.059628 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:39:59.059643 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:39:59.059687 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:39:59.059756 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:40:09.347512 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 10.29 s
I20230606 21:40:10.167308 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:40:10.167349 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:40:10.167404 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:40:10.167441 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:40:10.167452 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:40:10.272289 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:40:10.742357 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:11.255637 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:11.648445 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:12.014964 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:12.364245 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:12.700414 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:13.330469 3774748 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
W20230606 21:40:17.898676 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773208/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773208/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898737 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773214/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773214/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898757 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773215/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773215/root/usr/bin/bash
W20230606 21:40:17.898777 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773227/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/planner/compiler/ast_visitor_test for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773227/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/planner/compiler/ast_visitor_test
W20230606 21:40:17.898794 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773228/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773228/root/usr/bin/bash
W20230606 21:40:17.898813 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773235/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773235/root/usr/bin/sleep
W20230606 21:40:17.898830 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773684/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773684/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898847 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773685/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773685/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898865 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773686/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773686/root/usr/bin/bash
W20230606 21:40:17.898882 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773696/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/planner/compiler/compiler_test for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773696/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/planner/compiler/compiler_test
W20230606 21:40:17.898900 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773697/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773697/root/usr/bin/bash
W20230606 21:40:17.898916 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773701/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773701/root/usr/bin/sleep
W20230606 21:40:17.898932 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773702/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773702/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898948 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773703/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773703/root/home/james/.cache/bazel/_bazel_james/install/edecd2b1233e0abfae42ec33ee10b762/linux-sandbox
W20230606 21:40:17.898967 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773704/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773704/root/usr/bin/bash
W20230606 21:40:17.898985 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773714/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/carnot_test for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773714/root/home/james/.cache/bazel/_bazel_james/378b5f37a0ee36a56febb1595fb99436/execroot/px/bazel-out/k8-fastbuild/bin/src/carnot/carnot_test
W20230606 21:40:17.899003 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773715/root/usr/bin/bash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773715/root/usr/bin/bash
W20230606 21:40:17.899019 3774748 uprobe_manager.cc:760] Cannot analyze binary /proc/3773719/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3773719/root/usr/bin/sleep
I20230606 21:40:19.448841 3774748 uprobe_manager.cc:908] Number of uprobes deployed = 11390
I20230606 21:40:19.517427 3774902 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:40:19.550380 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300457407755886 --name=https_client_1300478295745860 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_19_https_client --http2=true --iters=2 --sub_iters=5
I20230606 21:40:19.781998 3753491 container_runner.cc:144] Container https_client_1300478295745860 status: running
I20230606 21:40:19.817109 3753491 container_runner.cc:175] Container https_client_1300478295745860 process PID: 3774969
I20230606 21:40:19.817147 3753491 container_runner.cc:177] Container https_client_1300478295745860 waiting for log message: {"status":"ok"}
I20230606 21:40:19.850095 3753491 container_runner.cc:189] Container https_client_1300478295745860 status: running
I20230606 21:40:19.850122 3753491 container_runner.cc:225] Container https_client_1300478295745860 is ready.
W20230606 21:40:21.098584 3774966 uprobe_manager.cc:760] Cannot analyze binary /proc/3774946/root/usr/bin/dash for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3774946/root/usr/bin/dash
W20230606 21:40:21.098644 3774966 uprobe_manager.cc:760] Cannot analyze binary /proc/3774947/root/usr/bin/podman for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3774947/root/usr/bin/podman
I20230606 21:40:21.098663 3774966 uprobe_manager.cc:908] Number of uprobes deployed = 55
I20230606 21:40:21.426651 3775028 uprobe_manager.cc:908] Number of uprobes deployed = 59
I20230606 21:40:28.139052 3775175 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:40:31.051141 3775204 go_syms.cc:66] Falling back to the runtime.buildVersion symbol for go version detection
W20230606 21:40:31.107789 3775204 uprobe_manager.cc:813] Failed to attach HTTP2 Uprobes to /proc/3775187/root/usr/bin/podman: Internal : Unable to find offset for binary /proc/3775187/root/usr/bin/podman symbol github.com/containers/podman/vendor/google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders address 0
I20230606 21:40:31.123049 3775204 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:40:39.597115 3753491 container_runner.cc:53] podman rm -f https_client_1300478295745860 &>/dev/null
I20230606 21:40:39.629444 3753491 container_runner.cc:53] podman rm -f https_server_1300457407755886 &>/dev/null
[       OK ] GoTLSTraceTest/2.BasicHTTP2 (42686 ms)
[----------] 2 tests from GoTLSTraceTest/2 (107379 ms total)

[----------] 2 tests from GoTLSTraceTest/3, where TypeParam = px::stirling::Go1_20TLSClientServerContainers
[ RUN      ] GoTLSTraceTest/3.BasicHTTP
I20230606 21:40:41.119590 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_20_https_server
I20230606 21:40:41.228438 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_20_https_client
I20230606 21:40:41.228510 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300499973883862 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_20_https_server
I20230606 21:40:41.439800 3753491 container_runner.cc:144] Container https_server_1300499973883862 status: running
I20230606 21:40:41.471391 3753491 container_runner.cc:175] Container https_server_1300499973883862 process PID: 3775529
I20230606 21:40:41.471422 3753491 container_runner.cc:177] Container https_server_1300499973883862 waiting for log message: Starting HTTPS service
I20230606 21:40:41.501849 3753491 container_runner.cc:189] Container https_server_1300499973883862 status: running
I20230606 21:40:41.501868 3753491 container_runner.cc:225] Container https_server_1300499973883862 is ready.
I20230606 21:40:41.502111 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:40:41.502161 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:40:41.502168 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:40:41.502194 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:40:41.502241 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:40:50.012264 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.51 s
I20230606 21:40:50.821755 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:40:50.821812 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:40:50.821867 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:40:50.821903 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:40:50.821913 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:40:50.924621 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:40:51.243664 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:51.885387 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:52.246434 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:52.649816 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:53.041460 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:53.432793 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:40:53.862509 3775577 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
W20230606 21:40:58.547820 3775577 uprobe_manager.cc:760] Cannot analyze binary /proc/3775575/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3775575/root/usr/bin/sleep
I20230606 21:41:00.044050 3775577 uprobe_manager.cc:908] Number of uprobes deployed = 10814
I20230606 21:41:00.110008 3775677 uprobe_manager.cc:908] Number of uprobes deployed = 18
I20230606 21:41:00.145191 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300499973883862 --name=https_client_1300518890559556 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_20_https_client --http2=false --iters=2 --sub_iters=5
I20230606 21:41:00.301157 3753491 container_runner.cc:144] Container https_client_1300518890559556 status: running
I20230606 21:41:00.334977 3753491 container_runner.cc:175] Container https_client_1300518890559556 process PID: 3775733
I20230606 21:41:00.335003 3753491 container_runner.cc:177] Container https_client_1300518890559556 waiting for log message: {"status":"ok"}
I20230606 21:41:00.366328 3753491 container_runner.cc:189] Container https_client_1300518890559556 status: running
I20230606 21:41:00.366358 3753491 container_runner.cc:225] Container https_client_1300518890559556 is ready.
I20230606 21:41:01.652113 3775743 uprobe_manager.cc:908] Number of uprobes deployed = 37
I20230606 21:41:01.939931 3775795 uprobe_manager.cc:908] Number of uprobes deployed = 59
I20230606 21:41:04.967063 3775855 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:08.314496 3775961 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:08.798010 3775986 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:10.414721 3776034 uprobe_manager.cc:908] Number of uprobes deployed = 45
I20230606 21:41:19.061192 3753491 container_runner.cc:53] podman rm -f https_client_1300518890559556 &>/dev/null
I20230606 21:41:19.093426 3753491 container_runner.cc:53] podman rm -f https_server_1300499973883862 &>/dev/null
[       OK ] GoTLSTraceTest/3.BasicHTTP (39945 ms)
[ RUN      ] GoTLSTraceTest/3.BasicHTTP2
I20230606 21:41:21.061506 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_20_https_server
I20230606 21:41:21.169705 3753491 container_runner.cc:36] Loaded image: localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_20_https_client
I20230606 21:41:21.169773 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --name=https_server_1300539915147734 localhost/bazel/src/stirling/testing/demo_apps/go_https/server:golang_1_20_https_server
I20230606 21:41:21.388166 3753491 container_runner.cc:144] Container https_server_1300539915147734 status: running
I20230606 21:41:21.419574 3753491 container_runner.cc:175] Container https_server_1300539915147734 process PID: 3776393
I20230606 21:41:21.419595 3753491 container_runner.cc:177] Container https_server_1300539915147734 waiting for log message: Starting HTTPS service
I20230606 21:41:21.450573 3753491 container_runner.cc:189] Container https_server_1300539915147734 status: running
I20230606 21:41:21.450593 3753491 container_runner.cc:225] Container https_server_1300539915147734 is ready.
I20230606 21:41:21.450845 3753491 source_connector.cc:35] Initializing source connector: socket_trace_connector
I20230606 21:41:21.450882 3753491 linux_headers.cc:94] Obtained Linux version string from `uname`: 5.19.0-1022-gcp
I20230606 21:41:21.450891 3753491 linux_headers.cc:642] Detected kernel release (uname -r): 5.19.0-1022-gcp
I20230606 21:41:21.450914 3753491 bcc_wrapper.cc:121] Using linux headers found at /lib/modules/5.19.0-1022-gcp/build for BCC runtime.
I20230606 21:41:21.450964 3753491 bcc_wrapper.cc:170] Initializing BPF program ...
I20230606 21:41:30.011158 3753491 scoped_timer.h:48] Timer(init_bpf_program) : 8.56 s
I20230606 21:41:30.823117 3753491 socket_trace_connector.cc:427] Number of kprobes deployed = 40
I20230606 21:41:30.823158 3753491 socket_trace_connector.cc:428] Probes successfully deployed.
I20230606 21:41:30.823212 3753491 socket_trace_connector.cc:362] Initializing perf buffers with ncpus=96 and scaling_factor=0.0865385
I20230606 21:41:30.823247 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kData buffers across all cpus: 268038720
I20230606 21:41:30.823256 3753491 socket_trace_connector.cc:351] Total perf buffer usage for kControl buffers across all cpus: 13353216
I20230606 21:41:30.925472 3753491 socket_trace_connector.cc:432] Number of perf buffers opened = 8
I20230606 21:41:31.350883 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:31.946213 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:32.327951 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:32.693809 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:33.075191 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:33.431862 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
I20230606 21:41:34.063288 3776447 uprobe_symaddrs.cc:849] Getting symbol offsets for version: major=16 minor=17 patch=1
W20230606 21:41:38.499164 3776447 uprobe_manager.cc:760] Cannot analyze binary /proc/3776445/root/usr/bin/sleep for uprobe deployment. If file is under /var/lib, container may have terminated. Message = Can't find or process ELF file /proc/3776445/root/usr/bin/sleep
I20230606 21:41:39.981185 3776447 uprobe_manager.cc:908] Number of uprobes deployed = 10967
I20230606 21:41:40.047283 3776461 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:40.082437 3753491 container_runner.cc:114] podman run --timeout=3600 --rm -q --pid=host --network=container:https_server_1300539915147734 --name=https_client_1300558827805296 localhost/bazel/src/stirling/testing/demo_apps/go_https/client:golang_1_20_https_client --http2=true --iters=2 --sub_iters=5
I20230606 21:41:40.274593 3753491 container_runner.cc:144] Container https_client_1300558827805296 status: running
I20230606 21:41:40.307360 3753491 container_runner.cc:175] Container https_client_1300558827805296 process PID: 3776511
I20230606 21:41:40.307387 3753491 container_runner.cc:177] Container https_client_1300558827805296 waiting for log message: {"status":"ok"}
I20230606 21:41:40.339344 3753491 container_runner.cc:189] Container https_client_1300558827805296 status: running
I20230606 21:41:40.339375 3753491 container_runner.cc:225] Container https_client_1300558827805296 is ready.
I20230606 21:41:41.617419 3776528 uprobe_manager.cc:908] Number of uprobes deployed = 55
I20230606 21:41:41.839206 3776574 uprobe_manager.cc:908] Number of uprobes deployed = 32
I20230606 21:41:48.457949 3776617 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:50.386976 3776654 uprobe_manager.cc:908] Number of uprobes deployed = 9
I20230606 21:41:58.745162 3753491 container_runner.cc:53] podman rm -f https_client_1300558827805296 &>/dev/null
I20230606 21:41:58.777207 3753491 container_runner.cc:53] podman rm -f https_server_1300539915147734 &>/dev/null
[       OK ] GoTLSTraceTest/3.BasicHTTP2 (39229 ms)
[----------] 2 tests from GoTLSTraceTest/3 (79174 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 4 test suites ran. (342838 ms total)
[  PASSED  ] 8 tests.
I20230606 21:42:00.192272 3753491 env.cc:51] Shutting down