Initialization failure with --pid and --listen, from sidecar container

[ctrahey]

Environment data

• debugpy version: 1.8.17
• OS and version:
• • Debian Bookworm container image on top of Amazon Linux 2023.9.20251110
• • dockerized in EC2.
• • uname -a: 6.12.55-74.119.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 4 23:51:53 UTC 2025 x86_64 GNU/Linux
• Python version: Python 3.13.9 via the python:3.13-slim-bookworm dockerhub image (for both the debugpy and the inferior program/pid
• Using VS Code or Visual Studio: VS Code

additional:

• gdb version 13.1

Actual behavior

The setup:

• Target/Inferior python program inside a container in kubernetes. (example_subject.py)
• debugpy in a sidecar which is running privileged and in a shared PID namespace.
• Both images are the exact same python.
• Use an initContainer to copy debugpy (with shared library .so) into the right absolute path in the inferior container.

flowchart LR
    VSCode-->debugpy
    subgraph Laptop
      VSCode
    end
    subgraph Server
     subgraph PID Namespace
      debugpy-->subject
     end
    end

Loading

VS Code config:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python Debugger: Remote Attach",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "my-remote-host.example.com",
                "port": 5678,
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/tests",
                    "remoteRoot": "/app"
                }
            ]
        }
    ]
}

Actions Taken On the remote host (the sidecar)

Running debugpy:

root@remote-debug-test-6686dbdb65-8mhcs:/# time debugpy --listen 0.0.0.0:5678 --pid 15
PYDEVD_GDB_SCAN_SHARED_LIBRARIES not set (scanning all libraries for needed symbols).
Running: gdb --nw --nh --nx --pid 15 --batch --eval-command='set scheduler-locking off' --eval-command='set architecture auto' --eval-command='call (void*)dlopen("/app/.venv/lib/python3.13/site-packages/debugpy/_vendored/pydevd/pydevd_attach_to_process/attach_linux_amd64.so", 2)' --eval-command='sharedlibrary attach_linux_amd64' --eval-command='call (int)DoAttach(0, "import codecs;import json;import sys;decode = lambda s: codecs.utf_8_decode(bytearray(s))[0] if s is not None else None;script_dir = decode([47, 97, 112, 112, <SNIP for brevity>108, 108, 125]));sys.path.insert(0, script_dir);import attach_pid_injected;del sys.path[0];attach_pid_injected.attach(setup);", 0)'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__GI___clock_nanosleep (clock_id=1, flags=1, req=0x7ffeada94ae0, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:71
71	../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
The target architecture is set to "auto" (currently "i386:x86-64").
$1 = (void *) 0x55ea65c07dc0
[Detaching after vfork from child process 53]
[Detaching after vfork from child process 54]
[New Thread 0x7f60fbd7d6c0 (LWP 60)]
[New Thread 0x7f60fb37c6c0 (LWP 61)]
[New Thread 0x7f60fa97b6c0 (LWP 63)]
[New Thread 0x7f60f9f7a6c0 (LWP 64)]
$2 = 0
[Inferior 1 (process 15) detached]

real	0m2.768s
user	0m1.335s
sys	0m0.161s

These log lines are generated during the above:

root@remote-debug-test-6686dbdb65-kjstv:/# tail -f /mnt/debuglogs/sidecar.39.log 
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

And here is what ends up running:

root@remote-debug-test-6686dbdb65-8mhcs:/# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
65535          1  0.0  0.0   1020   688 ?        Ss   07:30   0:00 /pause
root          15  0.3  0.3 371556 61384 ?        Ssl  07:30   0:02 python /app/example_subject.py
root          21  0.2  0.2  47688 40324 ?        Ss   07:30   0:01 /app/.venv/bin/python /app/.venv/bin/attach_debugger
root          30  0.0  0.0   4196  3612 pts/0    Ss   07:30   0:00 bash
root          56  0.0  0.1 478532 19196 ?        Sl   07:31   0:00 /usr/local/bin/python /app/.venv/lib/python3.13/site-packages/debugpy/adapter --for-server 42243 --host 0.0.0.0 --port 5678 --server-access-token <snip>
root         235  0.0  0.0      0     0 ?        Z    07:40   0:00 [echo] <defunct>
root         236  0.0  0.0   8108  4356 pts/0    R+   07:40   0:00 ps -aux

_{A note on attach_debugger: This is currently doing nothing of interest - will eventually be a supervisor using subprocess to run debugpy once I get this working. For now, just manually running debugpy with an exec shell into the sidecar.}

in VSCode

Click the play button to initiate the connection.

These are the only log lines from the python debugger log:

2025-11-21 00:27:26.838 [info] Resolving attach configuration with substituted variables
2025-11-21 00:27:26.839 [info] createDebugAdapterDescriptor: request='attach' name='Python Debugger: Remote Attach'
2025-11-21 00:27:26.839 [info] Connecting to DAP Server at:  debug.u10d.internal:5678

in Wireshark

Packet capture shows that TCP connection is established, and this request is sent:

{
  "command": "initialize",
  "arguments": {
    "clientID": "vscode",
    "clientName": "Visual Studio Code",
    "adapterID": "debugpy",
    "pathFormat": "path",
    "linesStartAt1": true,
    "columnsStartAt1": true,
    "supportsVariableType": true,
    "supportsVariablePaging": true,
    "supportsRunInTerminalRequest": true,
    "locale": "en",
    "supportsProgressReporting": true,
    "supportsInvalidatedEvent": true,
    "supportsMemoryReferences": true,
    "supportsArgsCanBeInterpretedByShell": true,
    "supportsMemoryEvent": true,
    "supportsStartDebuggingRequest": true,
    "supportsANSIStyling": true
  },
  "type": "request",
  "seq": 1
}

I do get a TCP ACK packet back, but in all cases I never get anything other than ACK or SYN ACK from the server.

At this point the inferior process proceeds, but VS Code hangs (little blue indicator moves back and forth under start-debugging button, forever until I click the red square to stop the connection attempt). No packets other than ACK are received

I know something is working across the processes, because at this point if I run the debugpy command a second time, I get this stack trace in the inferior process logs (showing just the end as I assume its not interesting besides knowing it happens):

...
[subject]   File "/app/.venv/lib/python3.13/site-packages/debugpy/server/api.py", line 145, in listen
[subject]     raise RuntimeError("debugpy.listen() has already been called on this process")
[subject] RuntimeError: debugpy.listen() has already been called on this process
[subject] 2025-11-21 08:05:28,172 - INFO - Hello world!
[subject] 2025-11-21 08:05:29,173 - INFO - Hello world!
...

Expected behavior

Expect to establish connection and successfully attach.

Steps to reproduce:

I understand this setup has more moving parts than is typical - but the lack of support for --pid on ARM meant my best place to test my sidecar setup was directly in kubernetes (which is my end game anyway so that's fine). I may find some time this weekend to get a simpler local-docker or local-pid setup going on my X86 machine.

For some added context, here is the important part from the kube manifest I'm deploying this with. It also includes a Service and TCPRoute which I think are immaterial.

    spec:
      shareProcessNamespace: true
      initContainers:
        - name: lib-installer
          image: ctrahey/python-debug-sidecar:latest
          command:
            - "sh"
            - "-c"
            - "cp -r /app/.venv/lib/python3.13/site-packages/debugpy /mnt/attach-linux-shared-lib/debugpy"
          volumeMounts:
            - mountPath: /mnt/attach-linux-shared-lib
              name: attach-linux-shared-lib
      containers:
        - name: subject
          image: ctrahey/periodic-logger-for-testing:latest
          env:
            - name: PYDEVD_DEBUG_FILE
              value: /mnt/debuglogs
          volumeMounts:
            - mountPath: /app/.venv/lib/python3.13/site-packages/debugpy
              subPath: debugpy
              name: attach-linux-shared-lib
            - mountPath: /mnt/debuglogs/inferior.log
              name: logs
        - name: debugger
          image: ctrahey/python-debug-sidecar:latest
          securityContext:
            privileged: true
          env:
            - name: ARG_MATCH
              value: example_subject.py
            - name: PYDEVD_DEBUG_FILE
              value: /mnt/debuglogs/sidecar.log
          volumeMounts:
            - mountPath: /mnt/debuglogs
              name: logs
      volumes:
        - name: attach-linux-shared-lib
          emptyDir: {}
        - name: logs
          emptyDir: {}

and here's the "inferior" (example_subject.py) dockerfile:

FROM python:3.13-slim-bookworm
WORKDIR /app
COPY example_subject.py /app/example_subject.py
CMD ["python", "/app/example_subject.py"]

And the contents of example_subject.py:

import time
import logging
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO)
def main():
    print("Starting up the example subject program.")
    while True:
        logging.info("Hello world!")
        time.sleep(1)

if __name__ == '__main__':
    main()

And this is a little messy at the moment, but here is the dockerfile for the sidecar. Reminder that I'm actually just shelling into this and running debugpy directly, so the actual pythong program is really just the keepalive.

FROM python:3.13-slim-bookworm AS builder
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
RUN echo "Acquire::http::Pipeline-Depth 0;" > /etc/apt/apt.conf.d/99custom && \
    echo "Acquire::http::No-Cache true;" >> /etc/apt/apt.conf.d/99custom && \
    echo "Acquire::BrokenProxy    true;" >> /etc/apt/apt.conf.d/99custom
RUN rm -rf /var/lib/apt/lists/*
RUN apt-get update -o Acquire::CompressionTypes::Order::=gz
RUN apt install -y gcc python3-dev python3-pydevd

RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=src/uv.lock,target=uv.lock \
    --mount=type=bind,source=src/pyproject.toml,target=pyproject.toml \
    uv sync --locked --no-install-project --no-editable

ADD src/ /app/

RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-editable

RUN --mount=type=cache,target=/root/.cache/uv \
    uv build --wheel --all-packages


FROM python:3.13-slim-bookworm
RUN echo "Acquire::http::Pipeline-Depth 0;" > /etc/apt/apt.conf.d/99custom && \
    echo "Acquire::http::No-Cache true;" >> /etc/apt/apt.conf.d/99custom && \
    echo "Acquire::BrokenProxy    true;" >> /etc/apt/apt.conf.d/99custom
RUN rm -rf /var/lib/apt/lists/*
RUN apt-get clean
RUN apt-get update -o Acquire::CompressionTypes::Order::=gz
RUN apt-get install -y gdb gcc python3-dev python3-pydevd

# Copy the environment, but not the source code
COPY --from=builder --chown=app:app /app/.venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"

# Run the application
CMD ["attach_debugger"]

[rchiodo]

It sounds to me like debugpy can't make the connection back to VS code. I think your best bet here would be to instrument the debugpy/pydevd code that happens after the initialize message.

That would be here:

debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_process_net_command_json.py

Line 232 in 82e409e

def on_initialize_request(self, py_db, request):

Additionally this should have happened while the initialize was being handled. It sounds like you don't have this output:

debugpy/src/debugpy/adapter/clients.py

Line 113 in 82e409e

sessions.report_sockets()

For example, here's a log for attach for me when it works:

debugpy.adapter-28700.log

[ctrahey]

Thanks! Is the log you attached taken verbatim from somewhere? VSCode or remote? I set LOG_LEVEL to debug on remote side and no change, just wondering if there is a way for me to crank the log verbosity before I cut a modified build.

[rchiodo]

I just set "logToFile": true in the launch.json for the attach config. It generates the log in the extension folder for the debugger extension. For me that's here:

C:\Users\rchiodo\.vscode\extensions\ms-python.debugpy-2025.16.0-win32-x64

I believe this would be somewhere under ~/.vscode-server or ~/.vscode-remote on a remote machine.