read init-p: connection reset by peer

[wusikijeronii]

Nomad version

Nomad v1.7.6
BuildDate 2024-03-12T07:27:36Z
Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e

Operating system and Environment details

NAME="Oracle Linux Server"
VERSION="8.9"

Issue

After updating Nomad from 1.5.3 to 1.7.6, I can't run any job on two of the three nodes (the same job). I get the error:

failed to launch command with executor: rpc error: code = Unknown desc = unable to start container process: error during container init: read init-p: connection reset by peer

I also tried to create a simple job that will run /bin/bash, but I still face the issue. I also tried to reboot servers and update all packages on host machines, but that didn't help.
I also tried to remove all cache data from all servers. I thought it was a file access issue at first. If you use the default user (anonymous), the same error occurs.

Job file

job "test-job" {
  datacenters = ["dc1"]
  type = "service"
  group "test-group" {
    count = 3
    constraint {
      operator  = "distinct_hosts"
      value     = "true"
    }
    
    restart {
      attempts = 2
      interval = "5m"
      delay = "15s"
      mode = "fail"
    }

    ephemeral_disk {
      size = 300
    }

    task "test-task" {
      user = "cored"
      driver = "exec"
      
      config {
        command = "/bin/bash"
        args = []
      }

      resources {
        cpu = 200
        }
      }
    }
}

Nomad logs

Details

-- Logs begin at Mon 2024-03-25 10:46:06 MSK, end at Mon 2024-03-25 11:14:14 MSK. --
Mar 25 10:58:00 srv2.prod nomad[5744]:     2024-03-25T10:58:00.101+0300 [DEBUG] client: updated allocations: index=443167 total=15 pulled=13 filtered=2
Mar 25 10:58:00 srv2.prod nomad[5744]:     2024-03-25T10:58:00.101+0300 [DEBUG] client: allocation updates: added=0 removed=0 updated=13 ignored=2
Mar 25 10:58:00 srv2.prod nomad[5744]:     2024-03-25T10:58:00.124+0300 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=13 ignored=2 errors=0
Mar 25 10:58:01 srv2.prod nomad[5744]:     2024-03-25T10:58:01.265+0300 [DEBUG] nomad: memberlist: Initiating push/pull sync with: srv1-prod.global 10.0.1.4:4648
Mar 25 10:58:01 srv2.prod nomad[5744]:     2024-03-25T10:58:01.791+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="150.062µs"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.417+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task path=/usr/bin/nomad args=["/usr/bin/nomad", "logmon"]
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.418+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task path=/usr/bin/nomad pid=37643
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.418+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task plugin=/usr/bin/nomad
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.451+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task network=unix @module=logmon address=/tmp/plugin4056566260 timestamp="2024-03-25T10:58:02.451+0300"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.452+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task version=2
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.453+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task @module=logmon path=/opt/nomad/alloc/c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2/alloc/logs/.test-task.stdout.fifo timestamp="2024-03-25T10:58:02.453+0300"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.453+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task @module=logmon path=/opt/nomad/alloc/c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2/alloc/logs/.test-task.stderr.fifo timestamp="2024-03-25T10:58:02.453+0300"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.468+0300 [INFO]  client.driver_mgr.exec: starting task: driver=exec driver_cfg="{Command:/bin/bash Args:[] ModePID: ModeIPC: CapAdd:[] CapDrop:[]}"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.468+0300 [DEBUG] client.driver_mgr.exec.executor: starting plugin: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task path=/usr/bin/nomad args=["/usr/bin/nomad", "executor", "{\"LogFile\":\"/opt/nomad/alloc/c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2/test-task/executor.out\",\"LogLevel\":\"debug\",\"FSIsolation\":true,\"Compute\":{\"tc\":8000,\"nc\":4}}"]
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.469+0300 [DEBUG] client.driver_mgr.exec.executor: plugin started: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task path=/usr/bin/nomad pid=37654
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.469+0300 [DEBUG] client.driver_mgr.exec.executor: waiting for RPC address: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task plugin=/usr/bin/nomad
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.505+0300 [DEBUG] client.driver_mgr.exec.executor: using plugin: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task version=2
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.507+0300 [DEBUG] client.driver_mgr.exec: task capabilities: driver=exec capabilities=["CAP_AUDIT_WRITE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_MKNOD", "CAP_NET_BIND_SERVICE", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"]
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.521+0300 [DEBUG] client.driver_mgr.exec.executor.nomad: time="2024-03-25T10:58:02+03:00" level=warning msg="cannot serialize hook of type configs.FuncHook, skipping": alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.655+0300 [DEBUG] client.driver_mgr.exec.executor.stdio: received EOF, stopping recv loop: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task err="rpc error: code = Unavailable desc = error reading from server: EOF"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.666+0300 [INFO]  client.driver_mgr.exec.executor: plugin process exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task plugin=/usr/bin/nomad id=37654
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.666+0300 [DEBUG] client.driver_mgr.exec.executor: plugin exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.666+0300 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task type="Driver Failure" msg="failed to launch command with executor: rpc error: code = Unknown desc = unable to start container process: error during container init: read init-p: connection reset by peer" failed=false
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.668+0300 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task error="failed to launch command with executor: rpc error: code = Unknown desc = unable to start container process: error during container init: read init-p: connection reset by peer"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.668+0300 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task reason="Error was unrecoverable"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.668+0300 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task type="Not Restarting" msg="Error was unrecoverable" failed=true
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.674+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task err="rpc error: code = Unavailable desc = error reading from server: EOF"
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.680+0300 [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task plugin=/usr/bin/nomad id=37643
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.680+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.680+0300 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.680+0300 [INFO]  client.gc: marking allocation for GC: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2
Mar 25 10:58:02 srv2.prod nomad[5744]:     2024-03-25T10:58:02.886+0300 [DEBUG] nomad.client: adding evaluations for rescheduling failed allocations: num_evals=1
Mar 25 10:58:03 srv2.prod nomad[5744]:     2024-03-25T10:58:03.078+0300 [DEBUG] client: updated allocations: index=443169 total=15 pulled=13 filtered=2
Mar 25 10:58:03 srv2.prod nomad[5744]:     2024-03-25T10:58:03.078+0300 [DEBUG] client: allocation updates: added=0 removed=0 updated=13 ignored=2
Mar 25 10:58:03 srv2.prod nomad[5744]:     2024-03-25T10:58:03.105+0300 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=13 ignored=2 errors=0
Mar 25 10:58:04 srv2.prod nomad[5744]:     2024-03-25T10:58:04.212+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:32790
Mar 25 10:58:04 srv2.prod nomad[5744]:     2024-03-25T10:58:04.344+0300 [DEBUG] client: updated allocations: index=443171 total=15 pulled=14 filtered=1
Mar 25 10:58:04 srv2.prod nomad[5744]:     2024-03-25T10:58:04.344+0300 [DEBUG] client: allocation updates: added=0 removed=0 updated=14 ignored=1
Mar 25 10:58:04 srv2.prod nomad[5744]:     2024-03-25T10:58:04.375+0300 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=14 ignored=1 errors=0
Mar 25 10:58:05 srv2.prod nomad[5744]:     2024-03-25T10:58:05.618+0300 [DEBUG] nomad.client: adding evaluations for rescheduling failed allocations: num_evals=1
Mar 25 10:58:05 srv2.prod nomad[5744]:     2024-03-25T10:58:05.815+0300 [DEBUG] client: updated allocations: index=443175 total=15 pulled=14 filtered=1
Mar 25 10:58:05 srv2.prod nomad[5744]:     2024-03-25T10:58:05.815+0300 [DEBUG] client: allocation updates: added=0 removed=0 updated=14 ignored=1
Mar 25 10:58:05 srv2.prod nomad[5744]:     2024-03-25T10:58:05.839+0300 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=14 ignored=1 errors=0
Mar 25 10:58:08 srv2.prod nomad[5744]:     2024-03-25T10:58:08.459+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="684.523µs"
Mar 25 10:58:11 srv2.prod nomad[5744]:     2024-03-25T10:58:11.792+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="732.682µs"
Mar 25 10:58:14 srv2.prod nomad[5744]:     2024-03-25T10:58:14.213+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:34694
Mar 25 10:58:18 srv2.prod nomad[5744]:     2024-03-25T10:58:18.461+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration=1.276092ms
Mar 25 10:58:18 srv2.prod nomad[5744]:     2024-03-25T10:58:18.581+0300 [DEBUG] nomad.deployments_watcher: deadline hit: deployment_id=d8885905-d4de-46c0-d7b5-7ac35e164f14 job="<ns: \"default\", id: \"cored\">" rollback=false
Mar 25 10:58:18 srv2.prod nomad[5744]:     2024-03-25T10:58:18.616+0300 [DEBUG] http: request complete: method=GET path="/v1/deployment/d8885905-d4de-46c0-d7b5-7ac35e164f14?index=443082&stale=" duration=3m30.444134347s
Mar 25 10:58:21 srv2.prod nomad[5744]:     2024-03-25T10:58:21.793+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="108.49µs"
Mar 25 10:58:24 srv2.prod nomad[5744]:     2024-03-25T10:58:24.214+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:45212
Mar 25 10:58:28 srv2.prod nomad[5744]:     2024-03-25T10:58:28.462+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="147.718µs"
Mar 25 10:58:31 srv2.prod nomad[5744]:     2024-03-25T10:58:31.795+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="169.04µs"
Mar 25 10:58:34 srv2.prod nomad[5744]:     2024-03-25T10:58:34.215+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:33958
Mar 25 10:58:38 srv2.prod nomad[5744]:     2024-03-25T10:58:38.464+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="102.038µs"
Mar 25 10:58:41 srv2.prod nomad[5744]:     2024-03-25T10:58:41.797+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="202.884µs"
Mar 25 10:58:44 srv2.prod nomad[5744]:     2024-03-25T10:58:44.216+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:41588
Mar 25 10:58:48 srv2.prod nomad[5744]:     2024-03-25T10:58:48.465+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="153.443µs"
Mar 25 10:58:51 srv2.prod nomad[5744]:     2024-03-25T10:58:51.798+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="146.317µs"
Mar 25 10:58:53 srv2.prod nomad[5744]:     2024-03-25T10:58:53.286+0300 [DEBUG] nomad: memberlist: Stream connection from=10.0.1.8:35898
Mar 25 10:58:54 srv2.prod nomad[5744]:     2024-03-25T10:58:54.217+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:44692
Mar 25 10:58:58 srv2.prod nomad[5744]:     2024-03-25T10:58:58.467+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="130.781µs"
Mar 25 10:59:01 srv2.prod nomad[5744]:     2024-03-25T10:59:01.268+0300 [DEBUG] nomad: memberlist: Initiating push/pull sync with: srv1-prod.global 10.0.1.4:4648
Mar 25 10:59:01 srv2.prod nomad[5744]:     2024-03-25T10:59:01.800+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="170.41µs"
Mar 25 10:59:04 srv2.prod nomad[5744]:     2024-03-25T10:59:04.219+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:60018
Mar 25 10:59:08 srv2.prod nomad[5744]:     2024-03-25T10:59:08.469+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="108.13µs"
Mar 25 10:59:11 srv2.prod nomad[5744]:     2024-03-25T10:59:11.802+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="156.268µs"
Mar 25 10:59:14 srv2.prod nomad[5744]:     2024-03-25T10:59:14.219+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:54736
Mar 25 10:59:18 srv2.prod nomad[5744]:     2024-03-25T10:59:18.470+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="126.923µs"
Mar 25 10:59:21 srv2.prod nomad[5744]:     2024-03-25T10:59:21.804+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="147.992µs"
Mar 25 10:59:24 srv2.prod nomad[5744]:     2024-03-25T10:59:24.220+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:60886
Mar 25 10:59:28 srv2.prod nomad[5744]:     2024-03-25T10:59:28.472+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="134.108µs"
Mar 25 10:59:31 srv2.prod nomad[5744]:     2024-03-25T10:59:31.806+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="168.62µs"
Mar 25 10:59:34 srv2.prod nomad[5744]:     2024-03-25T10:59:34.221+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:51442
Mar 25 10:59:34 srv2.prod nomad[5744]:     2024-03-25T10:59:34.371+0300 [DEBUG] nomad: memberlist: Stream connection from=10.0.1.4:46336
Mar 25 10:59:38 srv2.prod nomad[5744]:     2024-03-25T10:59:38.474+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="146.373µs"
Mar 25 10:59:41 srv2.prod nomad[5744]:     2024-03-25T10:59:41.807+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="149.074µs"
Mar 25 10:59:44 srv2.prod nomad[5744]:     2024-03-25T10:59:44.221+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:40574
Mar 25 10:59:48 srv2.prod nomad[5744]:     2024-03-25T10:59:48.475+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="138.924µs"
Mar 25 10:59:51 srv2.prod nomad[5744]:     2024-03-25T10:59:51.808+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="164.435µs"
Mar 25 10:59:53 srv2.prod nomad[5744]:     2024-03-25T10:59:53.289+0300 [DEBUG] nomad: memberlist: Stream connection from=10.0.1.8:37994
Mar 25 10:59:54 srv2.prod nomad[5744]:     2024-03-25T10:59:54.222+0300 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:48038
Mar 25 10:59:58 srv2.prod nomad[5744]:     2024-03-25T10:59:58.476+0300 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="141.933µs"

If you need to check anything on my end, let me know. I just don't know what else to check.
Reverting to 1.5.3 fixes the issue

[jrasell]

Hi @wusikijeronii and thanks for raising this issue. Nomad v1.7.0 and above introduced improved cGroup and isolation primitives, which might be involved here. Could you confirm that cGroups are enabled and provide other information such as how the Nomad client is being run?

[wusikijeronii]

Hi @wusikijeronii and thanks for raising this issue. Nomad v1.7.0 and above introduced improved cGroup and isolation primitives, which might be involved here. Could you confirm that cGroups are enabled and provide other information such as how the Nomad client is being run?

Hello. Thank you for the reply. I provide results from OL8 (which doesn't work) and Ubuntu (which works).
OL8:

[root@srv1-prod ~]# awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 1
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
hugetlb 1
pids 1
rdma 1
[root@srv1-prod ~]#

Ubuntu:

root@srv3-prod:~# awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 1
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
hugetlb 1
pids 1
rdma 1
misc 1
root@srv3-prod:~#

I just don't know anything about cgroups, and I didn't set any related settings manually.

I didn't understand your question about Nomad running additional info. Do you mean binary, Docker, etc.? I use Nomad just as a binary app (from native host). Do you want me to share the client config?

[tgross]

Hi @wusikijeronii to follow up on those questions, I think we're looking for:

• Which version of cgroups are you running? You can get this by mount | grep cgroup
• Are you running Nomad as root? (as documented)
• Are you running Nomad via systemd?
• Do you have SELinux in enforcing mode?

Digging into the logs a bit more, it looks like the relevant sections for investigation are here:

Mar 25 10:58:02 srv2.prod nomad[5744]: 2024-03-25T10:58:02.666+0300 [INFO] client.driver_mgr.exec.executor: plugin process exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task plugin=/usr/bin/nomad id=37654
Mar 25 10:58:02 srv2.prod nomad[5744]: 2024-03-25T10:58:02.666+0300 [DEBUG] client.driver_mgr.exec.executor: plugin exited: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 driver=exec task_name=test-task
Mar 25 10:58:02 srv2.prod nomad[5744]: 2024-03-25T10:58:02.666+0300 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=c4ff2ba3-82f5-96c0-bbb6-2f45d34507f2 task=test-task type="Driver Failure" msg="failed to launch command with executor: rpc error: code = Unknown desc = unable to start container process: error during container init: read init-p: connection reset by peer" failed=false

It looks like the plugin is starting and trying to launch the container, but libcontainer is returning the error you reported.

[wusikijeronii]

• Which version of cgroups are you running? You can get this by mount | grep cgroup

[root@srv2 ~]# mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755,inode64)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
[root@srv2 ~]# grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2
[root@srv2 ~]#

• Are you running Nomad as root?

Yep.

• Are you running Nomad via systemd?

Yes. I also provide the unit file:

[Unit]
Description=Nomad
Documentation=https://nomadproject.io/docs/
Wants=network-online.target
After=network-online.target

# When using Nomad with Consul it is not necessary to start Consul first. These
# lines start Consul before Nomad as an optimization to avoid Nomad logging
# that Consul is unavailable at startup.
#Wants=consul.service
#After=consul.service

[Service]
EnvironmentFile=-/etc/nomad.d/nomad.env
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
LimitNPROC=infinity
Restart=on-failure
RestartSec=2

## Configure unit start rate limiting. Units which are started more than
## *burst* times within an *interval* time span are not permitted to start any
## more. Use `StartLimitIntervalSec` or `StartLimitInterval` (depending on
## systemd version) to configure the checking interval and `StartLimitBurst`
## to configure how many starts per interval are allowed. The values in the
## commented lines are defaults.

# StartLimitBurst = 5

## StartLimitIntervalSec is used for systemd versions >= 230
# StartLimitIntervalSec = 10s

## StartLimitInterval is used for systemd versions < 230
# StartLimitInterval = 10s

TasksMax=infinity
OOMScoreAdjust=-1000

[Install]
WantedBy=multi-user.target

• Do you have SELinux in enforcing mode?

No. SELinux is disabled

Digging into the logs a bit more, it looks like the relevant sections for investigation are here:

Yes, but I see the main error: "rpc error: code = Unknown desc ". I thought it meant an error happened while reading a message through the gRPC channel. So, I updated only the Nomad app but not the libcontainer. So, I can conclude: 1.7.3 version created some changes that involved reading or recognizing messages from the gRPC stream.
Just it reminds me of issues with my own developing another product. There is a binary format specification. This format has meta tags with a message type field. And when I change the message binary structure without compatibility with the old version of the binary format, that type is returned as undefined. because the bits differ and the final value is invalid. It looks so similar. I just wanted to share my thoughts.

[nickwales]

The actual failure appears to be happening with runc calling libcontainer.
For me this works with Nomad 1.7.x with ubuntu 22.04 but fails with ubuntu 20.04

runc:[2:INIT][2220]: segfault at 40000000f8 ip 00007f2dd438b6f0 sp 00007ffcdca35550 error 4 in libc-2.31.so[7f2dd4313000+178000]

More complete log output:

Mar 28 16:58:47 ip-10-0-1-36 nomad[2118]:     2024-03-28T16:58:47.499Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=cdb74d18-5312-1042-6b71-a121cf1ee469 task=sleep type="T
ask Setup" msg="Building Task Directory" failed=false
Mar 28 16:58:56 ip-10-0-1-36 nomad[2118]:     2024-03-28T16:58:56.150Z [INFO]  client.driver_mgr.exec: starting task: driver=exec driver_cfg="{Command:sleep Args:[1000000] ModePID: ModeIPC
: CapAdd:[] CapDrop:[]}"
Mar 28 16:58:56 ip-10-0-1-36 kernel: [ 1176.786401] show_signal_msg: 24 callbacks suppressed
Mar 28 16:58:56 ip-10-0-1-36 kernel: [ 1176.786404] runc:[2:INIT][2220]: segfault at 40000000f8 ip 00007f2dd438b6f0 sp 00007ffcdca35550 error 4 in libc-2.31.so[7f2dd4313000+178000]
Mar 28 16:58:56 ip-10-0-1-36 nomad[2118]:     2024-03-28T16:58:56.459Z [INFO]  client.driver_mgr.exec.executor: plugin process exited: alloc_id=cdb74d18-5312-1042-6b71-a121cf1ee469 driver=exec task_name=sleep plugin=/usr/bin/nomad id=2206
Mar 28 16:58:56 ip-10-0-1-36 nomad[2118]:     2024-03-28T16:58:56.459Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=cdb74d18-5312-1042-6b71-a121cf1ee469 task=sleep type="Driver Failure" msg="failed to launch command with executor: rpc error: code = Unknown desc = unable to start container process: error during container init: read init-p: connection reset by peer" failed=false

[tgross]

For me this works with Nomad 1.7.x with ubuntu 22.04 but fails with ubuntu 20.04

Thanks @nickwales. Given the segfault, I'm having a strong suspicion this is a build issue where we might not be linking against the right version of glibc in the build environment. I'll follow-up on this and report back.

[tgross]

So here's what I'm seeing with ldd, which shows that Nomad has symbols that are looking for glibc 2.34.

$ ./nomad version
Nomad v1.7.6
BuildDate 2024-03-12T07:27:36Z
Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e

$ ldd -r -v nomad
        linux-vdso.so.1 (0x00007ffe6d761000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f82ed4a9000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f82ed4a4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f82ed27b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f82ed4d5000)

        Version information:
        ./nomad:
                libresolv.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libresolv.so.2
                libpthread.so.0 (GLIBC_2.3.3) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libc.so.6 (GLIBC_2.11) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.8) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.9) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libresolv.so.2:
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

Nomad releases should be linked against glibc 2.31 (internal ref, which we need to turn into docs at some point), so this looks dubious. However, @wusikijeronii's report was that 1.5.8 worked fine, but I see 2.34 symbols in the 1.5.8 build as well.

@wusikijeronii can you run /lib/x86_64-linux-gnu/libc.so.6 on your OL server to verify the specific glibc version you've got? That might be a clue here.

[wusikijeronii]

@wusikijeronii can you run /lib/x86_64-linux-gnu/libc.so.6 on your OL server to verify the specific glibc version you've got? That might be a clue here.

Hello. Sure. /lib/x86_64-linux-gnu doesn't exist on my system, but....:

[root@srv2 bin]# ls -al /lib/libc*
-rwxr-xr-x  3 root root 1940100 Mar  6 21:24 /lib/libc-2.28.so
-rw-r--r--  3 root root  104136 Mar  6 21:24 /lib/libc_nonshared.a
lrwxrwxrwx. 1 root root      17 Jan 12  2022 /lib/libcom_err.so.2 -> libcom_err.so.2.1
-rwxr-xr-x. 3 root root   16204 Jan 12  2022 /lib/libcom_err.so.2.1
lrwxrwxrwx  1 root root      19 Dec 18 16:40 /lib/libcrypto.so.1.1 -> libcrypto.so.1.1.1k
-rwxr-xr-x  3 root root 2977064 Dec 18 16:41 /lib/libcrypto.so.1.1.1k
lrwxrwxrwx. 1 root root      17 Oct  9  2021 /lib/libcrypt.so -> libcrypt.so.1.1.0
lrwxrwxrwx. 1 root root      17 Oct  9  2021 /lib/libcrypt.so.1 -> libcrypt.so.1.1.0
-rwxr-xr-x. 3 root root  139496 Oct  9  2021 /lib/libcrypt.so.1.1.0
-rw-r--r--  3 root root     238 Mar  6 21:15 /lib/libc.so
lrwxrwxrwx  1 root root      12 Mar  6 21:16 /lib/libc.so.6 -> libc-2.28.so
lrwxrwxrwx  1 root root      16 Oct 18 19:02 /lib/libcurl.so.4 -> libcurl.so.4.5.0
-rwxr-xr-x  3 root root  661556 Oct 18 19:02 /lib/libcurl.so.4.5.0
[root@srv2 bin]# ldd -r -v /lib/libc.so.6
        /lib/ld-linux.so.2 (0xf7f0f000)
        linux-gate.so.1 (0xf7f0d000)

        Version information:
        /lib/libc.so.6:
                ld-linux.so.2 (GLIBC_2.3) => /lib/ld-linux.so.2
                ld-linux.so.2 (GLIBC_PRIVATE) => /lib/ld-linux.so.2
                ld-linux.so.2 (GLIBC_2.1) => /lib/ld-linux.so.2

It looks like I use 2.28. By the way, it is the latest version in the OL8 Base Stream (official stable mirror).
Just in case:

[root@srv2 bin]# nomad --version
Nomad v1.5.3
BuildDate 2023-04-04T20:09:50Z
Revision 434f7a1745c6304d607562daa9a4a635def7153f
[root@srv2 bin]# ldd -r -v nomad
        linux-vdso.so.1 (0x00007ffd76de9000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f7750804000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f77505e4000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f775021f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7750a1c000)

        Version information:
        ./nomad:
                libresolv.so.2 (GLIBC_2.2.5) => /lib64/libresolv.so.2
                libpthread.so.0 (GLIBC_2.3.3) => /lib64/libpthread.so.0
                libpthread.so.0 (GLIBC_2.3.2) => /lib64/libpthread.so.0
                libpthread.so.0 (GLIBC_2.2.5) => /lib64/libpthread.so.0
                libc.so.6 (GLIBC_2.11) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.8) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.9) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
        /lib64/libresolv.so.2:
                libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
        /lib64/libpthread.so.0:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
        /lib64/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

[wusikijeronii]

@tgross So, What the plan? Build fix? Either glibc 2.31 requirement or building from sources? I think I can install the newer version through unstable channels but I don't think it a good solution. At least because OL8 is supported untill 2032.