racy RLIMIT_NOFILE setting with Go 1.19+

[jrivera-px]

Description

We noticed an issue with 'ubuntu:bionic' where runc does not continually honor "RLIMIT_NOFILE " setting defined in the spec. After configuring the setting to '65536 ' Its keeps flipping between '65536' & '1024' intermittently. Does not appear to be consistent. Looking back, this issue appears to have started happening in runc-1.1.10 and above. Hoping that someone can provide a fix or some insight to why this is happening. The reproduction is using the latest runc-1.1.12

Steps to reproduce the issue

Running on an 'ubuntu:focal (20.04.6 LTS )' server with kernel 5.4.0-164-generic, using bash shell

1. As root create 'rootfs' directory: mkdir -p Issue/bionic/rootfs"
2. Pull ubuntu:bionic and unpack rootfs: docker export $(docker run -d ubuntu:bionic) |tar -C Issue/bionic/rootfs -x
3. Get runc-1.1.12: cd Issue && wget -q https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64 -O runc-1.1.12 && chmod +x runc-1.1.12
4. Create initial spec: cd bionic && ../runc-1.1.12 spec
5. Update 'RLIMIT_NOFILE' settings in spec : sed -i -e 's/"hard": 1024/"hard": 65536/' -e 's/"soft": 1024/"soft": 65536/' config.json
6. In another window on the same server as root in the same directory 'Issue' directory created in step 1 above. Start a monitoring process which will continually exec into the container and display the ulimit 'nofiles' setting. Install 'ts' utility if not present: apt-get install moreutils. Since we are specifying in the configuration that the setting should be '65536' we cut down on the output by searching for 1024. Printing a start date and timestamp to show the time of the flipping happens: date; while true; do sudo $(pwd)/runc-1.1.12 exec -t bionic /bin/sh -c 'ulimit -a' 2>&1 | grep nofiles; sleep 0.1; done | ts '%F %T' | egrep 1024
7. Coming back to the original window start the runc bionic container: cd .. && ./runc-1.1.12 run -b $(pwd)/bionic bionic

Describe the results you received and expected

In the monitoring window you initially just see a date, then intermittently you will see lines appear: nofiles 1024
e.g.
date; while true; do sudo $(pwd)/runc-1.1.12 exec -t bionic /bin/sh -c 'ulimit -a' 2>&1 | grep nofiles; sleep 0.1; done | ts '%F %T' | egrep 1024
Wed 07 Feb 2024 07:34:12 PM UTC
2024-02-07 19:35:18 nofiles 1024
2024-02-07 19:35:44 nofiles 1024
2024-02-07 19:36:08 nofiles 1024
2024-02-07 19:37:08 nofiles 1024

I would have expected 'nofiles' to never flip back to 1024 and remain at '65536' so we should never see any output.

What version of runc are you using?

./runc-1.1.12 --version
runc version 1.1.12
commit: v1.1.12-0-g51d5e946
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.4

Host OS information

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Host kernel information

Linux ip-10-13-108-144.pwx.purestorage.com 5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[lifubang]

Sorry, I can't reproduce it with your provided step.

date; while true; do sudo /workspaces/runc/runc exec -t test /bin/sh -c 'ulimit -a' 2>&1 | grep nofiles; sleep 0.1; done | ts '%F %T'
I printed all logs out, it looks like:

2024-02-08 02:07:44 nofiles              65536
2024-02-08 02:07:44 nofiles              65536
2024-02-08 02:07:45 nofiles              65536

[jrivera-px]

I am so sorry I left out an important detail. This may not happen right away. It's random and can take some time to see the flip. I usually see at least one flip within 15 minutes of letting this run.

…

On Wed, Feb 7, 2024 at 6:10 PM lfbzhm ***@***.***> wrote: Sorry, I can't reproduce it with your provided step. date; while true; do sudo /workspaces/runc/runc exec -t test /bin/sh -c 'ulimit -a' 2>&1 | grep nofiles; sleep 0.1; done | ts '%F %T' I printed all logs out, it looks like: 2024-02-08 02:07:44 nofiles 65536 2024-02-08 02:07:44 nofiles 65536 2024-02-08 02:07:45 nofiles 65536 — Reply to this email directly, view it on GitHub <#4195 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFPCEZMOPWMVTZTTQPY3XNDYSQXYJAVCNFSM6AAAAABC6XHBU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZTGI2TINBSGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jrivera-px]

@lifubang was looking to see if there was any progress made on this issue. We are looking to get an update for this because of vulnerabilities which are being flagged for go1.20.3

[lifubang]

Sorry, I just returned from vacation and haven't reproduced your question yet. If you could take the time to point out the problem, it would be greatly appreciated.

[lifubang]

Sorry, I spent twenty minutes to reproduce your issue, but unfortunately, I can't see the problem.

[jrivera-px]

Wow, this is strange. In our runtime env this issue is very consistent. I am attaching images of the latest repro. May I ask what type of a system you are running this on? Btw thank you for your help with this.

[ls-ggg]

I successfully reproduced this problem. I think the root cause is the changes related to rlimit-nofile in the new version of go. I've submitted a PR which should fix this issue.

[kolyshkin]

Reproduced locally, 40 times out of 1 million.