[bug] Too Many Open Files

[charitarthchugh]

🐛 Describe the bug

I am getting the too many open files error again, on latest release with 28k files being sent to be converted.

ping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-21 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-20 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-10 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-18 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-27 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,065 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-22 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,067 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-30 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,067 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-28 attempt 0: OSError: [Errno 24] Too many open files. Backoff 2/10, sleeping 20s
2026-01-24 20:28:38,067 - __main__ - WARNING - Connection error on ./pdfs/pdfDownload8/doc6483536.pdf-29 attempt 0: OSError: [Errno 24] Too many open file

Versions

argcomplete==3.1.4
attrs==23.2.0
Babel==2.10.3
bcc==0.29.1
blinker==1.7.0
Brlapi==0.8.5
certifi==2023.11.17
chardet==5.2.0
click==8.1.6
cloud-init==25.2
colorama==0.4.6
command-not-found==0.3
configobj==5.0.8
cryptography==41.0.7
cupshelpers==1.0
dbus-python==1.3.2
defer==1.0.6
distro==1.9.0
distro-info==1.7+build1
gpg==1.18.0
httplib2==0.20.4
idna==3.6
Jinja2==3.1.2
jsonpatch==1.32
jsonpointer==2.0
jsonschema==4.10.3
language-selector==0.1
launchpadlib==1.11.0
lazr.restfulclient==0.14.6
lazr.uri==1.0.6
louis==3.29.0
Markdown==3.5.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
netaddr==0.8.0
oauthlib==3.2.2
olefile==0.46
packaging==24.0
pexpect==4.9.0
pillow==10.2.0
pipx==1.4.3
platformdirs==4.2.0
psutil==5.9.8
ptyprocess==0.7.0
pycairo==1.25.1
pycups==2.0.1
Pygments==2.17.2
PyGObject==3.48.2
PyJWT==2.7.0
pyparsing==3.1.1
pyrsistent==0.20.0
pyserial==3.5
python-apt==2.7.7+ubuntu5.1
python-dateutil==2.8.2
python-debian==0.1.49+ubuntu2
pytz==2024.1
pyxdg==0.28
PyYAML==6.0.1
requests==2.31.0
rich==13.7.1
screen-resolution-extra==0.0.0
setuptools==68.1.2
six==1.16.0
ssh-import-id==5.11
systemd-python==235
typing_extensions==4.10.0
ubuntu-drivers-common==0.0.0
ubuntu-pro-client==8001
ufw==0.36.2
unattended-upgrades==0.1
urllib3==2.0.7
userpath==1.9.1
wadllib==1.3.6
wheel==0.42.0
xdg==5
xkit==0.0.0

Python 3.12.3, via conda

[jakep-allenai]

Yeah, I noticed that too, I just bumped the open file limit using ulimit... But perhaps this is tied into how many PDF pages get rendered at once, ex. those are uncapped, while GPU/vllm usage is limited by a boundedsemaphore.

[charitarthchugh]

Hi,
I ended up doing the same as well. Would this be worth mentioning in the readme or creating a FAQs document?

[jakep-allenai]

Updated the README for now, would be nice if the application could detect the limit and warn if your max concurrent requests was > 2x the limit or something similar.