[Bug]: Large folder downloads delayed by HEAD request processing for authenticated users #57790
Description
⚠️ This issue respects the following points: ⚠️
- This is a bug, not a question or a configuration/webserver/proxy issue.
- This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- I agree to follow Nextcloud's Code of Conduct.
Bug description
When downloading large files as an authenticated user via the webui, there is a significant delay before the download actually starts. The delay is proportional to the file count and total folder size. This only happens when downloading folders, single files download instantly. During the delay I can see high read disk io on the server.
However, when sharing the folder via a public link and downloading it from there starts instantly.
Using the browser console I see that when clicking download it first does a HEAD request (https://cloud.X.com/remote.php/dav/files/name/folder/?accept=zip ) first. During this time I can see a loading spinner and the dik io on the server, I do not receive any data to the client though. After a long time (e.g. a 70gb folder takes 6 minutes), the download starts. This time a GET request to the same url is sent and the expected download happens.
On the public link it does not send any HEAD request and just does the GET request and the download starts instantly.
Not sure if related but when I have debug logging enabled on the server I can see a lot of those errors (only when it sends the HEAD request):
Implicit conversion from float 1.8446744069414584E+19 to int loses precision at /var/www/html/3rdparty/deepdiver/zipstreamer/src/Count64.php#112
Implicit conversion from float 1.8446744069414584E+19 to int loses precision at /var/www/html/3rdparty/deepdiver/zipstreamer/src/Count64.php#119
The download does not fail however, it eventually starts and completes successfully.
Steps to reproduce
- When logged in click download in the web interface for any folder
- In the browser dev tools see a HEAD request being sent first which takes a long time
- See high read disk io (presumably it's reading every file)
- Eventually a GET request is sent after the HEAD request succeeds
- Download starts in the browser
Expected behavior
The download starts instantly.
Nextcloud Server version
32
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.2
Web server
Nginx
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
None
Are you using the Nextcloud Server Encryption module?
None
What user-backends are you using?
- Default user-backend (database)
- LDAP/ Active Directory
- SSO - SAML
- Other
Configuration report
{
"system": {
"htaccess.RewriteBase": "\/",
"memcache.locking": "\\OC\\Memcache\\Redis",
"memcache.distributed": "\\OC\\Memcache\\Redis",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": "0",
"password": "***REMOVED SENSITIVE VALUE***"
},
"apps_paths": [
{
"path": "\/var\/www\/html\/apps",
"url": "\/apps",
"writable": false
},
{
"path": "\/var\/www\/html\/custom_apps",
"url": "\/custom_apps",
"writable": true
}
],
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"cloud.x.com"
],
"trusted_proxies": "***REMOVED SENSITIVE VALUE***",
"default_language": "de",
"default_phone_region": "ch",
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "pgsql",
"version": "32.0.4.1",
"overwrite.cli.url": "https:\/\/cloud.x.com",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "6432",
"dbtableprefix": "oc_",
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_smtpmode": "smtp",
"mail_sendmailmode": "smtp",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "465",
"overwritehost": "cloud.x.com",
"overwriteprotocol": "https",
"loglevel": 3,
"log_type": "syslog",
"syslog_tag": "Nextcloud",
"logfile": "",
"maintenance": false,
"maintenance_window_start": 1,
"defaultapp": "",
"allow_local_remote_servers": true,
"simpleSignUpLink.shown": false,
"memcache.local": "\\OC\\Memcache\\APCu",
"enabledPreviewProviders": [
"OC\\Preview\\MP3",
"OC\\Preview\\TXT",
"OC\\Preview\\MarkDown",
"OC\\Preview\\OpenDocument",
"OC\\Preview\\Krita",
"OC\\Preview\\Imaginary",
"OC\\Preview\\Movie"
],
"preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
"allow_user_to_change_display_name": false,
"lost_password_link": "disabled",
"app_install_overwrite": {
"1": "htmlviewer"
},
"onlyoffice": {
"editors_check_interval": "0"
},
"auth.webauthn.enabled": false,
"upgrade.disable-web": false,
"theme": "",
"user_oidc": {
"enrich_login_id_token_with_userinfo": true
},
"session_lifetime": 86400,
"session_keepalive": true,
"session_relaxed_expiry": true,
"remember_login_cookie_lifetime": 1296000,
"mail_smtpauth": true,
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"mail_smtpsecure": "ssl",
"dbpersistent": true,
"tempdirectory": "\/mnt\/nextcloud-tmp"
}
}List of activated Apps
Enabled:
- activity: 5.0.0-dev.0
- admin_audit: 1.22.0
- app_api: 32.0.0
- appointments: 2.6.2
- bruteforcesettings: 5.0.0-dev.0
- calendar: 6.1.5
- circles: 32.0.0
- cloud_federation_api: 1.16.0
- comments: 1.22.0
- contacts: 8.3.0
- contactsinteraction: 1.13.1
- dashboard: 7.12.0
- dav: 1.34.2
- deck: 1.16.3
- end_to_end_encryption: 1.18.0
- external: 7.0.0
- federatedfilesharing: 1.22.0
- files: 2.4.0
- files_3dmodelviewer: 0.0.16
- files_downloadlimit: 5.0.0-dev.0
- files_fulltextsearch: 32.0.2
- files_linkeditor: 1.1.23
- files_lock: 32.0.1
- files_pdfviewer: 5.0.0-dev.0
- files_reminders: 1.5.0
- files_sharing: 1.24.1
- files_trashbin: 1.22.0
- files_versions: 1.25.0
- fileslibreofficeedit: 2.0.1
- forms: 5.2.4
- fulltextsearch: 32.0.0
- fulltextsearch_elasticsearch: 32.0.2
- groupfolders: 20.1.9
- logreader: 5.0.0-dev.0
- lookup_server_connector: 1.20.0
- mail: 5.6.8
- metadata: 0.23.0
- music: 2.5.1
- news: 27.2.0
- notes: 4.12.4
- notifications: 5.0.0-dev.0
- notify_push: 1.3.0
- oauth2: 1.20.0
- photos: 5.0.0-dev.1
- privacy: 4.0.0-dev.0
- profile: 1.1.0
- provisioning_api: 1.22.0
- recommendations: 5.0.0-dev.0
- related_resources: 3.0.0-dev.0
- richdocuments: 9.0.2
- serverinfo: 4.0.0-dev.0
- settings: 1.15.1
- sharebymail: 1.22.0
- socialsharing_email: 4.0.1
- spreed: 22.0.8
- suspicious_login: 10.0.0-dev.0
- tasks: 0.17.1
- text: 6.0.1
- theming: 2.7.0
- thesearchpage: 2.0.8
- twofactor_backupcodes: 1.21.0
- updatenotification: 1.22.0
- user_oidc: 8.3.0
- user_status: 1.12.0
- viewer: 5.0.0-dev.0
- weather_status: 1.12.0
- whiteboard: 1.5.3
- workflow_pdf_converter: 3.0.0
- workflowengine: 2.14.0
Disabled:
- announcementcenter: 7.3.0 (installed 7.3.0)
- checksum: 2.0.3 (installed 2.0.3)
- cookbook: 0.11.5 (installed 0.11.5)
- emlviewer: 2.0.1 (installed 2.0.1)
- encryption: 2.20.0
- epubviewer: 1.8.1 (installed 1.8.1)
- federation: 1.22.0 (installed 1.20.0)
- files_automatedtagging: 3.0.3 (installed 3.0.3)
- files_external: 1.24.0
- firstrunwizard: 5.0.0-dev.0 (installed 3.0.0)
- flow_notifications: 3.0.0 (installed 3.0.0)
- htmlviewer: 32.0.0 (installed 32.0.0)
- imageconverter: 2.1.0 (installed 2.1.0)
- nextcloud_announcements: 4.0.0-dev.0 (installed 1.18.0)
- password_policy: 4.0.0-dev.0 (installed 2.0.0)
- support: 4.0.0-dev.0 (installed 4.0.0-dev.0)
- survey_client: 4.0.0-dev.0 (installed 1.16.0)
- systemtags: 1.22.0 (installed 1.21.1)
- talk_matterbridge: 1.33.1026000 (installed 1.33.1026000)
- twofactor_nextcloud_notification: 6.0.0-dev.0
- twofactor_totp: 14.0.0 (installed 12.0.0-dev)
- user_ldap: 1.23.0
- webhook_listeners: 1.3.0 (installed 1.1.0-dev)
- workflow_ocr: 1.32.1 (installed 1.32.1)
- workflow_script: 3.0.0 (installed 3.0.0)Nextcloud Signing status
No errors have been found.Nextcloud Logs
Additional info
When I looked at the server-side code, I found that SabreDAV CorePlugin.php converts HEAD requests to GET requests internally. It does that around line 242 in the httpHead() function. It seems to clone the request and change the method to GET while adding a header X-Sabre-Original-Method: HEAD. (line 248)
The standard CorePlugin::httpGet() function (line 73) checks for this header and skips body creation if it detects a HEAD request.
However in the ZipFolderPlugin.php, I didn't see any similar check for the X-Sabre-Original-Metod header. The handleDownload (line 97) function seems to process HEAD requests the same way as GET requests, which might explain why it's taking so long and I only see read io on the server (like it would when downloading).
I'm not a php developer, so these are just assumptions based on what I observed.