Backing up using webDAV from DAVx⁵ to nginx fails with HTTP 409, HTTP 405

[aqweoijn]

Using DAVx⁵ initially from f-droid version 4.2.6-ose - then changed to a locally compiled debug version off commit bee7cf20, with seedvault in LineageOS 19-20221220-microG-mata, when I chose DAVx⁵ as the backup location to setup a webDAV mount, I get DAVx⁵ logs:

01-10 17:02:33.284 V/davx5   ( 8315): [HttpClient] --> MKCOL https://<domain>/.SeedVaultAndroidBackup http/1.1                                                                                                                    
01-10 17:02:33.284 V/davx5   ( 8315): [HttpClient] User-Agent: DAVx5/4.3-alpha.1-ose (2023/01/10; dav4jvm; okhttp/4.10.0) Android/12                                                                                                          
01-10 17:02:33.285 V/davx5   ( 8315): [HttpClient] Accept-Language: en-CA, en;q=0.7, *;q=0.5                                                                                                                                                  01-10 17:02:33.286 V/davx5   ( 8315): [HttpClient] Host: <domain>                                                                                                                                                                 
01-10 17:02:33.286 V/davx5   ( 8315): [HttpClient] Connection: Keep-Alive                                                                                                                                                                     
01-10 17:02:33.286 V/davx5   ( 8315): [HttpClient] Accept-Encoding: gzip                                                                                                                                                                      
01-10 17:02:33.287 V/davx5   ( 8315): [HttpClient] --> END MKCOL                                                                                                                                                                              
01-10 17:02:33.287 D/dav4jvm ( 8315): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Trying Basic auth preemptively
01-10 17:02:33.288 D/dav4jvm ( 8315): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://<domain>/.SeedVaultAndroidBackup
01-10 17:02:33.295 V/davx5   ( 8315): [HttpClient] <-- 409 Conflict https://<domain>/.SeedVaultAndroidBackup (7ms)
01-10 17:02:33.295 V/davx5   ( 8315): [HttpClient] Server: nginx/1.23.3
01-10 17:02:33.296 V/davx5   ( 8315): [HttpClient] Date: Wed, 11 Jan 2023 01:02:33 GMT
01-10 17:02:33.296 V/davx5   ( 8315): [HttpClient] Content-Type: text/html
01-10 17:02:33.297 V/davx5   ( 8315): [HttpClient] Content-Length: 151
01-10 17:02:33.297 V/davx5   ( 8315): [HttpClient] Connection: keep-alive
01-10 17:02:33.297 V/davx5   ( 8315): [HttpClient] <-- END HTTP

and nginx logs:

Jan 10 17:02:33 web nginx[11506]: 2023/01/10 17:02:33 [error] 11506#11506: *3304 MKCOL can create a collection only, client: 192.168.1.1, server: <domain>, request: "MKCOL /.SeedVaultAndroidBackup HTTP/1.1", host: "<domain>"

and no files are created on the server.

If I hack a trailing slash onto the MKCOL in DAVx⁵, then I can add the backup location without error, but when I try to start a backup I get the logs:

01-10 17:11:02.248 V/davx5   ( 9605): [HttpClient] --> PROPFIND https://<domain>/ http/1.1
01-10 17:11:02.249 V/davx5   ( 9605): [HttpClient] Depth: 1 
01-10 17:11:02.249 V/davx5   ( 9605): [HttpClient] User-Agent: DAVx5/4.3-alpha.1-ose (2023/01/10; dav4jvm; okhttp/4.10.0) Android/12
01-10 17:11:02.250 V/davx5   ( 9605): [HttpClient] Accept-Language: en-CA, en;q=0.7, *;q=0.5
01-10 17:11:02.250 V/davx5   ( 9605): [HttpClient] Content-Type: application/xml; charset=utf-8
01-10 17:11:02.250 V/davx5   ( 9605): [HttpClient] Content-Length: 346
01-10 17:11:02.251 V/davx5   ( 9605): [HttpClient] Host: <domain>
01-10 17:11:02.251 V/davx5   ( 9605): [HttpClient] Connection: Keep-Alive
01-10 17:11:02.251 V/davx5   ( 9605): [HttpClient] Accept-Encoding: gzip
01-10 17:11:02.252 V/davx5   ( 9605): [HttpClient] --> END PROPFIND
01-10 17:11:02.252 D/dav4jvm ( 9605): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Trying Basic auth preemptively
01-10 17:11:02.252 D/dav4jvm ( 9605): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://<domain>/
01-10 17:11:02.256 V/davx5   ( 9605): [HttpClient] <-- 207 Multi-Status https://<domain>/ (3ms)
01-10 17:11:02.257 V/davx5   ( 9605): [HttpClient] Server: nginx/1.23.3
01-10 17:11:02.257 V/davx5   ( 9605): [HttpClient] Date: Wed, 11 Jan 2023 01:11:02 GMT
01-10 17:11:02.258 V/davx5   ( 9605): [HttpClient] Content-Type: text/xml; charset=utf-8
01-10 17:11:02.258 V/davx5   ( 9605): [HttpClient] Content-Length: 368
01-10 17:11:02.259 V/davx5   ( 9605): [HttpClient] Connection: keep-alive
01-10 17:11:02.259 V/davx5   ( 9605): [HttpClient] <-- END HTTP
01-10 17:11:02.263 D/davx5   ( 9605): [webdav.DavDocumentsProvider] SELF Response(requestedUrl=https://<domain>/, href=https://<domain>/, status=null, propstat=[PropStat(properties=[DisplayName(displayName=/), GetL
astModified(lastModified=1673428253000), [DAV::collection]], status=HTTP/1.1 200 OK, error=null)], error=null, newLocation=null)
...
01-10 17:11:02.349 V/davx5   ( 9605): [HttpClient] --> MKCOL https://<domain>/.SeedVaultAndroidBackup%2F http/1.1
01-10 17:11:02.349 V/davx5   ( 9605): [HttpClient] User-Agent: DAVx5/4.3-alpha.1-ose (2023/01/10; dav4jvm; okhttp/4.10.0) Android/12
01-10 17:11:02.349 V/davx5   ( 9605): [HttpClient] Accept-Language: en-CA, en;q=0.7, *;q=0.5
01-10 17:11:02.350 V/davx5   ( 9605): [HttpClient] Host: <domain>
01-10 17:11:02.350 V/davx5   ( 9605): [HttpClient] Connection: Keep-Alive
01-10 17:11:02.350 V/davx5   ( 9605): [HttpClient] Accept-Encoding: gzip
01-10 17:11:02.351 V/davx5   ( 9605): [HttpClient] --> END MKCOL
01-10 17:11:02.351 D/dav4jvm ( 9605): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Trying Basic auth preemptively
01-10 17:11:02.352 D/dav4jvm ( 9605): [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://<domain>/.SeedVaultAndroidBackup%2F
01-10 17:11:02.357 V/davx5   ( 9605): [HttpClient] <-- 405 Not Allowed https://<domain>/.SeedVaultAndroidBackup%2F (4ms)
01-10 17:11:02.357 V/davx5   ( 9605): [HttpClient] Server: nginx/1.23.3
01-10 17:11:02.358 V/davx5   ( 9605): [HttpClient] Date: Wed, 11 Jan 2023 01:11:02 GMT
01-10 17:11:02.359 V/davx5   ( 9605): [HttpClient] Content-Type: text/html
01-10 17:11:02.359 V/davx5   ( 9605): [HttpClient] Content-Length: 157
01-10 17:11:02.359 V/davx5   ( 9605): [HttpClient] Connection: keep-alive
01-10 17:11:02.360 V/davx5   ( 9605): [HttpClient] <-- END HTTP
01-10 17:11:02.363 E/JavaBinder( 9605): *** Uncaught remote exception!  (Exceptions are not yet supported across processes.)
01-10 17:11:02.363 E/JavaBinder( 9605): java.lang.RuntimeException: at.bitfire.dav4jvm.exception.HttpException: HTTP 405 Not Allowed
01-10 17:11:02.363 E/JavaBinder( 9605):         at android.os.Parcel.writeException(Parcel.java:2211)
01-10 17:11:02.363 E/JavaBinder( 9605):         at android.database.DatabaseUtils.writeExceptionToParcel(DatabaseUtils.java:117)

and nginx logs:

Jan 10 17:11:01 web nginx[11506]: 2023/01/10 17:11:01 [error] 11506#11506: *3729 mkdir() "/webdav/data/.SeedVaultAndroidBackup" failed (17: File exists), client: 192.168.1.1, server: <domain>, request: "MKCOL /.SeedVaultAndroidBackup%2F HTTP/1.1", host: "<domain>"
Jan 10 17:11:02 web nginx[11506]: 2023/01/10 17:11:02 [error] 11506#11506: *3731 mkdir() "/webdav/data/.SeedVaultAndroidBackup" failed (17: File exists), client: 192.168.1.1, server: <domain>, request: "MKCOL /.SeedVaultAndroidBackup%2F HTTP/1.1", host: "<domain>"

which results in just an empty .SeedVaultAndroidBackup/ created on the server.

I think nginx might be being more strict about webDAV than some other servers. I'm not sure if the responsibility lies with seedvault to i.e. MKCOL with a trailing slash, or if that's DAVx⁵'s responsibility to translate a generic "mkdir" into the appropriate webDAV MKCOL with a trailing slash - if it's the latter I'd be happy to move this issue to them.

I realize none of this clearly points to seedvault, seedvault is just erroring once DAVx⁵ throws an exception.

Have you seen anything like this, or do know what could be causing it?

[chirayudesai]

Thanks for the detailed report!

One thing to try here is:

Open the default "Files" app, and then try to copy some files to your DAV mount. You could even try to create a new folder .TestSeedVault, and then try to create or copy some files in there, to see if it works or not.

[aqweoijn]

No problem, it's the least I can do.

Trying with the default files app, I copied a handful of files to the webDAV mount without issue.

Trying to create .TestSeedVault as a directory fails with the same error:

Jan 11 17:02:03 web nginx[11506]: 2023/01/11 17:02:03 [error] 11506#11506: *9817 MKCOL can create a collection only, client: 192.168.1.1, server: <domain>, request: "MKCOL /.TestSeedVault HTTP/1.1", host: "<domain>"

This seems like intentional behaviour by nginx based on the interpretation that lacking trailing slash does not need to be handled by the server: https://trac.nginx.org/nginx/ticket/1966 and seems like the same thing that I'm seeing with seedvault.

That is, the server MAY handle it as if the trailing slash were present. It is, however, not required to. And this is not what nginx normally does

If I add a trailing slash, to create .TestSeedVault/ instead, it creates without error, but can't be viewed or navigated to, I think because of nginx's refusal to auto-index hidden files. Using a webDAV client on my desktop (cadaver), I can navigate to the collection by name, so I don't think that should be causing issues.

If I create TestSeedVault/ it again works correctly but this time can be seen in the Files app. I can also copy files into there.

Trying to create the directory a second time, like I saw in the initial test, fails with the error:

Jan 11 17:32:01 web nginx[11506]: 2023/01/11 17:32:01 [error] 11506#11506: *10252 mkdir() "/webdav/data/TestSeedVault" failed (17: File exists), client: 192.168.1.1, server: <domain>, request: "MKCOL /TestSeedVault%2F HTTP/1.1", host: "<domain>"

So it seems like

1. First error: nginx refuses to MKCOL/mkdir without a trailing slash - effecting both a manual attempt and seedvault. Unsure if DAVx⁵ should be handling that, or if it's up the caller.
2. Second error, after I hacked in the slash: seedvault seems to be trying to mkdir again in my initial test, which nginx also refuses. Again not sure if DAVx⁵ should be eating that request, or if the caller shouldn't be making it, assuming nginx's spec handling is correct.
3. Auto-index won't show me hidden files, but I don't think this is causing issues in this case.

[iskur]

Regarding 1.: the DAVx⁵ documentation (DAVx⁵ (WebDAV) has been successfully tested with nginx) proposes a work-around in the form of an nginx configuration suggestion. With that, requests without trailing slashes are rewritten to have them.

This does not solve the problem that seedvault or DAVx⁵ tries to create the .SeedVaultAndroidBackup directory even if it exists already, which causes the backup to fail.

[grote]

ok, so it seems that only 2 is left as a potential seedvault issue. ANy idea why it would try to create the dir two times? Normally, it should only do that when it thinks the dir doesn't exist.

[aqweoijn]

Thanks, the workaround for DAVx⁵ using the nginx rewrite does resolve 1.

Thanks for the hint on 2, based on it being expected to create the directory if it doesn't think it's present, 3 seems likely related. Checking seedvault source:

seedvault/app/src/main/java/com/stevesoltys/seedvault/plugins/saf/DocumentsStorage.kt

Line 248 in c856048

suspend fun DocumentFile.findFileBlocking(context: Context, displayName: String): DocumentFile? {

it looks like it's checking for file existence by listing the directory, rather than stat-ing the specific file or remembering internally. So seedvault thinking the hidden dir isn't there because nginx is refusing to index it, attempting to re-create it, and erroring seems reasonable.

I don't see any nginx config to auto-index all files, but there are some source patches online. I could also just use another server.

So seems like
1: DAVx⁵ issue, or arguably nginx issue, with easy workaround in nginx.
2: Implementation could probably be changed by seedvault to workaround, but ultimately seems due to 3.
3: Limitation by nginx, with no config to change, but can be patched.

I guess the combination of 2&3 should make seedvault generally incompatible with stock nginx?

I haven't tested this yet - I'll separate out my hacked web server from the one it's on now that's being used for other things so might take a bit, but I'm pretty confident of the cause. Thanks to all of you for the help, and thanks for the tool!

[iskur]

You closed the issue, but if you or someone else is interested in investigating further: the cause seems to lie in nginx's PROPFIND implementation arut/nginx-dav-ext-module#41
Seedvault/DAVx⁵ issues a PROPFIND request and nginx omits the .SeedVaultAndroidBackup directory in the response, because it starts with a ".". Seedvault/DAVx⁵ then goes on to try to create the seemingly non-existent directory and fails because it exists.

I am not sure which project should do anything about the problem. If seedvault would be willing to tackle it, it could allow customizing the name of the backup root directory (so the dot at the beginning could be removed).

[iskur]

I am able to backup over DAVx⁵ to nginx by applying this patch: mid1221213/nginx-dav-ext-module@51185d0

[grote]

If seedvault would be willing to tackle it, it could allow customizing the name of the backup root directory (so the dot at the beginning could be removed).

that's #143

[aqweoijn]

Thanks, yes I saw the nginx-dav-ext-module patch possibility, but thought it would make ongoing updates more of a pain so opted to try reverse proxying through nginx to stock apache. I then hit the problem described in #503 with the repeated MKCOL's causing 405's from apache when the directory already exits..

So I went back and applied the patch to nginx-dav-ext-module and built it as an external module, so that I can still run stock nginx from repos and hopefully update without having to recompile, at least as long as the module ABI doesn't change(?)

And yes agreed nginx with this patch does seem to work for me, the only thing that's worked so far.

If seedvault is wanting to apply a workaround, checking for file location on hidden files does seem to work in my test with unpatched nginx with cadaver client, it's just not listed. So the strategy for checking if the directory exists could be modified, if wanted. It still seems like it's not really seedvault's issue, but man this webdav ecosystem is less compatible than I expected.

[StevenPolley]

I've spent time trying to set up nginx only to run into this same problem. Is there an alternative webdav server is there that's working with Seedvault? I really don't want to have to resort to doing crazy things like patching nginx just to support seedvault

Edit: I managed to get the webdav backup working with Apache on Debian Bullseye. Had issues with Alpine linux missing a driver and doesn't seem to be available in Alpine's package manager.

[jt0in3e]

Is there an alternative webdav server is there that's working with Seedvault? I really don't want to have to resort to doing crazy things like patching nginx just to support seedvault

@StevenPolley you could give a try to lighttpd )

[grote]

Seedvault now has native WebDAV support. Did anyone here try this yet with nginx?