HEAD to forward-auth protected URL returns HTTP 500

[D3N14L]

We have some URLs behind traefik-forward-auth and during some tests we found out that a HEAD (curl -I) to those URLs fails with a HTTP 500 return code after 30s.

❯ curl -k -v -I  https://prometheus.company.com
[...]
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 500
HTTP/2 500
< date: Wed, 29 Jul 2020 11:17:42 GMT
date: Wed, 29 Jul 2020 11:17:42 GMT

<
* Connection #0 to host prometheus.company.com left intact
* Closing connection 0

A GET to the same URL immediately returns the correct redirection with status code HTTP 307.

I have no idea what is supposed to happen upon a HEAD to those URLs, but if HEAD is not supported I would have expected a HTTP 405 method not allowed as the answer.

[thomseddon]

Hey,

I've just tested this and it works correctly when querying tfa directly, but something seems to be going wrong when traefik makes the forward auth request:

traefik-forward-auth_1  | time="2020-08-03T14:32:43Z" level=debug msg="Authenticating request" cookies="[]" handler=Auth host="whoami.localhost.com:8085" method=HEAD proto=http rule=default source_ip=172.18.0.1 uri=/
traefik-forward-auth_1  | time="2020-08-03T14:32:43Z" level=debug msg="Set CSRF cookie and redirected to provider login url" csrf_cookie="_forward_auth_csrf=a73c6f016165381376b19f21f44d9f94; Path=/; Domain=whoami.localhost.com; Expires=Tue, 04 Aug 2020 02:32:43 GMT; HttpOnly" handler=Auth host="whoami.localhost.com:8085" login_url="https://accounts.google.com/o/oauth2/auth?client_id=client_id.apps.googleusercontent.com&prompt=select_account&redirect_uri=http%3A%2F%2Fwhoami.localhost.com%3A8085%2F_oauth&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=a73c6f016165381376b19f21f44d9f94%3Agoogle%3Ahttp%3A%2F%2Fwhoami.localhost.com%3A8085%2F" method=HEAD proto=http rule=default source_ip=172.18.0.1 uri=/
traefik_1               | time="2020-08-03T14:33:13Z" level=debug msg="Error reading body http://traefik-forward-auth:4181. Cause: context deadline exceeded (Client.Timeout or context cancellation while reading body)" middlewareName=traefik-forward-auth@docker middlewareType=ForwardedAuthType
traefik_1               | 172.18.0.1 - - [03/Aug/2020:14:32:43 +0000] "HEAD / HTTP/1.1" 500 0 "-" "-" 1 "whoami@docker" "-" 30006ms

Looking at the code on traefik, it looks like they are throwing that error after attempting to read the body here: https://github.com/containous/traefik/blob/de458b735762ab12a66e92807ddc47345cdb44d0/pkg/middlewares/auth/forward.go#L104-L113

However from From RFC 2616:

9.4 HEAD

   The HEAD method is identical to GET except that the server MUST NOT
   return a message-body in the response. The metainformation contained
   in the HTTP headers in response to a HEAD request SHOULD be identical
   to the information sent in response to a GET request. This method can
   be used for obtaining metainformation about the entity implied by the
   request without transferring the entity-body itself. This method is
   often used for testing hypertext links for validity, accessibility,
   and recent modification.

So to me this looks like a bug in traefik?

[volker-schukai]

I have the same problem 'cause: context deadline exceeded' with a docker registry 2.7 connected. No image can be pushed via docker.

time="2020-11-07T14:28:39Z" level=debug msg="Error reading body http://traefik-forward-auth:4181/. Cause: context deadline exceeded (Client.Timeout or context cancellation while reading body)" middlewareType=ForwardedAuthType middlewareName=traefik-forward-auth@docker

curl -v https://docker-registry.example.com/v2/_catalog -> works
curl --head -v https://docker-registry.example.com/v2/_catalog -> 500 error

no authentication rules are defined:

DEFAULT_ACTION: allow

[jasonxh]

So to me this looks like a bug in traefik?

I'm hitting the same issue. I think in this case it's not really Traefik's fault, as Traefik always issues GET requests to forward auth service, and expects a proper response with a body, even if an empty one. The culprit is this line, where the original request method is overridden with that of the forwarded one:

traefik-forward-auth/internal/server.go

Line 59 in 4ffb659

r.Method = r.Header.Get("X-Forwarded-Method")

This confuses mux in its response writing. If X-Forwarded-Method is HEAD, it'll skip sending a response body (which is according to spec). It's easy to see how that's at odds with Traefik's expectation.

I tried simply commenting that line out, and it indeed resolved the issue for me. However, that'll have an impact on the rules functionality of this service, in that it won't be able to match against method anymore. I don't know of an elegant solution at the moment ...

[toxic0berliner]

I'm having the same issue here it seems, docker image push myrepo/myimage is not working, in traefik logs I see this :

[22/Dec/2021:22:35:13 +0000] "HEAD /v2/dev/stack-deployer/blobs/sha256:XXXXX HTTP/1.1" 500 0 "-" "-" 162135 "websecure-gitlab-registry@docker" "-" 30000ms

So I do feel I'm stuck as I've put this forwardauth middleware by default on the entrypoint which I very much need as it applies to some 50 or so container that don't have to explicitely enable this middleware.
But as traefik will not allow me to just simply remove this middleware for the entrypoint, I'd realy like to get HEAD working with this middleware...

I see in @jasonxh post that commenting out the line would resolve the issue but prevent me to use methos in my rules, which I don't use so I could go with that workaround...
But I deploy this middleware using the docker image thomseddon/traefik-forward-auth:latest and I see no way here to disable it...

Any chance we could get that fixed @thomseddon or maybe push another tag latest-nomethod ? Or maybe @jasonxh has already pushed a version with that commented out ?

Thanks in advance for any help, I'm going to bed now but I'll give it another try tomorrow and mayby get to build my own dowker image of this repo with the line commented out.

[jasonxh]

Or maybe @jasonxh has already pushed a version with that commented out ?

I've only built an image for arm64 with the workaround. If it happens to be what you need, it's at jasonxh/traefik-forward-auth:latest-arm64.

[toxic0berliner]

Thanks @jasonxh but I run it on x86_64. Will build it myself. Trying out something else with traefik first (a third entry point on a different port to bypass the middleware)

[toxic0berliner]

I finally got around to do this, and indeed I got this working by

• git clone this repo
• comment line 59 on internal/server.go
• docker build .
  meanwhile I did fork this on my personal gitlab and published the image there and sorry for the next ones it's not a publicly available repo, but hey the steps to build were made very easy by @thomseddon thanks a lot !

I'm going to stick with this workaround ad per @jasonxh it should only impact rules on the method, which is something I don't use, I stick to domain or path rules ;)

But I'll be monitoring here if someone finds a "more elegant solution" and I can go back to using the official image for this wonderful forward-auth plugin !

Note that I'm also counting on this issue at traefik hopefully moving forward, that would allow me to simply disable the forward-auth middleware just for the docker repository router, but I need to keep it on for all the rest (right now my allow rule makes this middleware transparent as long as it doesn't do strange things to the method...)

[mdinicola]

If anyone still needs it, I've built an x86_64 image with the workaround. It's available at mdinicola/traefik-forward-auth:latest

[shaif-dorIT]

Hi,
I'm using traefik and foward-auth to secure an Homer application.

Homer has an HTTP request to check connectivity with the server using HEAD request using the fetch API.

When I disable the forward-auth middleware the connectivity check works and I get 200 OK.
When I put the the middleware back it's hangs and gets a 500 error result.

So, it's seems that the plugin does not work with HEAD requests.

Beside using the workaround is there a way to make the plugin ignore the connectivity check based on some rule that check for specific QS parameter?

[toxic0berliner]

I didn't try because I was not wanting to disable auth even for Head request only but if that's acceptable to you there is a documentation here that explains the rules you can put in forward.ini and you would probably be able to always authorize requests based on the query string.