Reverse proxy issue

[andye2004]

I have a slightly different configuration to the norm. I have an on-prem cluster that uses an envoy reverse proxy running on one or more cluster nodes (I know, it's bad and I shouldn't do it), this envoy instance proxies external requests into the cluster using the cluster DNS provider to resolve service names to clusterIPs before forwarding on to the upstream services.

I have now installed argo-cd and configured it as per the ingress docs when sitting behind a reverse proxy, e.g. set it to be insecure and added a basehref of /argo-cd, envoy has a re-write rule that drops the 'argo-cd' before forwarding on to the argocd-server.

If I access the argocd-server using https://proxy/argo-cd/login everything works as expected, e.g. I login and away we go.

However, if I try to access https://proxy/argo-cd or https://proxy/argo-cd/, envoy (correctly I think) forwards on to the argo service as http://argocd-server:80/but instead of being redirected to the login page I get a series of API requests being issued, some of which actually succeed and others end up in 401s.

When looking at things in chrome I can see that the apps main....js has been downloaded and then runs through the following requests:-

https://proxy/argo-cd/api/v1/settings (actually called twice) with both requests succeeding,
https://proxy/argo-cd/api/v1/cluster which fails with unauthorised 401
https://proxy/argo-cd/api/v1/session/userinfo request succeeds
https://proxy/argo-cd/api/v1/applications?fields=items.metadata.name%2Citems.metadata.annotations%2Citems.metadata.labels%2Citems.metadata.resourceVersion%2Citems.metadata.creationTimestamp%2Citems.metadata.deletionTimestamp%2Citems.spec%2Citems.operation.sync%2Citems.status.sync.status%2Citems.status.health%2Citems.status.operationState.phase%2Citems.status.operationState.operation.sync%2Citems.status.summary&selector= which again ends in 401.

The browser then goes into an infinite cycle of api calls to the same endpoints all of which end with the same results, either successful or unauthorised 401s.

Log file excerpts:

time="2020-01-12T12:47:02Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2020-01-12T12:47:02Z" grpc.time_ms=0.333 span.kind=server system=grpc
time="2020-01-12T12:47:43Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2020-01-12T12:47:43Z" span.kind=server system=grpc
time="2020-01-12T12:47:43Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2020-01-12T12:47:43Z" grpc.time_ms=0.462 span.kind=server system=grpc
time="2020-01-12T12:47:43Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2020-01-12T12:47:43Z" grpc.time_ms=0.284 span.kind=server system=grpc
time="2020-01-12T12:47:44Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2020-01-12T12:47:44Z" grpc.time_ms=0.423 span.kind=server system=grpc
time="2020-01-12T12:47:44Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.claims=null grpc.request.content= grpc.service=session.SessionService grpc.start_time="2020-01-12T12:47:44Z" span.kind=server system=grpc
time="2020-01-12T12:47:44Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2020-01-12T12:47:44Z" grpc.time_ms=0.399 span.kind=server system=grpc
time="2020-01-12T12:47:44Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = no session information" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2020-01-12T12:47:44Z" grpc.time_ms=0.3 span.kind=server system=grpc

Envoy log excerpt showing header fields on forwarded request:

[2020-01-12 11:44:37.313][66][debug][router] [source/common/router/router.cc:381] [C107][S133633471102791135] router decoding headers:
':authority', 'proxy:8443'
':path', '/'
':method', 'GET'
':scheme', 'http'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36'
'sec-fetch-user', '?1'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'sec-fetch-site', 'none'
'sec-fetch-mode', 'navigate'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-GB,en-US;q=0.9,en;q=0.8'
'cookie', 'argocd.token='
'if-modified-since', 'Sun, 12 Jan 2020 07:13:39 GMT'
'x-forwarded-proto', 'https'
'x-request-id', '54d3e714-e5c1-41ba-9e61-b1be92252349'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-envoy-original-path', '/argo-cd'

[alexmt]

Hello @andye2004 , I think you need to add --basehref flag of argocd-server deployment:

      - command:
        - argocd-server
        - --basehref
        - /proxy
        - --staticassets
        - /shared/app

[andye2004]

Thanks for the response @alexmt, it might not have been obvious in my original post but that setting is actually already in the config, as is --insecure. My config looks like:

          - argocd-server
            - --staticassets
            - /shared/app
            - --basehref
            - /argo-cd
            - --insecure

My proxy address, for the sake of this example is at proxy, this is NOT the basehref that is in use.

Entering https://proxy/argo-cd in my browser, the proxy then strips the argo-cd from the request before forwarding on to argocd-server. The reason I have the basehref of argo-cd stripped is because the docs say that argocd-server root context in not changed by adding the basehref, instead that is just added to any generated URLs returned to the calling client, in this case my browser. This is also in the docs regarding ingress, they state quite clearly that a URL rewrite rule is required.

Given the above I'm assuming there is something causing the argocd-server not to redirect the root URL to the login page and instead to try to serve up content, e.g. the main.js script that then fails as I have not logged in.

Entering https://proxy/argo-cd/login in the same browser doesn't encounter the same issues as I am able to login and everything works fine. This would tell me that the configuration I have set actually works and that there is an issue on the server side, my thoughts were that it was possibly something to do with the headers being added by the reverse proxy and this was why I had included an example of those to try to help identify the issue.

[jwenz723]

I experienced an issue similar to this. In my Ingress resource I had configured that the path prefix should match /argo-cd/.

This problem was resolved for me by removing the trailing slash character from my ingress path prefix, so I now have /argo-cd in my Ingress.

[tim-skillcafe]

I found what @jwenz723 found but via F12 in my browser. After trying to load the page, I pressed F12 to bring up the source. It was minimal but there was a link to extensions.js. When I tried to open that in a new tab, it showed that it expected a sub-folder of /argo-cd/. Adding that fixed it.

[TheQue42]

I am having the same issue, trying to deploy argocd in my kubernetes cluster. I've got both an ingress and a gateway instance both configured for reverse-proxying :443 ==> :80.

But if I try to use the rewrite mechanism to strip the URL prefix /argo from the URL, the argocd-server will just answer with a 404.

Removing the prefix stripping, makes argo answer straight away. The same config with a /ceph- prefix stipping for my rook-ceph deployment works fine.