Airflow | Failed to fetch log file from worker 403 Client Error: FORBIDDEN

[deen992]

below is the error get it from log - :
Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://app01.xxx.xxx.xxx.com.my:8793/log/example_bash_operator/also_run_this/2021-08-09T10:09:33.421241+00:00/1.log

then, i tried to curl link (http://xxxx.xxx.xxx.xxx.com.my:8793) in webserver, it getting details like this

• Rebuilt URL to: http://app01.xxxx.xxx.xxxx.com.my:8793/
• Trying 10.xxx.xxx.xxx...
• TCP_NODELAY set
• Connected to app01.xxxx.xxx.xxxx.com.my (10.xxx.xxx.xxx) port 8793 (#0)

GET / HTTP/1.1
Host: app01.xxxx.xxx.xxxx.com.my:8793
User-Agent: curl/7.61.1
Accept: /

• HTTP 1.0, assume close after body
  < HTTP/1.0 403 FORBIDDEN
  < Content-Type: text/html; charset=utf-8
  < Content-Length: 234
  < Server: Werkzeug/1.0.1 Python/3.8.3
  < Date: Mon, 09 Aug 2021 14:27:44 GMT
  <

<title>403 Forbidden</title>

Forbidden

You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.

* Closing connection 0

so, i try to dig log also from airflow-worker.log

2021-08-09 10:13:56,671 INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
2021-08-09 13:16:50,987 ERROR - 10.xx.xx.xx - - [09/Aug/2021 13:16:50] code 400, message Bad request syntax ('ÿôÿý\x06')
2021-08-09 13:16:50,989 INFO - 10.xx.xx.xx- - [09/Aug/2021 13:16:50] "ÿôÿý" HTTPStatus.BAD_REQUEST -
2021-08-09 14:51:02,865 INFO - 10.xx.xx.xx- - [09/Aug/2021 14:51:02] "GET /log/example_bash_operator/also_run_this/2021-08-09T06:37:40.096254+00:00/1.log HT
TP/1.1" 403 -
2021-08-09 15:01:33,089 INFO - 10.xx.xx.xx- - [09/Aug/2021 15:01:33] "GET /log/example_bash_operator/runme_0/2021-08-09T07:00:30.844023+00:00/1.log HTTP/1.1
" 403 -
2021-08-09 16:10:31,898 INFO - 10.xx.xx.xx - - [09/Aug/2021 16:10:31] "GET /log/example_bash_operator/also_run_this/2021-08-09T08:09:47.087292+00:00/1.log HT
TP/1.1" 403 -
2021-08-09 16:17:04,796 INFO - 10.xx.xx.xx- - [09/Aug/2021 16:17:04] "GET / HTTP/1.1" 403 -
2021-08-09 16:25:36,939 INFO - 10.xx.xx.xx- - [09/Aug/2021 16:25:36] "GET /log/example_bash_operator/run_after_loop/2021-08-09T08:24:59.289433+00:00/1.log H
TTP/1.1" 403 -
2021-08-09 16:29:05,398 INFO - 10.xx.xx.xx- - [09/Aug/2021 16:29:05] "GET /log/example_bash_operator/run_after_loop/2021-08-09T08:24:59.289433+00:00/1.log H
TTP/1.1" 403 -

can advise how to solve it? based on my understanding, from webserver trying to fetch log from worker but failed with error 403 as showing above.

• [ ]

• [ ]

• @sudhakar-nallam @rssanders3

[rssanders3]

I dont see this being related to the airflow-rest-api-plugin. Did this issue come up as a result of deploying the plugin?

[deen992]

@rssanders3 nope.. based on log given, it might be tried to find log from my webserver but file not exist, next it tried to reach my app1 to get the log.. it seem failed to retrieved and throw with above error.. i've add app1-(worker) hostname/IP in hosts file for both env webserver&app1(worker) .. it still showing same error