Avoid infinite parse error logging loopback

[kenhys]

Without it, if it reads unexpected format, error log is emitted and also read. Then it also cause parse error again.

During that process, '' will be escaped and as a result, make pod evict.

1. emit parse error when it accepts unexpected format
2. Fluentd emits with pattern not matched: "..." warning
3. Kubeneters collect such a warning into /var/log/containsers/fluentd-xxx.log
4. If user didn't set FLUENT_CONTAINER_TAIL_EXCLUDE_PATH explicitly, collect fluentd-xxx.log which contains warning message
5. it results in parse error again... Thus in infinite parse error, '' will be escaped and escaped ...

[kenhys]

changed not to use use_default. (ensure to fallback)

[kenhys]

See #1577 (comment)

How to verify:

1. re-generate docker-image template. e.g. debian-forward

make src DOCKERFILE=v1.19/debian-forward VERSION=v1.19.2-debian-forward-amd64-1.0 TAGS=v1.19.0-debian-forward-amd64-1.0,v1.19-debian-forward-amd64-1

2. build daemonset image.

cd docker-image/v1.19/debian-forward
podman build -t k8s-forward .

3. import daemonset image into minikube

podman save localhost/k8s-forward:latest -o k8s-forward.tar
minikube image load k8s-forward.tar

4. kubectl apply -f fluentd-config.yaml which specify FLUENT_CONTAINER_TAIL_EXCLUDE_PATH.

cat fluentd-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: default
data:
  fluent.conf: |
    <source>
      @type tail
      # Read fluentd log to reproduce the problem
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.test
      read_from_head true
      exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH']}"
      <parse>
        @type json
      </parse>
    </source>

    <match **>
      @type stdout
    </match>

5. deploy fluentd with modified configuration (kubectl apply -f fluentd-test.yaml )

      containers:
      - name: fluentd
#        image: fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0
        image: localhost/k8s-forward:latest
        imagePullPolicy: Never
        env:
          - name: FLUENT_ELASTICSEARCH_HOST
            value: "localhost" # dummy

6. check logs with kubectl logs -l k8s-app=fluentd-logging

Expect no '\\\' escape hell.

1970-01-01 00:33:46.440223639 +0000 kubernetes.test: {"log":"W0303 05:06:24.440057       1 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice\n","stream":"stderr"}
1970-01-01 00:33:46.449230249 +0000 kubernetes.test: {"log":"W0303 05:06:24.449120       1 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice\n","stream":"stderr"}

[Watson1978]

👍🏻