Temporal services are not starting when deployed in istio enabled namespace #338
Description
What are you really trying to do?
Trying to deploy temporal in istio enabled namespace along with PostgreSQL DB on a Kubernetes cluster.
Deployment is done using helm charts available at - https://github.com/temporalio/helm-charts
Describe the bug
Post deployment history, matching and worker PODs are continuously restarting with CrashLoopBackOff with below mentioned logs.
Here is status of PODs:
NAME READY STATUS RESTARTS AGE
temporaldb-7d7f9c67b8-qqxbb 2/2 Running 0 22m
temporaltest-admintools-7bf55d4b8f-t47m2 2/2 Running 0 16m
temporaltest-frontend-546c569b9b-g4gv4 2/2 Running 0 16m
temporaltest-history-8648bd7db9-xsnz4 1/2 CrashLoopBackOff 6 (74s ago) 16m
temporaltest-matching-bc9f69487-9vc95 1/2 CrashLoopBackOff 6 (52s ago) 16m
temporaltest-web-67d46c8748-zf4xn 2/2 Running 0 16m
temporaltest-worker-5c58f8cc58-c6jwd 1/2 Error 6 (4m32s ago) 16m
LOGs:
{"level":"info","ts":"2022-12-07T17:48:04.721Z","msg":"bootstrap hosts fetched","service":"worker","bootstrap-hostports":"10.4.0.125:6939,10.4.0.124:6933,10.4.0.126:6935,10.4.0.127:6934","logging-call-at":"rpMonitor.go:285"}
{"level":"error","ts":"2022-12-07T17:48:46.557Z","msg":"unable to bootstrap ringpop. retrying","service":"worker","error":"join duration of 41.835509067s exceeded max 30s","logging-call-at":"ringpop.go:109","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap.func1\n\t/home/builder/temporal/common/membership/ringpop.go:109\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:170\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:194\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:171\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap\n\t/home/builder/temporal/common/membership/ringpop.go:114\ngo.temporal.io/server/common/membership.(*RingPop).Start\n\t/home/builder/temporal/common/membership/ringpop.go:84\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:135\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:268\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/internal/lifecycle/lifecycle.go:120\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/internal/lifecycle/lifecycle.go:85\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/app.go:683\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/app.go:773"}
{"level":"fatal","ts":"2022-12-07T17:48:46.557Z","msg":"unable to bootstrap ringpop. exhausted all retries","service":"worker","error":"join duration of 41.835509067s exceeded max 30s","logging-call-at":"ringpop.go:116","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/home/builder/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap\n\t/home/builder/temporal/common/membership/ringpop.go:116\ngo.temporal.io/server/common/membership.(*RingPop).Start\n\t/home/builder/temporal/common/membership/ringpop.go:84\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:135\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:268\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/internal/lifecycle/lifecycle.go:120\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/internal/lifecycle/lifecycle.go:85\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/app.go:683\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.17.1/app.go:773"}
Environment/Versions
- OS and processor: GCP VM - Intel(R) Xeon(R) CPU @ 2.80GHz with 4 procs and 32 GB RAM, Ubuntu 20.04.5 LTS
- Temporal Version: 1.18.5
- Are you using Docker or Kubernetes or building Temporal from source? Kubernetes (helm charts)
Additional context
Temporal deployment is done on KiND based Kubernetes cluster
KiND version: v0.17.0
K8s version: 1.25.3
Istio version: 1.16.0