KhaaS : Bug into playbook and GRPC server doesn't works

[theoberthier]

Describe the bug

1. To have ui-jupyter i must modifed docker-compose.release.yaml to add your jupyter ui image

2. kubehound dump remote => add env on host :

   • AWS_ACCESS_KEY_ID=<>
   • AWS_SECRET_ACCESS_KEY=<>
   • AWS_DEFAULT_REGION=<>
   • AWS_ENDPOINT_URL=http://:

3. GRPC server deny connection

To Reproduce
Steps to reproduce the behavior:

1. launch all stack with :
   "docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml up -d"
   in /Kubehound/deployments/kubehound/
   this error it's raise : service "ui-jupyter" has neither an image nor a build context specified: invalid compose project

2. GRPC aren't reachable :
   add env variable describe in 2.
   when i try to reach the endpoint :9000, with grpc client or ./bin/build/kubehound dump remote --bucket s3://kh-bucket --insecure --khaas-server 10.10.20.50:9000

INFO[17:05:58] Loading application from inline command      
INFO[17:05:58] Using /home/<user>/.config/kubehound.yaml for default config 
INFO[17:05:58] Initializing application telemetry           
WARN[17:05:58] Telemetry disabled via configuration         
INFO[17:05:58] Loading Kubernetes data collector client     
WARN[17:05:58] About to dump k8s cluster: "default" - Do you want to continue ? [Yes/No] 


-----------------
INFO[17:06:01] Launching ingestion on <ip>:9000 [rundID: 01j7h0s28d1m5hckz4gbgngwaa] 
FATA[17:06:01] call Ingest (default:01j7h0s28d1m5hckz4gbgngwaa): rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp <ip>:9000: connect: connection refused"

i tried to logs a container, and execute shell inside to debug it, but i can't launch anything with docker exec -it ....

Expected behavior
When i launch ./kubehound dump remote ........ i except to push dump into s3 bucket (it's works) and i want to send RPC request to my GRPC server

[jt-dd]

Thanks for reporting the issue. I spotted some errors regarding the deployment example. We are deploying a fix #265 . Can you try redeploying with the following file:

• https://github.com/DataDog/KubeHound/blob/91a3e4a3ba1a221af1368a8b7415fba9158095a0/deployments/kubehound/docker-compose.release.yaml
• Command: docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml up

Also for easier setup, we are adding env variable to setup the ingestor/grpc image #264. Regarding your config what did you use regarding the ingestor.api.endpoint and ingestor.api.insecure ?

[jt-dd]

Everything have been updated in v1.5.1. It should work out of the box now. You can setup your environment using the env variable KH_*.

[theoberthier]

I have try to deploy v1.5.1 and in docker-compose.yaml, in ui-jupyter, the field "profile" stop the deployment of jupyter ui.
When i move profile, the deployment works or i put --profile jupyter, but the documentation don't talk about this.

• For the GRPC server, i have configured the KH_* env and i have "connection refused"
• For the bucket env i don't saw the env variable for the endpoint of bucket, to default, they s3 command like aws s3 ls s3 used the domain name of amazon , my-bucket.s3.amazon .... and i have need to specified a enpoint-url to contact my local s3 bucket.
  In version 1.4.0, AWS_ENDPOINT_URL is understood by Kubehound and I saw that the blob storage step was successful, but this is not the case in the new version.

the process blocked in blob storage step, with this error : "dump core: empty bucket name"

Thank you for your answers

[jt-dd]

For the GRPC server issue can you post:

• docker ps output
• docker logs kubehound-release-grpc-1 output (just make sure you anonymise the bucket name)

For the bucket, I am going to push a fix for it.

[theoberthier]

For sure :

$ docker ps
ghcr.io/datadog/kubehound-binary:latest   "/kubehound serve"       2 days ago   Up 41 seconds           0.0.0.0:9000->9000/tcp                                                         kubehound-release-grpc-1

$ docker logs kubehound-release-grpc-1

time="09:14:41" level=fatal msg="factory config creation: graph database client creation: E0104: no successful connections could be made: Forbidden"
time="09:14:42" level=info msg="Loading application configuration from default embedded"
time="09:14:43" level=warning msg="No local config file was found (kubehound.yaml)"
time="09:14:43" level=info msg="Using /kubehound for default config\n"
time="09:14:43" level=info msg="Initializing application telemetry"
time="09:14:43" level=warning msg="Telemetry disabled via configuration"
time="09:14:43" level=info msg="Starting KubeHound Distributed Ingestor Service"
time="09:14:43" level=info msg="Initializing providers (graph, cache, store)"
time="09:14:43" level=info msg="Loading cache provider"
time="09:14:43" level=info msg="Loaded memcache cache provider"
time="09:14:43" level=info msg="Loading store database provider"
time="09:14:43" level=info msg="Loaded mongodb store provider"
time="09:14:43" level=info msg="Loading graph database provider"
2024/09/19 09:14:43 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:14:43 Error creating new connection for connection pool: Forbidden
2024/09/19 09:14:43 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:14:43" level=warning msg="Retrying to connect [1/5]"
2024/09/19 09:14:53 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:14:53 Error creating new connection for connection pool: Forbidden
2024/09/19 09:14:53 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:14:53" level=warning msg="Retrying to connect [2/5]"
2024/09/19 09:15:03 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:03 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:03 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:03" level=warning msg="Retrying to connect [3/5]"
2024/09/19 09:15:13 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:13 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:13 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:13" level=warning msg="Retrying to connect [4/5]"
2024/09/19 09:15:23 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:23 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:23 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:23" level=warning msg="Retrying to connect [5/5]"
2024/09/19 09:15:33 Failed to instantiate the new connection; setting connection state to closed.
2024/09/19 09:15:33 Error creating new connection for connection pool: Forbidden
2024/09/19 09:15:33 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
time="09:15:33" level=fatal msg="factory config creation: graph database client creation: E0104: no successful connections could be made: Forbidden"

Here are the main logs that keep coming back

[jt-dd]

Did you pull the latest version using docker compose -f docker-compose.yaml -f docker-compose.release.yaml -f docker-compose.release.ingestor.yaml pull ?

Can you post the image sha of your image ?

• docker inspect kubehound-release-grpc-1 --format='{{.Image}}'

[theoberthier]

• docker inspect : sha256:d53db372b4202989fab80f00b43abeba21ce765b4d4fb2c9195cc873a9286b95

I pulled new images and i restarted, i have same message in new release when i launch kubehound dump remote :

• bucket name are empty

in v1.4.1 binary with the same env, when i dump remote the connection to GRPC server is refused with new images.

[jt-dd]

How do you set your bucket name ? If you set it from the config file kubehound.yaml, which key is setting it up ?

It should bucket_url like that:

# Ingestor configuration (for KHaaS)
ingestor:
  blob:
    # (i.e.: s3://<your-bucket>)
    bucket_url: ""

[theoberthier]

Hello, Happy new year !
Sorry for the answer delay.

I used configs/kubehound.yaml and the config is :

ingestor:
  blob:
    bucket_url: "s3://<ip>:<port>/<bucket-name>"  # i give ip because endpoint variable cannot set
    region: "us-east-1"
  temp_dir: "/tmp/kubehound"
  archive_name: "archive.tar.gz"
  max_archive_size: 2147483648 # 2GB
  # GRPC endpoint for the ingestor
  api:
    endpoint: "127.0.0.1:9000"
    insecure: true

i launch dump with this config. I give you output :

./kubehound-Linux-x86_64 dump remote --config kh-all-release/kubehound-v1.6.3/configs/etc/kubehound.yaml 
19:08:08 INFO Using file for default config app=kubehound path=/home/admuser/.config/kubehound.yaml
19:08:08 INFO Loading application configuration from file app=kubehound path=kh-all-release/kubehound-v1.6.3/configs/etc/kubehound.yaml
19:08:08 INFO Initializing application telemetry app=kubehound
19:08:08 WARN Telemetry disabled via configuration app=kubehound
19:08:08 INFO Temporary directory created app=kubehound path=/tmp/kubehound1419736751
19:08:08 INFO Loading Kubernetes data collector client app=kubehound
19:08:08 WARN About to dump k8s cluster - Do you want to continue ? [Yes/No] app=kubehound
y
19:08:09 INFO Loaded collector client app=kubehound
19:08:09 INFO Dumping cluster info to directory app=kubehound path=/tmp/kubehound1419736751
19:08:09 INFO Compression enabled app=kubehound
19:08:09 INFO Multi-threading enabled app=kubehound worker_count=7
19:08:09 INFO Dumping entity app=kubehound entity=nodes
19:08:09 INFO Dumping entity app=kubehound entity=clusterroles
19:08:09 INFO Dumping entity app=kubehound entity=pods
19:08:09 INFO Dumping entity app=kubehound entity=rolebindings
19:08:09 INFO Dumping entity app=kubehound entity=roles
19:08:09 INFO Dumping entity app=kubehound entity=clusterrolebindings
19:08:09 INFO Dumping entity app=kubehound entity=endpoints
19:08:09 INFO Streaming data from the K8s API app=kubehound
19:08:10 INFO Dumping entity done app=kubehound entity=nodes
19:08:10 INFO Dumping entity done app=kubehound entity=roles
19:08:11 INFO Dumping entity done app=kubehound entity=rolebindings
19:08:12 INFO Dumping entity done app=kubehound entity=endpoints
19:08:12 INFO Dumping entity done app=kubehound entity=pods
19:08:13 INFO Dumping entity done app=kubehound entity=clusterrolebindings
19:08:14 INFO Dumping entity done app=kubehound entity=clusterroles
19:08:14 INFO Dumping entity app=kubehound entity=Metadata
19:08:14 INFO Stats for the run time duration app=kubehound run=4.458384s wait=4.166225s throttling_percent=93.447%
19:08:14 INFO Dumping entity done app=kubehound entity=Metadata
19:08:14 INFO result saved to file app=kubehound path=/tmp/kubehound1419736751/default/kubehound_default_01jh12njkh19rgnc6qgb05mjhn.tar.gz
19:08:14 INFO Putting data on blob store bucket app=kubehound bucket_name=s3://<ip-bucket>:<port>/<bucket-name>
19:08:14 INFO Opening bucket app=kubehound bucket_name=s3://<ip-bucket>:<port>/<bucket-name>
19:08:14 INFO Opening bucket app=kubehound bucket_name=s3://<ip-bucket>:<port>/<bucket-name>
19:08:14 INFO Opening archive file app=kubehound path=/tmp/kubehound1419736751/default/kubehound_default_01jh12njkh19rgnc6qgb05mjhn.tar.gz
19:08:14 INFO Uploading archive from blob store app=kubehound key=default/kubehound_default_01jh12njkh19rgnc6qgb05mjhn.tar.gz
19:08:19 ERROR Error occurred app=kubehound error="fatal error: operation error S3: PutObject, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded"
19:08:19 FATAL dump core: operation error S3: PutObject, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, request canceled, context deadline exceeded app=kubehound
main.main
	/src/cmd/kubehound/main.go:16
runtime.main
	/usr/local/go/src/runtime/proc.go:272

While a GRPC container restart all times :

{"component":"kubehound-ingestor","level":"info","message":"Loading application configuration from default embedded","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"warning","message":"No local config file was found (kubehound.yaml)","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Using /kubehound for default config\n","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Initializing application telemetry","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"warning","message":"Telemetry disabled via configuration","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Starting KubeHound Distributed Ingestor Service","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Initializing providers (graph, cache, store)","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Loading cache provider","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Loaded memcache cache provider","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Loading store database provider","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Loaded mongodb store provider","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
{"component":"kubehound-ingestor","level":"info","message":"Loading graph database provider","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
2025/01/07 19:13:30 Failed to instantiate the new connection; setting connection state to closed.
2025/01/07 19:13:30 Error creating new connection for connection pool: Forbidden
2025/01/07 19:13:30 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
{"component":"kubehound-ingestor","level":"warning","message":"Retrying to connect [1/5]","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:30Z"}
2025/01/07 19:13:40 Failed to instantiate the new connection; setting connection state to closed.
2025/01/07 19:13:40 Error creating new connection for connection pool: Forbidden
2025/01/07 19:13:40 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
{"component":"kubehound-ingestor","level":"warning","message":"Retrying to connect [2/5]","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:40Z"}
2025/01/07 19:13:50 Failed to instantiate the new connection; setting connection state to closed.
2025/01/07 19:13:50 Error creating new connection for connection pool: Forbidden
2025/01/07 19:13:50 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
{"component":"kubehound-ingestor","level":"warning","message":"Retrying to connect [3/5]","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:13:50Z"}
2025/01/07 19:14:01 Failed to instantiate the new connection; setting connection state to closed.
2025/01/07 19:14:01 Error creating new connection for connection pool: Forbidden
2025/01/07 19:14:01 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
{"component":"kubehound-ingestor","level":"warning","message":"Retrying to connect [4/5]","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:14:01Z"}
2025/01/07 19:14:11 Failed to instantiate the new connection; setting connection state to closed.
2025/01/07 19:14:11 Error creating new connection for connection pool: Forbidden
2025/01/07 19:14:11 Error occurred during operation NewDriverRemoteConnection: 'E0104: no successful connections could be made: Forbidden'
{"component":"kubehound-ingestor","level":"warning","message":"Retrying to connect [5/5]","run_id":"01jh12zczkt9qrc7a2af9jj635","service":"kubehound","time":"2025-01-07T19:14:11Z"}

To ingest remotly it's same issue:

./kubehound-Linux-x86_64 ingest remote --config kh-all-release/kubehound-v1.6.3/configs/etc/kubehound.yaml 
19:17:49 INFO Using file for default config app=kubehound path=/home/admuser/.config/kubehound.yaml
19:17:49 INFO Loading application configuration from file app=kubehound path=kh-all-release/kubehound-v1.6.3/configs/etc/kubehound.yaml
19:17:49 INFO Initializing application telemetry app=kubehound
19:17:49 WARN Telemetry disabled via configuration app=kubehound
19:17:49 INFO Launching rehydratation [latest] app=kubehound endpoint=127.0.0.1:9000
19:17:49 FATAL call rehydratation (latest): rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 127.0.0.1:32916->127.0.0.1:9000: read: connection reset by peer" app=kubehound
main.main
	/src/cmd/kubehound/main.go:16
runtime.main
	/usr/local/go/src/runtime/proc.go:272

[jt-dd]

Why are you using an IP and port for the bucket ? Do you have an internal blob storage like Minio ? (asking to try to reproduce the error locally)

[theoberthier]

Yes i use localstack (it's like minio).
I think that for developers who don't have a real bucket aws with domain name, you should specify the endpoint url, like awscli with AWS_ENPOINT_URL.