Clickhouse with S3 not accessiable by Langfuse web and worker containers

[Subramanian-ERS]

Describe your question

I wish to use S3 Blob Storage as Clickhouse disk for my on premise langfuse server setup on AWS EC2. My Clickhouse container has to use the AWS EC2 instance profile for authenticating to S3. My clickhouse config.xml looks like this:

<clickhouse>
    <merge_tree>
        <storage_policy>s3</storage_policy>
    </merge_tree>
    <storage_configuration>
        <disks>
            <s3>
                <type>object_storage</type>
                <object_storage_type>s3</object_storage_type>
                <metadata_type>local</metadata_type>
                <endpoint>https://s3.us-east-1.amazonaws.com/<bucket_name>/clickhouse_data/</endpoint>
                <use_environment_credentials>1</use_environment_credentials>
                <server_side_encryption_customer_key_id>AES256</server_side_encryption_customer_key_id>
                <skip_access_check>true</skip_access_check>
                <addressing_style>path</addressing_style>
            </s3>
        </disks>
        <policies>
            <s3>
                <volumes>
                    <main>
                        <disk>s3</disk>
                    </main>
                </volumes>
            </s3>
        </policies>
    </storage_configuration>
</clickhouse>

But I found out the the Clickhouse container was not able to use the AWS EC2 instance profile to authenticate to S3 until I made a change in docker-compose.yml to include network_mode: host for the clickhouse container config. After this, I could see that the Clickhouse container did start successfully and could see some data being populated in my S3 folder. But, the langfuse containers were not able to access the clickhouse container. When I started the containers I could see this warning next to my clickhouse container:

Published ports are discarded when using host network mode

Any help here is much appreciated.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.155.1 OSS

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The issue you're encountering is a networking conflict: when you use network_mode: host for ClickHouse, the container shares the host's network stack directly, which means it no longer participates in Docker's internal networking and the published ports are discarded.(1)

This creates a problem because the Langfuse web and worker containers are trying to reach ClickHouse via Docker's internal network (e.g., http://clickhouse:8123), but ClickHouse is now only accessible via the host's network interface.

Solution Options

Option 1: Use host networking for all containers

If ClickHouse requires host networking for EC2 instance profile credentials, you could also set network_mode: host for the Langfuse web and worker containers. Then configure the ClickHouse connection URLs to use localhost:

CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000

(1)

Option 2: Pass AWS credentials explicitly to ClickHouse

Instead of relying on the EC2 instance profile via host networking, you can explicitly provide AWS credentials to the ClickHouse container. In your config.xml, replace <use_environment_credentials>1</use_environment_credentials> with explicit credentials:

<endpoint>https://s3.eu-central-1.amazonaws.com/example-bucket-name/data/</endpoint>
<access_key_id>ACCESS_KEY</access_key_id>
<secret_access_key>ACCESS_KEY_SECRET</secret_access_key>

(1)

This allows ClickHouse to remain on Docker's bridge network, keeping it accessible to the Langfuse containers via the internal hostname.

Option 3: Use a credentials proxy or IAM Roles Anywhere

If you need to maintain the EC2 instance profile approach without host networking, you could explore using a credentials proxy service that runs on the host and is accessible to containers, or AWS IAM Roles Anywhere for container-based credential retrieval.

The recommended approach for most self-hosted setups is Option 2 - explicitly passing credentials - as it maintains Docker's internal networking and keeps the configuration simpler.(1)

📚 Sources:

1. ClickHouse

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @Subramanian-ERS, was inkeep able to help you here?

[Subramanian-ERS]

Hi @jannikmaierhoefer Yes, I had to configure host networking mode for all my docker containers. I have a small follow up on this architecture. Blob Storage as Disk(https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse#blob-storage-as-disk) says that the Clickhouse metadata is still stored on local server even if S3 disk is used. But in that case, how would I setup a different server with the same clickhouse data? I assume that just pointing the new server to the persistent S3 Clickhouse folder would not work due to the metadata being stored on the local server.

[Steffen911]

@Subramanian-ERS Happy to hear that this solved the issue.

Regarding adding additional servers, ClickHouse supports a replicated merge-tree strategy that uses ClickHouse Keeper to store metadata between nodes and replicate data parts. Please check out the ClickHouse documentation for more details on this.

[Subramanian-ERS]

@Steffen911 Thanks for the suggestion. After going through the documentation, I was able to come up with a backup strategy that involves taking backup of the metadata on the local storage along with S3.