Getting Internal Server Error: traces.metrics on langfuse dashboard

[rjayaram-godaddy]

Getting Internal Server Error: traces.metrics on langfuse dashboard.
We have self hosted langfuse( version 3.116.1 is the version I can see on dashboard)

The traces are visible on the dashboard but this error pops up randomly at times. Any idea why this is happening?

For more context, I was facing some clickhouse migration issue so I ran the migrations manually ( By cloning the langfuse repo and runnning the sql files from path
langfuse/packages/shared/clickhouse/migrations/unclustered

Is this related to db migration or is this some other issue?

[inkeep-ai-assistant[bot]]

The "Internal Server Error: traces.metrics" error you're encountering on your self-hosted Langfuse dashboard is likely related to ClickHouse query performance or resource issues.(1)

Since you mentioned running ClickHouse migrations manually, this could be related to:

1. ClickHouse Resource Constraints: The error appears intermittently, which suggests ClickHouse may be under-resourced when handling metric queries. For larger deployments, ClickHouse requires at least 16 GiB of memory.(1) The tracing data is indexed by projectId and time, so adding filter conditions on those should significantly improve performance.(1)

2. Migration State Issues: Running migrations manually from the repository could have resulted in inconsistent migration states. ClickHouse migrations marked as "dirty" can prevent proper operation.

3. Web Container Connection Issues: The langfuse-web container may be experiencing intermittent connectivity issues with ClickHouse.(1)

Recommended troubleshooting steps:

• Check your ClickHouse instance has sufficient memory (at least 16 GiB for larger deployments)(1)
• Verify ClickHouse migration status is clean and all migrations completed successfully
• Check langfuse-web container logs for specific ClickHouse connection errors
• Ensure time filters are being applied on queries to improve performance(1)
• Monitor ClickHouse CPU and memory usage when the error occurs

If the issue persists, you may want to verify that your ClickHouse instance meets the recommended specifications and that all migrations completed successfully without being marked as "dirty."

📚 Sources:

1. Scaling

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @rjayaram-godaddy, was inkeep able to help you here?

[rjayaram-godaddy]

@inkeep
You asked me to Verify ClickHouse migration status is clean and all migrations completed successfully
How to check this?

[inkeep-ai-assistant[bot]]

To check if your ClickHouse migration status is clean, you need to query the ClickHouse schema_migrations table.(1)

Check for "dirty" migrations:

Connect to your ClickHouse instance and run:

SELECT * FROM default.schema_migrations WHERE dirty = 1;

(1)

If this query returns any rows, it means your database has been marked as 'dirty' due to a failed migration.(1) When a migration error occurs, the migration tool prevents additional migrations from running on the same database.(1)

What "dirty" means:

A dirty database indicates that a migration was partially applied or interrupted.(1) You'll need to investigate whether the migration was partially applied or not applied at all.(1) After determining the actual state, you should force your database to the version that accurately reflects its current state.(1)

Recovery process if migrations are dirty:

If you find dirty migrations, you need to force the database to the last successful version. For example, if migration 16 failed and you've corrected it, you would force the database to version 15 (the last successful version).(1)

Navigate to the root of your Langfuse project folder and run:

migrate -path migrations/ -database clickhouse://test:test@localhost:8123/dummy force 15

(1)

Make sure to replace the database credentials with your own and adjust the version number to match your last successful migration.(1)

For background migrations:

You can also monitor background migrations within the Langfuse UI by clicking on the Langfuse version tag and selecting "Background Migrations".(2) This shows all migrations that ever ran and their status.(2) You can also monitor progress via the worker container logs.(2)

📚 Sources:

1. Clickhouse handling failed migrations in self-hosted Langfuse
2. Background Migrations

---

Have another question? Just tag @inkeep.

[rjayaram-godaddy]

@inkeep
My langfuse web container logs show this whenever the ui shows that internal error message
`

error tRPC route failed on traces.metrics: Internal error Internal error	2025-11-04T13:36:38.272Z error tRPC route failed on traces.metrics: Internal error Internal error
	2025-11-04T19:06:38.272+05:30TRPCError: Internal error
	2025-11-04T19:06:39.715+05:302025-11-04T13:36:39.715Z error ClickHouse query failed with non-retryable error:
	2025-11-04T19:06:39.715+05:302025-11-04T13:36:39.715Z error middleware intercepted error with code INTERNAL_SERVER_ERROR
`

Is this issue related to clickhouse or some other configuration related to langfuse web?

[inkeep-ai-assistant[bot]]

Based on the error logs showing "ClickHouse query failed with non-retryable error" followed by <html>, this is a ClickHouse-related issue.(1) The <html> response indicates ClickHouse is returning an HTML error page instead of a proper query response, which typically happens when ClickHouse encounters a critical error or is misconfigured.

The error occurs in the tRPC middleware when processing the traces.metrics route.(2)(3) The middleware intercepts the error and converts it to an INTERNAL_SERVER_ERROR.(2)

Primary troubleshooting steps:

1. Check ClickHouse resource allocation: Ensure your ClickHouse instance has at least 2 CPU and 8 GiB Memory as minimum requirements.(4)

2. Verify ClickHouse is responding correctly: Connect directly to your ClickHouse instance and run a simple query to ensure it's operational and not returning HTML error pages.

3. Review ClickHouse logs: Check your ClickHouse container/service logs for specific errors that coincide with the timestamp of the Langfuse web errors.

4. Confirm migration state: Since you ran migrations manually, verify the schema_migrations table shows no dirty migrations (as discussed in the previous response).(1)

5. Check ClickHouse connection settings: Verify that the CLICKHOUSE_URL and related connection environment variables in your langfuse-web container are correctly configured.(5)

The issue is specifically related to ClickHouse not being able to process the metrics query, rather than a general Langfuse web configuration problem.(1)

📚 Sources:

1. Getting Internal Server Error: traces.metrics on langfuse dashboard
2. langfuse/langfuse/web/src/server/api/trpc.ts
3. bug: middleware intercepted error with code INTERNAL_SERVER_ERROR
4. Scaling
5. Deployment Guide (v2)

---

Have another question? Just tag @inkeep.

[Toto0407]

Hi @rjayaram-godaddy did you fix the issue? because currently I'm facing the same after lagefuse update from version 3.112.0 to 3.161.0