Langfuse Eval Observability

From black box to glass box

LLM apps are hard to trust when you cannot see inside them. Langfuse gives the GenAI stack one place to trace every model call, track tokens and cost, run evals, and debug agent behavior. I built the platform path that runs it self-hosted inside Shipt, split into a web role for UI and trace ingest and a worker role for background jobs.

See everything

• Traces and scores land in a self-hosted ClickHouse cluster, one shard across three replicas, so every node holds a full copy and the cluster keeps serving when a node drops.
• Coordination runs on a separate ClickHouse Keeper tier, its own three-node StatefulSet, which replaces ZooKeeper. Both tiers run the same image and just start with a different command.
• Storage is split by shape: Postgres holds the transactional side like projects, users, and prompts, while ClickHouse takes the high-volume, append-only trace and score data, each store matched to the work it is good at.
• A GCS bucket is the event queue between the web and worker roles, with Valkey for queue and cache traffic, pinned to noeviction so a backlog never drops a trace.
• Langfuse ships no alerting, so I built a small FastAPI exporter that turns its Metrics API into Prometheus gauges: latency p50/p95/p99, tokens, cost, errors, and eval scores. New agents show up on their own, nothing hardcoded.

Getting the cluster to boot

My first cut ran Keeper embedded in each ClickHouse pod, one process doing both jobs. It kept deadlocking on startup. With Keeper baked into the server, a pod could not come up until it reached its peers, but every peer was blocked on the same wait, so the rollout stalled on the first pod and the cluster never formed. Every deploy turned into babysitting.

So I componentized it: pull Keeper out of the servers and run the keepers as their own tier. Now each tier boots on its own terms. The keepers come up independently and form Raft quorum among themselves, and the ClickHouse servers connect to Keeper lazily, so a server is healthy the moment it is up, even before Keeper answers. One image for both tiers, just a different entrypoint per role. The cluster comes up on its own now.

Gate every change

The goal here is leverage, not one-off tests. I put a process and a set of wrappers around Promptfoo that make end-to-end agent testing a CI/CD step any team can adopt. Plug an agent in and it gets gated on task accuracy, answer grounding from RAG, off-topic handling, and prompt injection, mixing latency and regex assertions with LLM rubric judges run through the gateway. Bad changes get caught in the pipeline before they reach a production workflow.

Why it matters

This turns LLM platform work from guesswork into an observable, gated system. Teams can see traces, latency, cost, and errors, and trust that agent quality is checked before changes ship.