Feature Store

The shared substrate for ML

Most of our ML services read their features from one place. When that place is slow, everything downstream is slow, so the feature store has to be fast and boring in the best way.

The bottleneck

The original store was in Python. It worked, but its latency made it hard to adopt widely. As more teams wanted in, the tail latencies and overhead started to show, and it could not comfortably carry that load.

Async first, then a Rust rewrite

I took it in two steps.

• First, added proper async support and observability so we could actually see where the time was going.
• Then rewrote the whole thing in Rust as a workspace, with one binary that speaks both HTTP and gRPC, built on top of Google’s API protos.
• The online store serves real time reads and writes through Bigtable, so low latency lookups go straight to the source.
• The offline store lands in Snowflake. Every update flows through a Kafka producer into Kafka Connect, which writes the history teams use for training and analysis.
• Every write path is backpressured, online and offline, so a burst upstream never tips the store over.

Built for teams to actually adopt

Speed only matters if people use it, so I made it easy to get in.

• Self-service and multi-tenant: it spins up a Bigtable table and a Snowflake table per project on first use, so a team onboards without filing a ticket.
• Two front doors: services that want raw speed use gRPC, everyone else uses plain HTTP, both off the same service.
• The numbers are measured, not guessed. Reads are benchmarked with Criterion, and the API runs with OpenTelemetry tracing and Prometheus metrics so its behavior is visible in production.

The payoff

• p99 read latency halved versus the Python version, with p95 reads landing around 8 ms.
• Leaner memory use, so each pod requests well under a gig and still carries its share. Read, write, and gRPC fleets each autoscale from 4 to 20 pods on load.
• Became the default. Over 20 services read from it now at a sustained 900+ QPS, built on it instead of routing around it.

Why it matters

A feature store is infrastructure. The win is that it got fast, steady, and self-service enough to become the thing other teams reach for first, instead of something they work around.