Speech-to-Text Serving

Self-hosted speech to text

A REST API that turns audio into text by running Whisper on our own infrastructure instead of calling a paid API. It is how the platform does transcription, including for the LLM gateway.

How a request flows

I built the whole service in Rust on top of whisper.cpp. A request walks a tight, predictable path:

• Auth at the door with a bearer token, then accept uploads up to 200 MB.
• Decode almost any audio format with Symphonia and downmix to mono.
• Resample to the 16 kHz Whisper expects with a Rubato FFT resampler.
• Pass through a semaphore that caps how many transcriptions run at once, so a box stays healthy under load instead of thrashing.
• Run inference through whisper.cpp, then return json, verbose json, srt, vtt, or plain text so it drops straight into existing tooling.

It also handles many languages with auto detection, plus a translate to English mode.

Why Rust over the Python version

The earlier path was Python on the reference Whisper. Moving to Rust on whisper.cpp bought us a few real wins:

• Faster inference. whisper.cpp runs the same model weights through quantized, CPU-tuned kernels. On CPU that lands around 3-4x faster than the PyTorch reference on the tiny model with 30s clips, at the same accuracy.
• Lighter on memory. whisper.cpp uses quantized weights and skips the whole PyTorch runtime, so the service runs in roughly half the RAM of the Python reference. It also loads a single WhisperContext once and shares it across every concurrent request, while a GIL-bound Python service has to fork a worker per request, each holding its own full copy of the model.
• A lean runtime. It ships as one static binary in a slim container. No interpreter, no heavy ML runtime, faster cold starts when a pod comes up.

Running it in production

It runs on GCP, fronted by Axum and Tokio, and autoscales from 1 to 10 replicas on CPU. It handles graceful shutdown so in-flight transcriptions are not dropped on a deploy, and the same endpoint serves tiny through large-v3-turbo models just by swapping a config value.

Why it matters

Transcription we own. No per minute bill, no audio leaving our walls, and one fast endpoint the rest of the platform can lean on. Faster inference on lighter pods means we serve the same load on roughly a third of the compute the Python version would need, so the running cost drops with it.