LLM Gateway

One door to every model

The gateway is how the whole company talks to LLMs, more than 200k requests a day. One API sits in front of many providers and our own self-hosted models, so teams do not each wire up their own keys, logging, and limits.

Building on a solid base

When I came in, it was already a working proxy on LiteLLM, routing requests reliably. The opportunity was to grow it into a real platform, with deeper audit logging, guardrails, budgets, and support for more than just text. That is where I spent my time.

What I worked on

I took it well beyond the initial proof of concept and built a lot of it out. The pieces I focused on:

• Audit and observability: every request and response streams to Kafka and lands in Snowflake, with artifacts in GCS, so usage is fully traceable.
• Guardrails at the edge: pre-call checks for PII, secrets, and invisible text, so risky prompts get caught before they ever leave.
• Per-user budgets and cost tracking, so spend is visible and capped.
• Reach across modalities: live realtime audio, multi-modal image, text to speech, transcription, and embeddings.
• Batch processing support, with end-to-end examples that work no matter which framework a team uses, plus a CLI so they can fire off large jobs without writing their own glue code.

Why it matters

It grew from a proxy into a platform. Teams get one safe, observable, governed way to use any model, and the platform team gets a single place to see and control all LLM usage.