Making It Work in Production

Every path, build, buy, or rent, has to work in production. What behaves smoothly in development changes under real traffic. The point isn't to avoid that; it's to be ready. The pattern below walks through turning a single naive API call into something that holds up: shorten the user-facing path, shield the dependencies, then observe the system.

The naive starting point

One request goes to the model, one comes back. It ships quickly and works fine, until a launch or seasonal spike hits rate limits. Requests return 429s, timeouts climb, and users wait.

Shorten the user-facing path

queues and workers Move slow model calls off the web request using a queue and a worker: the page answers immediately, and the worker finishes the model call in the background.

Streaming St Where possible, enable streaming so users see output as it's generated instead of watching a spinner.

Shield the dependencies

• Circuit breaker: a smart switch that stops calling a service that's repeatedly failing, so it fails fast instead of letting threads pile up waiting for timeouts.
• Caching: sits behind the API so common questions don't trigger identical compute, at orders of magnitude less cost per call.
• Retries with backoff and jitter: random delays so clients don't all retry at once and create another spike.

The larger design choices

Every option is a trade-off, not a free win:

• Running models locally keeps data on your system but runs slower.
• Calling cloud APIs scales easily but moves data over the network and lives within someone else's quotas.
• RAG (retrieval-augmented generation) gives the model relevant documents to reference, it works well when retrieval is correct, but a wrong retrieval leaves the model confident and still wrong.
• Vector databases speed up finding relevant content, but they're another service to run and fix when it breaks, make sure you actually need one.

There's no perfect setup, only a balance. Each situation is unique; model and plan the trade-offs before you start building.

AI fails quietly

Traditional systems fail with errors. AI can return output that looks valid but is totally wrong, quality can drift slowly with no obvious signal, and context windows can drop information without warning when overloaded. Handle this by watching and observing, not guessing:

• P95 latency: what users actually feel.
• User-visible errors: not internal retries.
• Cost per request: so the system doesn't quietly become unaffordable.
• A quality signal over time: confidence scores or a lightweight accuracy check, to catch silent degradation before users report it.

When things go wrong, degrade gracefully: serve cached answers, fall back to a simpler model, or route users to a basic path while the main path recovers.

Where to start

If you only make one or two changes this week: move the slow model call off the request path and put a circuit breaker in front of it. Then add caching for repeat questions. After that, start watching P95 latency, user-visible error rate, cost per request, and one quality signal. That sequence turns a demo into something that survives production.

These aren't optional patterns, they're the difference between a brief disruption and a long outage. These conditions will happen. Teams that handle them are the ones that expected them from the start.