Operational Reality

Most teams monitor too much and still learn too little. Fifty graphs and twenty alerts, and when something breaks nobody can say why. The fix isn't more instrumentation, it's a small set of signals that each answer a question you'll actually ask at 2am.

Unfamiliar term below? See the AI Glossary.

The five signals worth having

Signal	Why this one, for AI specifically
P95 latency at the user boundary (P95 is the worst case for the routine 95%, the boundary, not the typical experience.)	Catches provider trouble. Averages hide provider trouble. The mean looks fine while 5% of users wait far too long, and that 5% is who complains. this is waht people actaully feels.
User-visible errors	Not every retry. Only the failures that reached someone. Retries are noise until they stop working
Cost per request	Catches prompt bloat Context creeps up a little at a time until you're paying ten times per query and nobody changed anything deliberately. Check max tokens per request
One quality signal over time: e.g. a confidence-score distribution	Catches slow degradation. Answers look fine at a glance and get less useful week over week. Nothing else in your stack will tell you this
Queue depth or backlog age	Tells you the data path is falling behind before users notice. Documents becoming chunks becoming embeddings is where it stops keeping pace

Measure at the user boundary, not inside the system. A component can be healthy while the thing the customer waits on is not.

Capacity arrives in steps, not a straight line

This is the part that surprises people who've scaled ordinary web services.

You're comfortable at 100 requests a minute. At 200, cache limits are hit and queues stay deep, and suddenly you need three times the resources rather than twice. Past that you hit the provider's rate limit and latency spikes for everyone at once, including the users doing nothing unusual.

So find the first constraint deliberately. Load-test until something gives, and note which thing gave:

• rate limits
• memory constraints
• queue depth limits.

Production will find it for you otherwise, at a time of its choosing.

Plan the degradation before you need it

The question isn't whether the primary path fails. It's what the system does in the ten minutes afterwards.

• Primary model down → fall back to a smaller or older one. Worse answers beat no answers.
• Real-time overloaded → offer batch. Slower, but it completes.
• Everything under pressure → shed the lowest-priority traffic, having decided in advance which that is.

Put each of these behind a feature flag. The fastest fix is the one that needs no deploy, and during an incident, "we can turn that off" is worth more than any amount of cleverness.

Write "When AI Goes Wrong" on one page

Three columns, filled in before anything is on fire:

Symptom	Check	Action
High latency	Provider status, rate limits, queue depth	Trip the circuit breaker, shed batch traffic
Bad answers	Recent prompt changes, model version, retrieval quality	Roll back the prompt, pin the model, enable fallback, clear the cache
Cost spike	Context size, retry loops, a runaway job	Cap tokens, kill the job, disable the feature by flag

Then rehearse it quarterly. The worst time to learn how the runbook works is the first time you need it, and a page nobody has read is decoration.

Keep it boring

Queues you already know. Databases you trust. Frameworks your team can debug. The clever architecture that saves 10% on compute is a poor trade the first Saturday night you spend inside it.

Boring scales. Boring is debuggable. Boring lets you sleep.

None of this is extra work bolted on at the end, it's the difference between a demo and something that stays up.