The Reality Check

Every augmentation is an amputation. For builders, that means every use of AI adds capability, but it also adds load. A string scan runs in microseconds; a model call that interprets that same string takes seconds. That's roughly a 10,000x gap, and it's not something to design around hoping to optimize away later, it's a given to design with, from the start.

Unfamiliar term below? See the AI Glossary.

Latency is a design input, not an afterthought

Pick a latency target before designing the path, not after:

• Set a P95 target, not an average. Users feel the tail, not the mean, a fast average with a slow P95 still reads as a slow product.
• Match the target to context. Sub-500ms for an interactive UI; a couple of seconds is fine for a background update.
• Assume the model step is the slowest part, and size the rest of the path around that assumption rather than being surprised by it later.

Cut cost with caching and shared infrastructure

• Precompute and cache anything reusable: embeddings for common documents, frequent lookups, common completions, so the model isn't asked for the same answer twice.
• Don't embed a model inside every service. Run it as shared infrastructure instead. One model serving five applications is cheaper and easier to optimize than five separate models each paying their own memory bill.

The 10x tabletop exercise

Run this on paper, before an incident, not during one. Ask what breaks at 10x current traffic: database connections, provider rate limits, queue depth, memory. Ask whether storage and indexes keep up at 10x data volume. Ask whether the product is still worth running at 10x the cost.

For each breaking point, write down two things:

1. The immediate response: raise a rate limit, add caching.
2. The longer fix: the actual architectural change the immediate response is standing in for.

Watch the same signals in production

A small, consistent set of numbers, tracked continuously:

• Latency: what users actually feel.
• User-visible errors: not the internal retries you already handle silently.
• Cost per request: so the system doesn't quietly become unaffordable.
• A quality signal over time: confidence scores or accuracy checks, to catch model drift before users do.

When any of these start moving the wrong way, degrade gracefully: serve cached or last-known-good results, or fall back to a simpler model or a more basic path while recovery happens.

Context windows are still a hard constraint

They keep growing, but the limit is still real. Dropping input to fit the window produces confidently wrong answers, the model can only work with what it actually sees. The fix is a more selective retrieval step: less filler, tidier chunks, only what earns its place in the window. For long-running sessions, summarizing the context window behind the scenes can keep it clean and focused without the user noticing anything changed.

Switching architectures late is expensive, write your exit criteria now

A year in, switching architecture typically costs three to five times the original build. The code rewrite is the easy part. The hard part is migrating the data, retraining the people who learned the old system, and running both systems in parallel through the cutover.

So choose something you can actually live with, or something you can afford to leave. Either way, write down the exit criteria now, at the point of choosing, not once you're already stuck inside the decision and every signal says it's time to go.