Why AI integration is different - Content

Sort the same list six times and a normal system returns the exact same result each time. AI breaks that assumption. The same input can come back with different wording, different formatting, or a shape you didn't ask for, plain text grows Markdown or HTML because the model learned those patterns in training; a JSON request for answer and sources comes back with numbers as strings, an extra field, or the whole thing wrapped in backticks. The code didn't change. The behavior did.

That's the actual difference from a normal integration: the contract can shift while you're using it. When a database returns malformed data, that's a bug. When AI returns a slightly different format, it's expected, you're not fixing errors, you're handling normal variation. Plan for that from the start rather than treating each drift as a one-off surprise.

Unfamiliar term below? See the AI Glossary.

Validate every response, then fall back deliberately

The simplest version that actually works, applied on every request:

1. Validate what comes back. Expecting JSON, parse it.

2. If parsing fails, try the small fixes:

   • strip code-block wrapping
   • fix single quotes vs double quotes issues
   • trailing commas
   • unescaped characters.

3. If it still doesn't parse, stop trying and take the fallback path. A generic response, a simpler model, or a human review queue, the specific fallback matters less than having one planned before it's needed live.

Lean on the provider's constraints, not your own prompt. Fix things before they become bugs

• Use the provider's JSON mode when it's offered. Built-in format constraints are more reliable than asking nicely in a system prompt.
• Pin model versions where you can. Where you can't, run a startup check, send a test prompt, verify the response still matches the expected shape, and log when the format drifts so the pattern is visible.
• Normalize at the edge. Encoding, timezones, units, fix them on the way in, before they get the chance to become bugs further down the pipeline.

Version the contract when the response shape changes

Accept both the old and new formats temporarily, log which one is actually being served, and cut over cleanly once the data shows it's safe to.

Make drift visible before the customer does

• Tag every request with an ID that follows it from the browser, through the model, and back, so when something looks wrong, the exact input and output are traceable. The request/response trail is what makes auditing possible at all.

• Keep a small set of test prompts and run them daily. If the responses start changing shape, different formatting, different field names, it shows up in the morning check, not in a customer complaint. Catching it there is strictly cheaper than catching it in support.

Predictable Failure at the Edges - Boundary

The drift above is about the content of a response. This is about the boundary, the seam between your app and the model. These failures aren't unusual cases; they're the ordinary consequences of putting a slow, metered, expensive call inside a request lifecycle designed for fast, cheap ones.

The single idea: a model call breaks four assumptions your stack already makes:

- about time
- about quota
- about payload size
- and about being called once.

Each one has a standard fix.

Time, the call is 20 seconds, your endpoint expects one

If you leave the call on the request path -> the user may very possibly hit a timeout

The web tier times out while the model finishes in the background with nowhere to send the result.

Solutions

• Move the call to a worker behind a queue and return immediately with something the user can see.
• Stream if the API supports it. Time to first token is what the user actually experiences.
• Set timeouts per tier: short on the web tier, long on the worker tier, so each layer fails the way you intended.

Quota, one shared budget means one batch job takes everybody down

A batch job burning most of the quota produces 429s for interactive users who did nothing wrong.

• Separate the workloads that compete for quota. Interactive traffic (a human is waiting), background batch (embedding runs, backfills) and internal use (evals, CI, dev machines) should not draw on the same budget. A 50k-document backfill at 2am eats the minute's token allowance, and the customer typing at 2:01 gets a 429, nothing was wrong with either workload, they just shared one bucket with no priority between them.

  Minting a second API key doesn't achieve this. Limits are enforced per organisation or per project, not per key, so two keys in the same project share one ceiling. What actually separates them:

  • Separate projects or organisations, each with its own allocated limit, the provider does the enforcing for you.
  • Your own gateway in front of the provider: a single path all traffic goes through that knows the class of each request and queues or sheds by priority. Worth having anyway: it's the difference between graceful degradation and raw 429s reaching users.
  • The provider's batch API where one exists, cheaper, hours instead of seconds, and outside your interactive quota entirely.

• Retry with backoff and jitter so clients don't resynchronise into a second wave.

• Decide in advance which traffic wins when you're near the limit. That's a product decision, and it gets made badly under pressure.

Data paths - Volume

One document becomes many chunks, then tokens, then embeddings

Counts and payload sizes multiply at every hop, and queues sized for small web events start to struggle.

• Cap it early and hard: max document size, max chunks per document, max tokens per request.
• Add backpressure so workers slow their intake instead of crashing.
• Compress large payloads, and keep raw inputs only as long as replaying a failed batch requires.

Repetition

A brower refresh, a retry and a cron job can all schedule the same work

• Tag work with a stable request ID or content hash and treat repeats as the same job.
• Make the write idempotent: running it twice changes nothing, and you don't pay twice.

And don't let the slowest component hold the page

Render the base page immediately and fill in the AI sections as they arrive, updating in place over server-sent events or websockets. If a result is late or unavailable, show a basic or cached answer rather than blocking the whole surface/app/page.

The through-line: handle these inside the normal request lifecycle, queues, limiters, caps, idempotency keys, progressive rendering. They're all pre-existing patterns. Nothing here is specific to AI except how reliably it triggers them.