The Processing Pattern Evolution

In production, your processing pattern decides latency, cost, and how painful debugging becomes. The progression below, sequential, then parallel, then distributed, isn't messy; it's maturity. Each step is a response to a specific symptom, not a default to reach for early.

Start sequential

One request in, one request out. Simple and easy to reason about, use it while each request finishes before the next one starts and users aren't left waiting.

Signs it's time to change: requests start to overlap, queue depth grows, and response time climbs faster than traffic. AI makes this arrive sooner than usual: the sequential pattern that worked fine for database lookups falls apart once each request takes 5–10 seconds for a model call.

Move to parallel processing

Handle more than one request at a time, but make repeated writes safe first. If the same action happens twice, data shouldn't end up duplicated or corrupted:

• Put shared data behind clear guards.
• Use transactions, a single writer per key, or a small queue so only one update happens at a time.

Watch your metrics. If CPU rises but throughput stays flat, you're contending, not scaling. If high CPU or growing lock waits persist under normal load, simple parallelism isn't enough.

Move to distributed processing

When contention persists after writes are safe, spread the work across machines, but don't just add servers at random. Split the work into logical groups (by customer, document type, or region) so each group operates independently. Think of it like checkout lanes at a store: dedicated lanes that move smoothly beat everyone crowding into one.

This is partitioning: the key is keeping work that belongs together on the same partition. Otherwise you're constantly reaching across the network to complete a single request, which defeats the purpose.

Real-time vs. batch

Alongside all this, keep one question in mind: who's waiting?

• If a person is watching a screen, keep it real-time and keep it short.
• If nobody's watching, reports, analytics, batch it and run it on a schedule.

Moving work off the user path isn't giving up; it's being smart about where to spend your latency budget.

The sequence

Start simple. Add concurrency when you see overlap. Distribute when contention remains after writes are made safe. Batch anything the user doesn't need immediately.