Production Realities

Given everything covered so far, pattern matching, processing limits, probability-based decisions, what actually works once AI is in production?

What works

• Pattern matching within known distributions. A model trained on 5 years of credit card transactions, applied to transactions that still look similar, works well.
• High-volume classification with stable patterns: fraud detection, garbage filtering, content moderation, works because the underlying patterns don't shift quickly.
• Recommendation systems work because people with similar click patterns tend to click similar things. It's correlation at scale, and that's enough for most use cases, Netflix doesn't need to understand why you like dystopian fiction, just that you fit a pattern.
• Document processing with consistent formats: OCR on standard forms, invoice extraction where vendors share a layout, contract analysis where the language follows patterns. The key word is consistent.

What doesn't

• Reasoning about novel situations is hit or miss. A customer service bot handles "reset my password" fine, that's a pattern. It can't handle "here's my genuinely unique situation", that needs reasoning, not pattern matching.
• Guaranteed accuracy is impossible. Whatever a vendor promises, if a use case needs 100% accuracy, AI is the wrong tool, not pessimism, just the nature of probabilistic systems.
• Self-correction is a myth. A model doesn't learn from its production mistakes. Wrong today means wrong again tomorrow, unless you retrain or tune it, there's no feedback loop unless you build one yourself.

Deployment choices, each with real trade-offs

• APIs (OpenAI, Anthropic, etc.), lowest barrier to entry, but you inherit their uptime, rate limits, and costs.
• Embedded models: more control, but real infrastructure to run and maintain.
• Edge deployment: runs on-device, but caps how large the model can be.

Monitoring AI isn't like monitoring normal systems

Uptime isn't enough. You also need:

• Prediction drift: do production patterns still match training data?
• Accuracy decay: is performance degrading over time?
• Tail latency, not just averages. P95 = 95% of requests finish within X; P99 = 99% do. A P95 of 200ms with a P99 of 2000ms means 1 in 100 requests takes 10x longer than typical, and at thousands of requests per minute, that's dozens of frustrated users an hour, hidden behind an average that looks fine.
• Correctness, not just uptime. A model can be up and responding while being completely wrong, dashboards don't catch that.

Deploying changes, not just code

You can't roll out a model swap the way you roll out code:

• Shadow mode: run the new model alongside the old one and compare results before switching.
• Canary deployments: send 1% of traffic to the new model and watch for problems.

Passing your tests doesn't mean production is fine, a swap can go live, tests green, and customer complaints spike 400%+ because the model was working, just working badly.

Costs add up faster than expected

Every conversation, every image analyzed, costs money, and it compounds at scale. One real example: a team running agents burned almost $1,000 of tokens in a single workday. Ways to manage it:

• Temperature = 0 for more predictable (and often cheaper) output.
• Token limits on API calls.
• Caching repeated queries: one e-commerce site cut costs nearly in half by caching product description enhancements.
• Batch vs. real-time: batch processing is roughly 10x cheaper but adds latency; real-time is responsive but expensive. Most teams start real-time, hit the bill, then redesign for batch.
• Indirect costs: data pipelines, result storage, human review for edge cases. The AI API cost itself is often the smallest part of the total bill.

The POC-to-production gap is real

A proof of concept ignores rate limits; production can't. A POC gets clean data; production gets messy reality. "AI-first architecture" sounds great until the AI service goes down and takes the whole system with it, which means you need fallbacks and graceful degradation: AI handles 90% of cases, humans (or simple rule-based fallbacks) handle the rest. Have an actual plan for "what happens when this goes down," not just an assumption it won't.

Vendor lock-in

Every provider has different APIs, different behaviors, different quirks, GPT and Claude don't behave the same way. Switching providers later means rewriting prompts, retesting behavior, retraining your team. Plan for an abstraction layer early, or pay for its absence later.

Security gets weird with AI

• Prompt injection: manipulating prompts to break the system or leak information it shouldn't expose.
• Model extraction: competitors reconstructing your fine-tuned model through carefully crafted queries.
• PII leakage: a model accidentally surfacing private training data (a real GDPR exposure). Most security teams haven't seen these attack vectors yet, get ahead of it rather than discovering it in an incident.

Integration is always harder than it looks

These aren't APIs returning predictable JSON, they return probabilistic text that needs parsing, validation, and error handling. The service itself can go down, rate-limit, or change behavior without warning. Your architecture has to expect all of it.

The takeaway

Success with AI in production isn't really about the technology, it's about understanding its constraints and building around them: good architecture, quality data, real operational discipline, and thinking it through before jumping in.