How AI Actually Makes Decisions

Start with a misconception: when people say "AI decides," it isn't deciding the way a person does. It's calculating probabilities based on patterns, closer to weather prediction than judgment. A model sees pressure, temperature, humidity patterns; it doesn't understand that low pressure means storms, it just knows those patterns correlate with rain in the training data. Correlation without causation: that's how AI "decides."

Different types of models decide in fundamentally different ways.

Generative models: temperature and sampling

For models like Claude or Gemini, output is shaped by temperature and a few sampling parameters:

• Temperature = 0 → close to deterministic. Ask the same question 10 times, get nearly identical answers.
• Temperature closer to 1 → more variety, which reads as creativity, different phrasings, different starting points, sampled from a broader range of probable outputs.
• Top-k: only consider the k most likely next tokens (e.g. top 50 of 1,000 candidates).
• Top-p: only consider tokens up to a cumulative probability (e.g. the smallest set that adds up to 90%).

These parameters get tweaked blindly more often than they're actually understood. A customer service bot wants low temperature, consistent, reliable responses. A creative writing assistant wants higher temperature, variety and surprise. Same model, different parameters, completely different behavior.

Classification models: the threshold is the decision

For fraud detection, credit scoring, medical diagnostics, the model outputs a probability, not a decision. "75% likely to be fraud." The model doesn't decide whether to block the transaction, you do, by setting the threshold (block above 70%? 85%?).

That threshold is a balance between two costs:

• Block too many legitimate transactions → lose real customers trying to buy groceries, pay bills.
• Let too much fraud through → lose money, and victims lose more.

AI doesn't weigh those costs, it just returns a number. The ethical judgment is yours, not the model's.

Confidence is not accuracy

A model saying "99% confident this is fraud" does not mean it's right 99% of the time. It means the input strongly matches a learned pattern, if the model learned the wrong patterns, it can be extremely confident and extremely wrong. This shows up in credit risk models that claim 90% confidence on default predictions, then turn out right only ~70% of the time when checked against actual outcomes. That gap breaks downstream risk pricing and automated decisions, test confidence against real accuracy before trusting it in production.

Explainability: the wall that kills projects

"Why did the AI reject this loan application?" Legal wants to know, compliance needs documentation, the customer wants an answer, and most models are black boxes. They can tell you the decision, not the why. Post-hoc explanation tools exist ("the model weighted income highly"), but these are approximations, not true explanations. Sometimes explainability requirements are strict enough that AI simply can't be used for that problem, decide whether a decision needs to be explainable before reaching for AI, not after.

Feedback loops

Threat-model this the way you would security: ask how it could go wrong before you build it. AI decisions create new data, and that data shapes future behavior, which sounds fine until it becomes a self-fulfilling prophecy:

• A recommendation system shows what it thinks users want; users click those recommendations; the system "learns" they like those things and shows more of the same, until everyone converges on the same narrow set of content.
• A credit model rejects applicants with a certain profile; those people go elsewhere, so you never learn whether they'd have been good customers; the model keeps rejecting similar profiles, locking out entire customer segments.

These loops don't just let a model degrade, they actively reinforce its own biases over time.

Chained models: errors compound

Chain three models at 90% accuracy each, and the combined accuracy isn't 90%, it's 0.9 × 0.9 × 0.9 = 72.9%. Three steps, a 27% error rate. Each step adds noise; five steps deep, the output can be largely random. Chaining models to seem "smarter together" often does the opposite: uncertainty compounds.

Deterministic and unpredictable, at once

Same input → same output makes a model deterministic. But nobody fully knows what patterns it learned, so the output can't be predicted in advance, that's unpredictable. This tension is why prompt engineering exists at all: reverse-engineering learned patterns to trigger the output you want, like navigating a maze blindfolded by memorizing turns that worked before.

Boundaries and human oversight aren't optional

Given probability-based decisions, confidence/accuracy gaps, and feedback loops, AI cannot be deployed without boundaries:

• Business rules that take over when a suggestion falls outside acceptable ranges.
• Human review on every high-stakes decision.
• Monitoring to catch drift when the model starts to stray.

Every successful production AI deployment shares one thing: the team understood the limitations, used the model where it excelled, and put guardrails where it didn't.

The takeaway

AI decision-making is probability calculations over learned patterns, whether it's choosing the next word or flagging a transaction as fraud, it's pattern matching turned into percentages. Not reasoning, not understanding, statistical correlation at massive scale. Once that's internalized, it gets much easier to judge where AI genuinely helps and where it doesn't.

Next: given all these limitations, where does AI actually deliver value in production?