What AI Can Actually Do

We know AI learns patterns from data, so what can it actually do with those patterns? Start with what most developers hit first: language models.

Language models are predictive text, scaled up

When you use Claude, GPT, or any language model, it isn't reading your text the way you do. Think of predictive text on your phone: type "How are" and it suggests "you." That's the same fundamental mechanism, just scaled up massively, whether it's predicting one word or generating whole paragraphs, it's pattern matching at different scales.

Tokens and context windows

A token is roughly 4 characters, a word, or part of one ("understanding" might be "under" + "standing"). The model sees everything as tokens, not words or sentences.

Models have a hard limit on how many tokens they can hold at once, the context window. It's grown fast (a bit over 4,000 → 8,000 → 100,000 → 200,000+), but the limit is still real: try feeding a 50-page document, or holding a long conversation, and you'll hit it. This is why chat models "forget" earlier turns, not forgetfulness, a fundamental limit. The window keeps growing and workarounds keep improving, but right now it's still a real constraint to design around.

Embeddings: turning meaning into numbers

Everything, text, images, anything, has to become numbers for AI to work with it. "King" becomes a list of numbers; "queen" becomes a different list. These numbers capture relationships: the classic example is KING − MAN + WOMAN = QUEEN, which works because the numerical representations capture semantic relationships. This is why a search for "car" also surfaces "automobile" and "vehicle", not because the model understands transportation, but because the number patterns are similar.

Preprocessing: the unglamorous time sink

Models are picky about input format, and this quietly eats a huge share of dev time:

• An image model trained on 224×224 pixels needs exactly 224×224, not 223, not 225.
• Wrong color format (RGB where grayscale is expected) fails outright.
• Wrong text encoding fails outright.

Raw data almost never works as-is. Expect to spend as much time formatting data for the model as integrating the model itself, that's normal, not a sign you're doing something wrong.

Pattern matching vs. understanding

This is the line that explains both AI's best tricks and its strangest failures:

• It can write Shakespeare-quality prose, explain quantum mechanics (without understanding it), and generate working code.
• It struggles to count the letters in "strawberry" without using a tool.
• It'll get 2 + 2 right, because that pattern appears a million times in training data, but ask 2.01 + 1.99 and it's a coin flip.

None of this is a bug. It's the difference between pattern matching and actual computation. Once that clicks, a lot of LLM failure modes become predictable: unreliable character counting, hit-or-miss arithmetic, logic puzzles that break them, and confidently wrong facts, not lies, just word patterns that were statistically likely given the training data.

Vision models: same idea, different failure shape

To a human, a cat is a cat whether it's sleeping, jumping, or half-hidden. To AI, those can be completely different pattern sets, rotate a cat 45°, and it might as well be a different animal, because the pixel patterns changed.

Vision preprocessing is even stricter than text: exact resolutions (512×512 means exactly that), matching color channels, and normalization (pixel values scaled to a specific range) all have to line up or the input fails. The failures are predictable too, rotate an image and accuracy drops; partially cover an object and the model gets confused. This predictability is also what makes adversarial attacks possible: changing a handful of pixels, invisible to a human eye, can make a model see a stop sign as a speed-limit sign, or a rifle as a turtle, because it was never seeing objects, only numbers.

The cost/capability/speed tradeoff

Every new model is more capable, and also slower and more expensive. Simpler models are faster and cheaper, but noticeably less capable, true for vision models too. Picking a model is really picking a tradeoff:

• Fast, "good enough" models for something like product search, where instant results matter more than perfect matches.
• Slow, accurate models for something like the actual purchase step, where precision matters.

Better accuracy costs more compute, which costs more money and more latency, a live tradeoff to manage deliberately, not a problem that gets solved once and forgotten.

Multi-modal: one model, many formats

Text, images, code, and audio used to need separate models. Increasingly they don't, the most capable models can read a PDF, look at a chart and produce a table, or look at a table and produce a chart, all from one model.

The throughline

Every format, text, image, audio, gets converted to numbers, and those numbers get matched against learned patterns. That single idea explains the impressive capabilities and the bizarre failures: human-like text generation works because the model learned those patterns; simple counting fails because that's computation, not pattern matching. Give the model a tool that can actually compute (like running code), and it can suddenly do the thing it couldn't do natively, so "it can't do X today" isn't necessarily true tomorrow.

Understanding these capabilities and limits matters because they directly shape how a model makes decisions, which is next: how does AI go from patterns to actual outputs? How does it choose what to generate, classify, or predict?