Matching Architecture to Your Problem

No single architecture handles every workload well. Map the problem first, then choose a pattern: keeping context in mind throughout: what the model needs to see, how you fetch it, and how you spend that limited window.

Language tasks

• Problem: Language tasks

• Context: Log misses, track failures

• Pattern: Group and categorize

• Add retrieval first (so the model has gthe right context, the right docs) OR

• Custom training: Only for repeated gaps with clear examples that fix them

Start with a baseline: run real prompts for a few days, then log the misses. Group them, terms the model doesn't know, private facts it couldn't see, prompt issues. Add retrieval first so the model sees the right context. Only consider custom training when you can point to repeated gaps with examples that fix them, write down the specific failures you expect fine-tuning to solve. If you can't list them, you're probably not ready to train.

Structured decisions

For simple yes/no or small-set classifications:

• begin with rules you can explain and trace.
• Implement a "why" trail so each decision records its reason
• log the cases rules miss or misclassify.
• Only then add a small model for the leftover cases, and keep the decision path auditable: show the rule's reason when a rule fires, the model's reason when the model is used, and a threshold you can tune without a new release.

RAG and search

Basically, giving the model relevant documents to reference

• Keep it literal at first
• ship keyword search and measure where it fails, queries with no results, users reformulating the same question repeatedly.
• Add semantic search, or search by meaning, when keywords miss wording variations. If top results are still noisy, re-rank using what users actually click, or a lightweight model.
• Set a schedule to refresh the index so new content is searchable without manual steps.

Treat context as a budget

Every token you send costs money and fills the model's limited attention, spend it on the task, not boilerplate:

• Keep the system prompt short, remove duplicates, compress long quotes.
• Keep chunks small with a little overlap so sentences aren't cut in half.
• Order matters:
  • task first
  • then recent conversation
  • then the most relevant facts
  • then everything else. Models tend to pay more attention to what comes early and late, less to what's in the middle, that may change, but it's the current reality.
• Always cite sources so people can verify what the model used.

The one-sentence fit test

State in plain English what you chose and why: "We used APIs for standard language tasks so costs stay predictable," or "we added retrieval because the answers needed internal docs," or "we trained a small model because our domain language wasn't covered and we had labeled examples."

If you can't say it that plainly, the match probably isn't right.