Evaluating Your Options

If you've been building long enough, you've been burned, vaporware you bought into, a framework whose problems never got fixed, API costs you underestimated. Those scars are useful: they help you spot trouble early with AI, whether you're building, buying, or renting.

Red flags to listen for

• A build plan promising delivery in 6 months. Not a sign of efficiency, a sign the team hasn't done this before.
• Pricing "per interaction" without defining what an interaction is.
• "Our solution handles everything." The biggest red flag of all.
• Vague pricing. "Contact us for enterprise pricing" usually means they'll charge whatever they think you'll pay. Usage-based pricing without clear metrics means you can't budget, and with AI, usage can explode: one bad loop, one eager PM enabling a feature for everyone, and a monthly bill jumps from hundreds to tens of thousands. "Flexible" pricing often just means they haven't figured it out yet.
• How they talk about failure. If it never comes up, they haven't run anything in production. Teams that have deployed AI tell it to you straight, the model that produced garbage at 2am, the cost spike that jumped 100x overnight, the competent-sounding response that created a customer service nightmare.

The real cost math

Building it yourself: half a million in training costs is just a start. Add at least two senior ML engineers ($250–300K each), GPU infrastructure (~$10K/month for something small, but realistic training loads can hit $50–200K/month), and someone to keep it all running. Best case, over $1M in year one; more realistically $2–5M for something competitive.

APIs: an assistant averaging 2,000 tokens/query (context + response) at $2/million tokens, handling 100,000 queries/month, runs about $400/month in base API costs, closer to $850 once you add rate limiting, fallback models, and monitoring. Manageable and predictable.

Vendor packages: a reasonable-looking $10,000/month ($120K/year) needs inspection. What are you getting beyond API access? If it's compliance, support, SLAs, or integration, maybe worth it. If it's prompt engineering wrapped in a nice UI, probably not.

Beyond cost: production behavior

Once you understand costs, look at how the system actually behaves in production:

• What's the P95 latency: the response time 95% of users experience when things are slow?
• Can they show real error logs from last week?
• What happens when the model produces something wrong, because it will? Is there a plan?

Good providers answer directly: dashboards, failover strategies, rate limits, no hedging. If the conversation shifts away when you ask, that's your signal to go elsewhere.

The bottom line

Evaluate on real costs over time, production metrics, not demo videos, and a clear failure plan. Whether you're looking at API pricing, a vendor package, or your own build estimate, that's what tells you what you're actually signing up for.