The demo was free. The month after the demo is where AI projects quietly die or quietly earn. Three numbers to watch, where the money leaks, and a weekend setup that pays for itself.
The first project is easy: prompt, model, wow. The surprise arrives around week six, when three "small demos" are running in production, nobody can say what they cost last month, and an unannounced model update has changed how two of them answer. The layer that prevents this has an unfashionable name. Observability. Dashboards, logs, and a number on a wall that someone checks weekly.
The three numbers that matter
Cost per real outcome. Not per token. Per answered ticket, per booking, per completed lesson. Tokens are the fuel gauge; outcomes are the odometer, and a car with a perfect fuel gauge and no odometer is how every AI overspend story starts. Divide last month's spend by this month's finished work and put that number where a human sees it.
Latency at the p95, not the average. Your customers never feel the average. They feel the one call in twenty that took forty seconds, and they feel it publicly. A proxy that logs every call is already measuring this; set the alarm on the slow tail and complaints stop being news.
Quality over time. Keep twenty or thirty questions with known-good answers and re-run them weekly against whatever the current model is. Models update under you without asking. Sometimes the update fixes your edge case. Sometimes it eats a rule your whole workflow depended on. The eval finds out in four minutes; customer emails find out slower and cost more.
Where the money actually leaks
The long system prompt, multiplied by every call, including the calls that never needed it. The chatty agent that re-reads the same document four times to answer one question. The frontier model doing a small model's job; asked honestly, most "we need the bigger model" stories are retrieval problems or prompt problems wearing an invoice. And caching, which is free money: the cheapest token is the one you already paid for, and most providers now hand you that discount explicitly if you structure calls around it. Every one of these leaks is visible in the logs the moment you have logs.
The weekend setup
Route every call through one proxy that records model, tokens, latency and a trace id per request. Point a simple dashboard at it, the open-source options are good. Build the eval sheet this week, not next quarter. The setup costs a Saturday, and the first surprise it kills pays for the rest of the year. A boring layer with a boring name, and the difference between an AI project with margins and an AI project with a story.
Kit for this lesson
A proxy, a dashboard and a weekly eval loop on one box with 128GB of memory, which means you stop renting a server just to watch the money. It runs the small models that lower the bill too.
View on Amazon
Boring and true: the observability box dies in a power cut, the logs stop, and the mystery month begins. A pure sine wave UPS is the cheapest insurance in the rack.
View on Amazon
Self-hosting the eval model wants memory more than graphics. 96GB of DDR5 gets a 70B quantised model onto consumer hardware and makes the weekly quality check cost nothing.
View on AmazonAs an Amazon Associate, AI Universe City earns from qualifying purchases. Prices link to Amazon and may change; the picks above are what we actually use in the Labs.