Hey there 👋, I’m Bryan
Welcome to my newsletter. I share practical advice and tackle tough questions on putting AI to work. Thank’s for being here! Send me your questions and in return I’ll humbly offer actionable real-talk advice 🤜🤛
Not long ago, organizations incentivized software engineers by the volume of code they wrote. Unsurprisingly, when performance was judged on volume, the result was bloated repos, low-quality output, and engineers who looked busy while the codebase quietly decayed. In other words, organizations got exactly what they measured.
Organizations rolling out AI today are at risk of making a similar mistake. The AI adoption mandate is the modern equivalent of grading on lines of code. This time, however, AI lets anyone produce an abundance of material fast, which means anyone can produce lots of junk fast, then wave it around as proof of “adoption.” Once quality is out of the equation, all that’s left is activity. At worst, people start gaming the metrics.
When adoption becomes the goal
A growing part of everyone’s job is cleaning up the AI slop from people trying to operate outside their expertise. Shopify’s CEO, Tobias Lütke, calls these “slop grenades.” Yes, people have cleaned up work slop forever, and AI slop is just the newest flavor, but the difference now is that AI slop is hollow while presenting as confident. Oh, and there’s exponentially more of it, which makes it impossible to review everything.
In the race to look AI-forward, behaviors like “token-maxxing” and “budgeting for tokens” made their way to the C-suite and into the mainstream conversation. Amazon employees built an internal token-usage leaderboard that backfired, as people routed unnecessary work through AI to inflate their scores. Amazon shut it down, citing the compute costs it generated, and a senior VP told staff to stop using AI just for the sake of using AI. Many companies seeing similar behavior are responding with usage limits and token budgets.
The fallacy of budgeting for tokens
On the surface, budgeting for tokens makes sense. Tokens cost money, so it’s only logical to budget for them. If an organization’s end goal with AI is to hand its workforce a chatbot and throttle usage, then yes, budgeting for tokens is probably the right approach. Organizations with ambitions beyond adoption and chatbot usage will need to evolve their thinking to match how AI agents and automations actually show up in the enterprise.
Traditional software is budgeted by the seat, because its cost grows with the number of people using it. Chatbots are budgeted by the token, because their cost grows with how much people use them. Agents take action, whether that’s issuing a refund, updating the CRM, or processing a claim, so their cost grows with the work they do. A refund issued correctly is worth something, whereas a refund issued wrong costs you money, however few tokens used. That’s why the unit of measure has to move from the token to the task.
So when it comes to ROI, where does cost per token fit?
Cost per token is an input, and inputs make poor scorecards. Yes, every task is still made of tokens, and token price belongs in the math, but it shouldn’t be the number you manage to.
Imagine an insurance workflow
Instead of asking, “how much does this model cost per million tokens?” or “how many millions of tokens, and at what rate, did we pay last month?” ask, “how much does it cost to process one claim correctly?”
Then decompose the workflow into tasks:
Intake: cheap model
Document interpretation: stronger model
Fraud anomaly detection: specialized model
Decision recommendation: frontier model
High-stakes exception: human
Each step gets the level of intelligence it needs and no more. Now suppose someone decides to save money by moving document interpretation to a cheaper model. The token bill goes down, but the cheaper model might misread more documents, so more claims get flagged for human review. Because one human review can cost more than every model in the workflow combined, the token budget shows a win while the cost per claim goes up.
Where this is headed
Here’s what I’ve come to believe will define mature AI-native operations. Companies will stop standardizing on a single model, or even a single model provider, and workflow steps will be routed to the least expensive level of intelligence that still clears the quality bar. Microsoft is already building for this. Its Foundry model router reads each prompt and sends it to one of dozens of models from OpenAI, Anthropic, Meta, and others. Microsoft 365 Copilot now runs on models from both OpenAI and Anthropic, and this month Microsoft added the newest from each. Once routing is the norm, the price per token stops telling you much.
Humans already work this way
Nobody asks a VP to handle every administrative chore just because they’re more capable than a director. Similarly, AI-native organizations won’t run every task through their most expensive model either. Model routing is organizational design, and because agents take action, model choices become part of how the business operates.
Nobody measures an employee in words per dollar. So why would we measure AI that way?
Cost per token
↓
Cost per task
↓
Cost per workflow
↓
Cost per outcome
Each step down the stack gets closer to an actual business case for AI. A model that’s 3x more expensive per token can be cheaper per completed workflow if it uses fewer steps, needs fewer retries and requires less human cleanup.
How to start
Pick one workflow that ends in a clear outcome.
Find out what it costs today. Before any AI, what does one completed outcome cost in people’s time and tools? This is the baseline.
Break it into tasks and set a quality bar for each step.
Assign each task the right model (or person) that clears the quality bar.
Measure cost per completed outcome, including any human intervention, and compare it to the baseline. The difference is the real cost (or savings) of AI.
Beyond the token
Most organizations starting an AI transformation want to grow beyond chatbots and usage dashboards. This is a marathon, cost per token is the starting line, and cost per outcome is the finish. What organizations are really paying for is the outcome. Organizations that measure what they actually want may be surprised how much of the slop was never worth the tokens in the first place.
Thanks for reading!
Sincerely,
Bryan 👋
PS - Let me know in the comments if you found this newsletter helpful. Your feedback would mean the world to me 🙏
Know someone that would benefit from this newsletter? Feel free to forward this to them.