Hey there 👋, I’m Bryan
Welcome to my newsletter. I share practical advice and tackle tough questions on putting AI to work. Thank’s for being here! Send me your questions and in return I’ll humbly offer actionable real-talk advice 🤜🤛
In my earlier article on building an intake for AI projects, I covered how to get your arms around all the AI opportunities in your organization. You survey your employees and, best-case scenario, you get a great response. The next challenge is deciding which opportunities deserve your company’s time, money, and attention.
Your marketing team wants to automate a workflow. A vendor has a new feature that sounds promising. An executive wants to build an enterprise “brain.” Someone simply wants help drafting emails.
Now all of these projects are on the same list, waiting for you to decide what happens next. The trap? A hundred ideas can easily become a hundred distractions if you don’t know how to evaluate and prioritize them.
“Am I allowed to experiment?”
This is the first question your employees need answered. If the answer requires a meeting with three departments, you’ve probably made everyday experimentation too difficult.
The trick is to implement enough governance to make good investment decisions while leaving room for people to experiment and innovate. An employee organizing their meeting notes shouldn’t be subject to the same approval process as a team connecting AI tools to your customer database. Your process needs to make that distinction.
I would give employees permission to explore when all of the following are true:
They’re using an approved tool within its permitted uses.
They aren’t using sensitive or protected data.
The experiment stays within their own work and doesn’t affect a shared process.
It requires no additional spending or dedicated resources from another team.
The output is internal, and a human checks it before using it.
The tool cannot independently send messages, change shared records, or take other consequential actions.
Anything outside those boundaries should have a clear review process. That includes new tools, additional spending, engineering support, customer-facing uses, sensitive data, and changes that affect other people’s work. For example, an employee using an approved assistant to organize their own non-sensitive notes could fall comfortably within those boundaries. Connecting that assistant to a shared system and having it assign work to colleagues would require approval.
This first step should take a meaningful portion of your inventory out of the approval queue. Those ideas should still remain visible so others can learn from them, but they don’t need a committee’s attention.
Understand what kind of work you’re funding
Once you’ve separated everyday experimentation from initiatives that need review, organize the remaining ideas into a few useful categories.
I’d start with four:
Work with AI: Help employees do their work, such as preparing presentations, researching questions, or drafting documents.
Offer with AI: Improve the products and services customers receive, such as adding an AI assistant to an existing service.
Operate with AI: Improve how the organization runs, such as coordinating an intake process or automating a shared workflow.
Build with AI: Help technical teams design, develop, test, and maintain software and other technical solutions.
Some initiatives will fit more than one category. As a general guideline, put each initiative in the category that best reflects its primary purpose, and move on. The categories are there to help you make decisions, not slow down your process.
Then add a second dimension: experiment, pilot, or scale. These describe the stages of an initiative and highlight what you are being asked to approve next. Here’s a set of examples of how the two dimensions come together:
Worth noting: initiatives in the experiment stage here can still require formal approval. Using sensitive information or connecting to a live system doesn’t make it harmless just because someone calls it a test. Likewise, an expensive engineering pilot may carry relatively little privacy risk.
The category tells you where you’re investing, while the stage tells you what you’re committing to next. The risks determine which safeguards and reviews are necessary.
Decide what your organization values
Questions to consider before anyone starts scoring initiatives:
What are your organization’s strategic goals?
Does your team agree on the outcomes your entire AI portfolio should support and the balance of investment you want across the four categories?
How much capacity should go toward immediate productivity gains?
What resources are available to support new customer offerings or changes to how the business operates?
Your organization’s strategy may be focused on growth, customer experience, operating costs, clinical outcomes, something very specific, or something else entirely. This is generally a moving target than shifts year to year. Those priorities should shape how you evaluate AI initiatives, not the other way around. You may receive dozens of suggestions for personal productivity and very few for a difficult enterprise problem that deserves major resource investment and attention.
I like a weighted scorecard because it makes your priorities explicit. Here’s a starting point:
The first three criteria describe the opportunity’s alignment and value. The remaining four describe what resources it will take to deliver.
These weights give the criterion related to strategic alignment and business value a combined 45% of the score, and that’s intentional. Even so, the delivery criteria account for roughly half the total, which can favor inexpensive, fast, straightforward projects. Test whether the resulting rankings reflect your strategy. Otherwise, you may fill your portfolio with small improvements while repeatedly postponing the work that could transform the business.
Using the scorecard
Each criteria on the scorecard gets a value between 1 and 5, but before you start tossing out numbers, define with your team what each rating means. For example, a business value rating of 1 could mean the benefit is below a certain threshold or not yet measurable. A rating of 3 might mean a meaningful, measurable improvement to a single department or workflow. A score of 5 could then mean substantial financial benefit across the enterprise. Once you’ve agreed on rating definitions it’s time for the fun part.
Let’s get scoring! Rate each criterion from 1 to 5, with 5 representing the most favorable result. An easy way to remember this is that high business value earns a high score. High cost does not. An initiative that requires less investment should score more favorably on investment required, and an initiative with manageable risks should score more favorably on risk.
Optional enhancements to your process:
As an exercise to test your scorecard, take 3 to 5 real initiatives and score them against each other. Where people disagree, ask why. You’ll quickly discover whether the definitions are clear enough to use.
Request a short explanation and supporting evidence for each rating. Also, consider including a confidence measure as an additional dimension. A measured result from a pilot deserves different confidence than a vendor’s promise.
Time to calculate! Each criterion’s weight determines its available points. Business value, for example, has 25% weight and can contribute 25 points. Divide the rating by five, then multiply by the available points for that criterion. For example, a business-value rating of 4 earns 20 of the available 25 points. Add the seven results for a total out of 100.
The example below shows an initiative with strong strategic alignment and business value, but substantial investment requirements.
Hot tips 🔥
Calibrate investment thresholds to your organization and use a consistent cost horizon. A $100,000 initiative means something different to a small business than it does to a national enterprise.
Remember to consider vendor fees, internal labor, integration, infrastructure, maintenance, and ongoing usage costs.
Account time for training, workflow redesign, and the effort required to get people using the solution. An inexpensive tool can still require an expensive organizational change.
For time to value, define what a meaningful benefit looks like and estimate how long it takes to produce. Launching a pilot is a milestone, but it doesn’t establish that anyone’s work has improved, or that those improvements have showed up in the bottom line.
Once the rubric has been calibrated, keep it consistent for that prioritization cycle, and revisit it deliberately as strategy and experience change.
As a final note, keep mandatory approval requirements outside the score. Clinical safety, privacy, security, regulatory compliance, and vendor due diligence deserve their own reviews. This prevents an initiative from offsetting unacceptable risks by earning extra points elsewhere.
Decide when you’ll stop before you start
The scoring is complete, and now you have a ranking system to help improve the quality of your decisions. Hooray! This next part is often overlooked.
Most of us have experienced working on a project far longer than we should have, maybe even knowing it was a dead end. This is because organizations spend a lot of time defining how projects get approved and less time defining when projects should end. Planning for the end is not a silver bullet, but it can help ease the tension that often accompanies shutting down a project (letting go is hard!).
Before an experiment or pilot begins, agree on:
What it must demonstrate.
How you’ll measure success.
Resources you’re willing to invest.
Which problems would require an immediate pause.
What is your review date, and who will decide whether to expand, revise, or stop the work.
Suppose an AI tool reduces the time employees spend drafting something, but increases the time others spend checking and correcting it. You need to know whether the overall workflow improved. Measuring only the drafting step could give you a very flattering answer to the wrong question.
When the review date arrives, ask whether the evidence justifies the next investment. If it doesn’t, stopping is a useful outcome. You’ve learned something while limiting the commitment, and you can put those resources toward a more promising idea.
That decision gets harder after people have invested months of effort and attached their reputations to the project. Agreeing on the criteria upfront gives everyone a fairer basis for the conversation.
Recap: What will you commit to?
Remember:
A useful AI portfolio should help you explain why you’re funding one initiative, why another needs more evidence, and why a third can proceed without anyone waiting for approval.
Employees should know where they have permission to explore. Teams should know what it takes to earn investment. Leaders should know what evidence would make them change course.
Now when your next meeting opens with a list of 100 AI ideas, you’re prepared to ask: what do we value as an organization? which of these ideas are we prepared to commit people and resources to, and what do we need to learn before committing more?
That’s the conversation that turns a growing inventory into strategic decisions.
Thanks for reading!
Sincerely,
Bryan 👋
PS - Let me know in the comments if you found this newsletter helpful. Your feedback would mean the world to me 🙏
Know someone that would benefit from this newsletter? Feel free to forward this to them.