Thinking in Bets for CRO: How to Make Better Decisions When the Data is Incomplete
TL;DR
- Thinking in Bets, Annie Duke's book on decision-making under uncertainty, offers a simple practice: write down what you believe before a test runs, record the evidence behind it, then use the result to update your confidence.
- Pairing Thinking in Bets with the BETS framework turns a single CRO experiment into part of an ongoing decision process.
- A winning or losing CRO test only tells you how one audience behaved in one window, so Edgar Allan treats every result as evidence that updates a belief rather than settles it.
Part 4: Thinking in Bets for CRO: How to Make Better Decisions When the Data is Incomplete
Read Part 3: From Customer Insight to Experiment: Building a CRO Program in Webflow
This is the fourth article in Edgar Allan's five-part series on how we build CRO programs for clients, combining customer research and insights, experimentation, and Webflow implementation under one team.
How much should a winning CRO (Conversion Rate Optimization) test change what you believe about a customer’s behavior?
I picked up Annie Duke's Thinking in BETS after Linda Tong, CEO of Webflow, mentioned it as one of her favorites, and came back to it while we were building out our own approach to CRO programs for B2B SaaS companies. We run that work at Edgar Allan as a Webflow brand, design, development, and performance agency that combines customer research, experimentation, and Webflow implementation under one global team for clients like Island.
The connection showed up pretty quickly.
CRO asks us to make decisions with incomplete information. We have the customer research, analytics, site behavior, sales conversations, and previous experiments, but at some point, we have to decide what we think is happening and put something into the market.
Duke's book looks at decision-making under uncertainty, and one of the ideas she uses is resulting, which is what happens when we judge the quality of a decision from its outcome.
Resulting is the trap of grading a CRO test by its outcome instead of the belief that led to it. A test can win for the wrong reason and still get treated as proof, or lose for reasons that have nothing to do with the underlying customer insight. The discipline against it is recording what you believed, and why, before you see a single result.
CRO gives us plenty of opportunities to do that.
A CRO test result is just one piece of evidence
Let’s say we start with this customer insight:
Visitors who arrive from paid search bounce before scrolling past the hero because the page doesn't confirm the promise in the ad.
From there, we create a homepage variation that echoes the ad’s language directly in the hero, and the variation performs well. Our confidence in the insight should increase, but the question is by how much?
Our experiment tells us how that experience performed with the audience during the test, but there may still be questions about which visitors responded, what part of the change influenced behavior, and whether the pattern continues elsewhere.
This is where Thinking in Bets becomes useful for CRO teams. Duke’s broader argument is that decisions should be evaluated using the information available when the decision was made, while recognizing that uncertainty plays a role in what happens next.
That gives us a better way to look at an experiment.
Write down the bet before you run the test
In the second article of this series, we walked through the BETS framework, which grades a customer belief on a scale from Speculation to Backed as evidence accumulates, and how Edgar Allan uses it in our own CRO practice. Thinking in Bets adds a companion practice to that process: capturing where a belief sits on that scale before you know how the test performs.
This doesn’t need to become a giant experiment brief; the point is to capture the thinking while the outcome is still unknown.
Once you know which variation won the test, your memory of the original decision can get surprisingly flexible. Duke's work on resulting is useful here because knowing the outcome can affect how we evaluate the thinking that came before it.
Use the test result to update your confidence
I’ve started thinking about CRO experiments as a way to change our confidence in a customer belief.
Imagine this begins as a theory:
Returning enterprise visitors are looking for customer proof points from companies like theirs.
Research gives us some support, and we bring relevant customer evidence into the experience earlier and run the experiment.
Now we have another piece of information, so maybe the belief moves toward being an educated guess.
The test result may also make the insight more specific. Perhaps company size matters, or maybe the behavior is stronger for visitors coming from a particular campaign. That refines the belief we track on the BETS board and points to a sharper question for the next test.
A flat result works in the same way.
If moving the proof earlier doesn’t change the behavior, go back to the customer insight and decide what the result tells you about your confidence. The belief may need more research, the audience may need to be narrower, or the experience may not have expressed the insight clearly enough, but either way, that result becomes part of the evidence behind the next decision.
Decide what evidence would change your mind before choosing a metric
We used this kind of thinking in our CRO pilot with Island, an enterprise B2B SaaS company where a single metric like form submissions wouldn’t have told the whole story.
While form submissions did matter, they only captured one part of how people used an enterprise B2B site. We also measured CTA engagement, hero interaction, scroll depth, movement into other areas of the site, downloads, and other behaviors around the experience.
That gave us more context when looking at what happened.
Two homepage variations might produce the same number of form submissions while sending visitors into product content in very different ways. Depending on the belief behind the experiment, that second signal, not the form count, might be the one that actually matters.
So before choosing the metric, ask:
What evidence would change our confidence in this belief?
Then decide what you need to measure.
Keep the decision record
After the experiment, add two fields to the original bet:
Result: What happened?
Updated belief: What do we think now?
That gives the CRO program some memory and allows the team to see why an experiment existed, what evidence supported it, and how the result changed the original customer insight.
For me, that’s the connection between Thinking in Bets, the BETS framework, and CRO.
CRO means making decisions with incomplete information, and the useful discipline is keeping track of what we believed, why we believed it, and how new evidence should change what we do next.
That’s the discipline we run at Edgar Allan every time a CRO test comes back, whether the result is a clear win or nothing at all.
Next: How mature is your Webflow CRO program?
FAQs
How does Edgar Allan decide how much a single CRO test result should change what it believes about customers?
Edgar Allan, a Webflow design, development, and performance agency, treats each CRO test result as one piece of evidence, not proof. Before running a test, we write down the customer insight, its confidence level, and what evidence would need to appear to change our mind. After the test, we update that confidence up or down rather than treating a single win or loss as final, because the test only tells us how one audience behaved during one experiment.
What is "resulting" and why does it matter when reading an A/B test?
Resulting is judging the quality of a decision by its outcome rather than by the reasoning behind it, a concept from Annie Duke's Thinking in Bets. In CRO, this shows up when a winning test gets treated as proof the original hypothesis was correct, even if the win came from an unrelated factor. Guarding against resulting means recording the belief and its supporting evidence before the test runs, so the original reasoning can be checked against the result honestly.
Should a business change its strategy after one winning A/B test?
Not entirely. A winning test increases confidence in the underlying customer belief, but it answers a narrower question than it appears to: how did this specific variation perform, with this audience, during this window. Questions about which segment responded, what part of the change mattered, and whether the effect holds elsewhere usually remain open, so a single win should shift confidence incrementally rather than settle the belief outright.
What should a CRO team do when a test result is flat or shows no change?
A flat result is still evidence, not a failure to act on. It should prompt a return to the original customer insight to ask whether the belief needs more research, the audience was too broad, or the experience didn't express the insight clearly enough. Treating a flat result as new information, rather than discarding it, keeps it useful for the next hypothesis instead of wasting the test.