Is it better to spend extra time and make sure you're right before committing to something or is it better to take a risk, save the time and be okay with failure? My Uncle used to say "Close only counts for horseshoes and hand grenades". Context is important though as is risk tolerance.
With Foundry we approached this problem as a risk problem. Are we worried that a potential winning variant will be pruned due to bad data? This is a big fear in the old hat world of A/B testing and ensuring statistical confidence verifies assumptions but today the fear is unnecessary.
Our learning loop and constant iteration understands what is and isn't working and the language is tailored to the context of the site, specific known content strategies and whatever additional helpful information you may have included in the case of a personalization.
We chose a greedy, aggressive algorithm compared to modern standards because we think the risk of wasting paid ad traffic on low performing copy outweighs the risk of having a potential winner get pruned. The copy will come back around and the system will realign as the data compounds. This same concept applies to false positives as well. Bad copy that gets a few easy conversions will quickly get outpaced by copy that actually works.
It is more than just that though. It's not just about what gets pruned but it's also about promoting what IS WORKING. If something is working we don't want to wait 4 weeks to shift the majority of the traffic to it like traditional A/B testing. We want the traffic to go there now. 4 weeks is a lot of wasted traffic and clicks.
The Math of Regret (A/B vs. Greedy)
Let's look at the numbers. In data science, we call this "Regret Minimization" - measuring how many conversions you regret not getting because you were busy "testing."
Imagine you spend $5,000 to send 2,000 visitors to a landing page.
- Headline A (Control): Converts at 2%.
- Headline B (Winner): Converts at 4%.
Scenario 1: The Traditional A/B Test
You split traffic 50/50 for the entire campaign to reach 95% statistical significance.
- 1,000 people see the loser (2%) = 20 Sales.
- 1,000 people see the winner (4%) = 40 Sales.
- Total Sales: 60.
Scenario 2: The Greedy Algorithm (Foundry)
Our algorithm spots the winner early (say, after the first 200 visitors) and aggressively shifts 90% of the remaining traffic to the winner.
- The Shift: Instead of a 50/50 split, ~1,800 people now see the Winner.
- The Result: You get closer to 72 Sales from the same traffic.
The ROI Difference
You got 20% more customers for the exact same ad spend.
In Scenario 1, you paid for the privilege of being "statistically sure" that Headline A sucked. In Scenario 2, you just made money. We're not even accounting for the learning loop and optimization curve in this scenario which will potentially improve conversion rate even further.
We pursue performance and ROAS over everything else because we can. It enables the full cycle of Promote → Prune → Iterate to work fast like your ads. We realized at some point while dealing with the wide variance on PMax campaigns that this was the way. User interest can change quickly, your copy and ad matching should be able to keep up. Stagnant, long running A/B tests are cash burners by nature.
Do you agree the risk is worth the payoff? We'd love to hear from you.