- Klaviyo's attributed revenue counts orders from people who opened or clicked within the attribution window. It doesn't tell you which of those orders the email caused.
- A holdout group is a random set of people who get nothing. Compare their revenue per person with the group that got the emails.
- In Klaviyo, you can build a flow holdout with a multi-branch split using a random sample, and a campaign holdout with a random sample of a list or segment.
- Small groups lie. Run the test until both groups have enough orders that one big order can't flip the result.
Why attributed revenue isn't the same as incremental revenue
Klaviyo's help center says a conversion is tracked when someone opens or clicks your message and then places an order within the attribution window. For new accounts after October 9, 2024, the default window is 5 days for email. Apple Mail Privacy Protection auto-opens count by default too, unless you change the setting.
So if a loyal customer was going to reorder on Thursday anyway, and your campaign landed in her inbox on Tuesday and Apple auto-opened it, that order gets credited to the email. That's just what attribution is: credit, not cause.
A common fear founders have about agencies is that they take credit for sales that would have happened anyway. A holdout test is the honest answer to that. I go deeper on the reporting side in Klaviyo attribution.
What is an email incrementality test?
It's a simple experiment with two groups.
- Test group: gets the email or flow like normal.
- Holdout group (or control group): a random set of people who would have gotten it, but get nothing.
Everything else stays the same. Same time period, same site, same ads, same offers. Then you compare how much each group bought per person. If the test group spends more per person, the difference is what the email added.
The word "random" matters most. If you hand-pick the holdout, or hold back your least engaged people, the test is broken before it starts.
How do you set up a holdout group in Klaviyo?
There are two versions, one for flows and one for campaigns.
For a flow. Klaviyo's multi-branch split lets you add a random sample condition to a path, set as a percentage of the profiles evaluated for that path. Put the split right after the trigger. Send 90% down the path with your emails, and send 10% down a path with no emails in it, so they leave the flow without hearing from you. Tag or log who went where so you can compare later.
For campaigns. Klaviyo lets you take a random sample of a list or segment. Its help center puts it under Audience > Lists & segments, then Manage Segment > Sample Segment Members (or Manage List > Sample list members for a list). You type in a sample size and Klaviyo creates a new list. Make that list your holdout, and exclude it from every campaign for the length of the test. Flows keep running for both groups, so you're measuring campaigns on top of flows.
Don't use a flow A/B test for this. Klaviyo's flow A/B test compares two or more versions of an email, and needs at least 2 variations. It picks a winner on click rate or open rate. That's useful for testing subject lines, but it can't tell you what happens when someone gets nothing.
What should you compare?
Compare the whole group, not just the people who opened.
- Revenue per person. Total order value from everyone in the group, divided by the number of people in the group. Include all orders, not just the ones attributed to email.
- Conversion rate. The share of people in each group who placed an order.
- Incremental revenue. The difference in revenue per person, times the number of people who got the emails.
Here's a made-up example to show the math. Say 10,000 people enter your abandoned cart flow in a month and you hold back 10%.
| Group | People | Revenue | Revenue per person |
|---|---|---|---|
| Got the flow | 9,000 | $45,000 | $5.00 |
| Holdout (nothing) | 1,000 | $4,000 | $4.00 |
The flow added about $1.00 per person. Across 9,000 people, that's about $9,000 the flow caused. If Klaviyo credits the flow with more than that, the rest is orders that would have happened anyway. Hypothetical numbers, but that's the shape of the answer you're looking for.
Use revenue per recipient to decide which flows to test first. Across 4 consumable brands, welcome email #1 earned about $4.50 per person in attributed revenue. A holdout is how you'd find out how much of that $4.50 is truly extra.
How long should a holdout test run?
Long enough to cover a full buying cycle, and long enough to collect real orders in both groups.
For a cart or checkout flow, people decide fast, so a few weeks of traffic is often enough. For a welcome flow, give it longer, since some people take a while to make a first order. For a reorder reminder, you need a full reorder window or more. If your product lasts 30 days, a two-week test tells you nothing.
A few timing rules I'd follow:
- Don't stop early because it looks good. Early results swing around.
- Avoid big sale weeks unless that's what you're testing. Black Friday behavior isn't your normal month.
- Don't change the flow mid-test. New copy, new offer, new timing all reset the clock.
How big does the holdout need to be? Sample size in plain English
This is where most tests go wrong. The question isn't how many people are in the holdout. It's how many orders.
If your holdout has 1,000 people and 12 of them buy, one $300 order can swing the result by a lot. You'd be reading noise. You want enough orders in both groups that one or two big orders can't flip the answer.
What that means in practice:
- Bigger flows can use smaller holdouts. If 50,000 people a month hit your browse flow, 10% is plenty.
- Smaller brands need longer tests. Same holdout percentage, more weeks.
- Run it twice. If a second run lands close to the first, you can trust it. If it doesn't, you didn't have enough data.
- Look for big gaps, not small ones. A small difference between the groups on a small test isn't an answer.
There's a cost, too. Everyone in the holdout skipped an email that probably would have made you some money. That's the price of knowing, so keep the holdout small.
Which test should you run first?
Start where the decision is biggest.
- The flow with the largest attributed number. If it's carrying your email revenue, you want to know how much is real.
- A flow you're thinking of cutting. Test before you delete.
- Campaigns as a whole, if you're sending a lot and wondering if it's worth it. In a small sample of 4 consumable brands, 3 saw no rise in unsubscribes when they sent more. A campaign holdout tells you if the extra sends are also earning.
If your account is a mess, fix the basics first. Broken tracking ruins tests. Across my portfolio, about 20 sends showed opens and revenue but zero clicks, which meant click tracking was broken. Run through a Klaviyo audit checklist before you trust any number.
Where these numbers come from
The welcome #1 and unsubscribe numbers come from 4 consumable brands I work with, 2,897 campaigns and $17.7M in email revenue. It's a small sample, so treat them as directional. The cart flow table is made up to show the math.
FAQ
Is email incrementality testing worth it for a small brand?
It can be, but it takes longer. With less traffic, you need more weeks to get enough orders in both groups. If you're under about $25K a month, I'd get your core flows working first and test later.
What percentage should the holdout be?
I'd start at 10%. Go lower on very high-traffic flows, higher if you have a small list and want a faster read. The key is that it's random.
Can I test incrementality with Klaviyo's A/B test feature?
Not directly. Klaviyo's flow A/B test compares versions of an email and needs at least 2 variations. To measure what happens when people get nothing, use a random sample split in the flow, or a random sample list held out of campaigns.
Should the holdout ever be permanent?
Some brands keep a small permanent holdout to keep measuring over time. It's a trade-off. You always know what email adds, but you always give up some revenue from that group.
Founder of Bardown Digital, a Chicago-based Klaviyo email and retention agency for Shopify brands. He's worked with brands doing $500K to $100M a year, with $100M+ in email-attributed revenue across them.