Klaviyo A/B test: how to set it up and pick a winner

This is for Shopify founders who want their tests to actually change what they send next.

What can you A/B test in Klaviyo?

Campaigns. Per Klaviyo's help center, you can test subject lines, message content and send times. Klaviyo says to test only one variable at a time, and for a send time test, the content and subject line have to stay exactly the same.

Flows. You can test any single email inside a flow. Each variation can have its own subject line, preview text, sender and content.

Where should you start testing?

Campaigns and abandonment emails.

Campaigns because that's where most of your sends go, so you get answers fast. Abandonment emails because that's where you learn what gets people across the finish line when they're close to buying. Someone who left a checkout is one decision away from an order, so a test there tells you exactly what moves a ready buyer.

The welcome email is fine to test, but it's fairly standard. And selling in the non-customer branch of a welcome series is a different situation. You don't know how close those people are to placing an order, so a result there is harder to read and harder to reuse elsewhere.

How do you pick a winner in a Klaviyo A/B test?

By the metric that matches what you changed. This is the framework I use:

The mistake I see most is mixing these up. If you run a CTA or content test for clicks, trust the click winner. That's the variable you isolated. Judging that same test on revenue per recipient is a misuse of what you were testing.

Revenue per recipient answers a different question: who should get this email, and how many of them. That's a segment test. When you shrink or tighten a segment, you'll likely see fewer total opens but a higher click rate, and revenue per recipient is how you tell if the trade was worth it. I go deeper on that metric in revenue per recipient.

One note on opens from Klaviyo itself: if more than about 45% of your opens come from Apple Mail, Klaviyo says open rates may be inflated by Apple's Mail Privacy Protection. My take on reading opens and clicks is in open rate vs click rate.

How do you set up a Klaviyo A/B test on a campaign?

Here's the setup, using what Klaviyo describes on its help page:

1. Create the test. Build the campaign, then click Create A/B test above the subject line field. Klaviyo creates a second, identical variation. You can add more, but Klaviyo recommends only 2 variations. Every extra version splits your audience thinner.
2. Change one thing. One subject line. One hero image. One offer. Not all three.
3. Pick the winning metric. The options are open rate, click rate or placed order rate. Klaviyo recommends open rate for subject line, preview text and sender tests, and click rate for content tests, which lines up with the table above.
4. Set the test size and duration. Klaviyo recommends a test size based on your audience size. In its example, a 40% test size sends 20% of recipients version A and 20% version B, picks a winner after 6 hours, and sends the winner to everyone else. You can only change the duration when the test size is under 100%.

If you set the test size to 100%, the whole send splits between the versions and there's no later winner send. Klaviyo notes the reported winner can change as new engagement comes in. You can also pick a winner by hand with Choose as winner on the A/B test results tab, which ends the test and sends that version to anyone still waiting.

How do you A/B test a flow email in Klaviyo?

Flow tests work a little differently. Per Klaviyo's help page:

• Variations: a test starts with 2. You can only add more while the test is in draft.
• Split: you set a weight for each version. Klaviyo generally recommends equal weights, and there's an automatic option that shifts people toward the better performer as the test runs.
• Winning metric: only open rate and click rate are available. Click rate is the default.
• Ending the test: "End test by statistical significance" is on by default. You can also set a date limit, and the test ends at whichever comes first.

One catch: when you add a test to an email that's already live, Klaviyo creates two new copies and stops tracking analytics for the original. Write down the original's numbers before you start so you have a baseline.

How many times should you run a test before trusting it?

Run it across about five campaign sends before you treat it as a rule, and keep it consistent the whole time. Same kind of test, same kind of audience.

Each test send also has to reach a number of people comparable to a regular send. If a test goes to far fewer people than normal, the rates might look higher, and you won't know whether the winner holds up at scale.

Keep a simple log: date, segment, both versions, the metric you judged on, and the winner.

What did my subject line data find?

Across 3,176 Klaviyo campaigns from 9 brands, question subject lines earned about 18% less revenue per email than the same brand's typical email.

My guess at why, and it's a guess, not a fact: question subject lines tend to reach people who are interested but confused about the product or brand and not ready to buy yet. And they usually go out with content that leans the same way.

I don't treat that as a rule for your list. I treat it as a reason to test subject lines on your own list. Subject line patterns flip by audience, and I cover more of them in email subject line best practices.

What should you test first?

1. A subject line test. Run it on campaigns or automations. If you already use different types of subject lines across your automations, you can compare them all at once: download Klaviyo's export file for your flows. It shows where the volume is in your automations, so you know where a subject line test will get answers fastest. Give it at least 100 emails before you make a decision.

2. An offer test. Send 50/50 to your email list, percentage off vs dollars off, over a weekend, and judge it on revenue. If you're rethinking discounts more broadly, see my email discount strategy.

Where these numbers come from

The subject line finding comes from 3,176 Klaviyo campaigns across 9 brands, each measured against that brand's own typical revenue per email. It's a small sample, so treat it as directional. Klaviyo feature details come from Klaviyo's help pages, listed in Sources. For the wider picture, see the CPG email marketing benchmarks.

FAQ

WRITTEN BY Jack White

Founder of Bardown Digital, a Chicago-based Klaviyo email and retention agency for Shopify brands. He's worked with brands doing $500K to $100M a year, with $100M+ in email-attributed revenue across them.

Instagram @jackintheinbox LinkedIn