How to A/B Test Cold Emails: What to Test and How to Read Results

Guessing which cold email works? ColdMailer lets you split a list, send two variants over the inboxes you own through your own SMTP email sender, and compare replies without touching a spreadsheet. Start free.

To A/B test a cold email, change one variable at a time, send each version to at least 200 prospects from the same list, run it for one to two weeks, and pick the winner by reply rate, not open rate. Test in order of leverage: subject line first, then the opening line, then the call to action, then sequence length. Anything that changes two things at once tells you nothing, because you cannot say which change moved the number.

That is the whole method in one paragraph. The detail below is about not fooling yourself, which is where most cold email tests go wrong.

What to test, in order of leverage

Not every element is worth a test. Work top down, because the earlier a reader drops off, the more a fix is worth. A great call to action is useless if nobody opened the email.

• Subject line. Decides whether the email gets opened at all, so it is the highest-leverage test. Try short (under 50 characters) versus longer, a question versus a statement, and a personalization token like company name versus none.
• Opening line. The first sentence decides whether they keep reading. Test a prospect-specific observation against a generic value statement. This is where AI email personalization software earns its keep, because a relevant first line is what separates a reply from a delete.
• Call to action. Test a soft ask ("worth a quick look?") against a specific one ("open to a 15-minute call Thursday?"). Interest-based CTAs usually beat calendar asks on a first touch.
• Sequence length and timing. Once the copy is dialed in, test three follow-ups versus five, or a two-day gap versus four.

Test one of these at a time. If your A and B versions differ in the subject line and the CTA, a lift could come from either, and you have learned nothing you can repeat.

How many emails do you need per variant?

Send each version to at least 200 prospects before you trust the result. For small lifts (under about 15 percent), you need 500 or more per variant to separate a real difference from noise. Below 200 sends, a "winner" is usually just random variation, and you will chase a subject line that was never actually better.

Keep the two groups comparable. Split one list randomly rather than testing variant A on one industry and variant B on another, or the audience difference, not your copy, drives the result.

How long should you run a cold email A/B test?

Run it for one to two weeks. Replies to cold email trickle in over days, not minutes, and people read on different schedules, so a test you call after 24 hours misses most of the signal. Let each variant reach its full sample and give follow-ups time to land before you declare a winner.

Which metric decides the winner? (Not open rate)

Judge the test on reply rate, and ideally positive reply rate, not open rate. Apple Mail Privacy Protection and similar features preload tracking pixels, which registers an "open" even when no human read the email. That inflates open rate and makes it unreliable for deciding which subject line actually worked. A subject line that wins on opens but loses on replies is the wrong winner.

Here is how the common cold email metrics rank for A/B testing decisions:

How to read the results without fooling yourself

A difference is only worth acting on if it is both large and built on enough sends. A 4.0 percent reply rate versus 3.8 percent across 150 emails each is noise. The same gap across 600 emails each starts to mean something. When two variants land within a point of each other on a few hundred sends, treat it as a tie, keep either version, and go test a bigger lever instead of splitting hairs.

When you do find a clear winner, make it the new control and test the next element against it. That is how testing compounds: each round locks in a small gain and raises the floor for the next test. Treat it the same way you would tighten the copy and layout on a landing page, one change at a time, measured against the version it replaced.

Common cold email A/B testing mistakes

• Changing two things at once. You cannot attribute the result. One variable per test.
• Calling it too early. Under 200 sends per variant is a coin flip dressed up as data.
• Deciding on opens. Inflated and unreliable. Reply rate decides.
• Testing on mismatched lists. If the audiences differ, the audience is your variable, not the copy.
• Never restarting. A winner today is a control to beat tomorrow, not a permanent answer.

Frequently asked questions

What should I A/B test first in a cold email?

Test the subject line first. It decides whether the email is opened at all, so it is the highest-leverage element. Once you have a subject line that earns opens, move to the opening line, then the call to action, then the follow-up sequence.

How many emails do I need for a valid A/B test?

Send each variant to at least 200 prospects, and 500 or more per variant if you are trying to detect a small lift under about 15 percent. Below 200, the result is usually random variation rather than a real difference between the two versions.

Should I use open rate to pick the winner?

No. Open rate is inflated by privacy features that preload tracking pixels and register opens no human triggered. Use reply rate, or better, positive reply rate, to decide which variant actually performed. Treat open rate as a rough directional read only.

How long should a cold email A/B test run?

Run each test for one to two weeks. Cold email replies arrive over several days as people check inboxes on different schedules, and follow-ups need time to send, so a shorter window misses most of the response and skews the result.

Once a subject line and opener are winning, the next lever is usually relevance at scale. See how to personalize cold emails at scale and the benchmarks in our cold email reply rate guide to know what a good result looks like.