Pop-up A/B Testing: Elements That Move the Needle
The four elements worth testing in order of effect size, and how long a pop-up test needs to run.
On this page
The short version
- Four elements have effects large enough to resolve on an ordinary amount of traffic. Everything else is smaller than the noise.
- Test the trigger first. It moves the number more than the offer, the copy or the design, and it is the one most people never touch.
- Measure signups per session, not per impression. A change that shows the pop-up to fewer people will always look worse per impression and may be better overall.
Pop-ups are unusually good candidates for testing, because they generate impressions quickly and the effects being looked for are large. That is the opposite of email A/B testing, where the sample is fixed and the effects are small.
The risk is different too: it is easy to run a test that raises signups and costs you readers, because only one of those appears in the result.
The denominator decides what you learn
Per impression is the default in every tool and it is the wrong basis for anything that changes how often the pop-up appears.
Move the trigger from five seconds to sixty percent scroll and impressions fall sharply. Conversion per impression rises, because the remaining impressions land on engaged readers. Both numbers move for the same reason and neither tells you whether you got more subscribers.
Signups per session is the number that answers the question. It holds the audience constant and lets impressions vary, which is exactly the right way round when the thing under test is when to show it.
The four elements worth your traffic
| Element | Effect | Measure on |
|---|---|---|
| Trigger and timing | Largest | Signups per session |
| The offer itself | Large | Signups per session, plus 90-day engagement |
| Format (overlay, slide-in, bar) | Large | Signups per session, plus return visits |
| Headline | Moderate | Conversion per impression |
| Button copy | Small | Conversion per impression |
| Field count | Small — there should be one | Conversion per impression |
| Colour | Below the noise | Do not bother |
| Image choice | Below the noise | Do not bother |
How long to run one
Long enough to cover a full weekly cycle, minimum, because weekday and weekend traffic behave differently and a test that ran Tuesday to Thursday has sampled one kind of visitor.
Two weeks is a reasonable default for most sites. If your traffic is seasonal or campaign-driven, run it across a period that includes both the spike and the baseline, or split the analysis by source.
And decide the stopping point before starting. Pop-up tests generate impressions fast enough that watching them makes it very easy to stop at the moment one variant happens to lead.
Testing the trigger without a tool that supports it
Most pop-up tools will split-test the content and not the trigger, which is unfortunate given the trigger is the largest lever.
The workable substitute is sequential: run configuration A for two weeks, configuration B for two weeks, compare signups per session. It is not a controlled test — traffic mix changes between periods — but for an effect this large it resolves, and it is available to everyone.
Run it twice in alternating order if you can. A-B-A-B removes most of the seasonality objection at the cost of another month.
Write down the losses
A test that produced no difference is a result, and it is the one nobody records. The consequence is that the same test gets proposed again a year later by someone who was not there, and runs for another fortnight to reach the same conclusion.
Keep a single page: what was tested, over what period, what the numbers were, and what was decided. Four lines per test. It takes a minute and it is the only thing that stops a small team relitigating settled questions.
It also protects the good decisions. A change that raised signups and was reversed because it hurt return visits will look like an obvious win to whoever finds it next, unless the reason it was reversed is written down beside it.
Pop-up test check
- The metric is signups per session
- Return visit rate is tracked alongside
- The trigger has been tested, not just the copy
- The run covers at least one full weekly cycle
- The stopping point was decided in advance
- Only one thing differs between variants
- 90-day engagement is checked for any offer change
Frequently asked questions
How much traffic do I need?
Far less than for an email test, because the effects are larger. A few thousand sessions per variant will resolve a trigger or format change; copy changes need considerably more.
Can I test two offers at once?
You can, and each visitor should still only meet one. Splitting visitors between offers is a test; showing both to everyone is two interruptions.
The tool says a variant won. Should I trust it?
Check what it measured and when it decided. Most declare on conversion per impression, which is the wrong denominator for anything that changes how often the pop-up fires.