A/B testing ad creatives: generating variants that teach you something
Many creative tests end with "B got a slightly higher CTR" and no idea why. That happens when the variants differ in several ways at once, the test stops too early or it measures the wrong thing. This guide covers how to choose what to test, how much data you need (probably more than you think), how to run the test in Google Ads and how to make variants that differ in only one thing.
Key takeaways
- Change one thing per test. If the variants differ in two ways, you can't tell which one caused the result.
- Test big things first: offer and message, then the visual idea, then the headline and CTA. Leave colors for last.
- At a 0.5% CTR, detecting a 20% lift takes about 86,000 impressions per variant.
- Choose the winner by conversions or cost per acquisition, not only by CTR.
Contents
Why most creative tests teach you nothing
| Mistake | What goes wrong | Fix |
|---|---|---|
| Several changes at once | B wins, but you don't know which change did it | One variable per test |
| Stopping on a good day | You pick a winner that isn't really better | Set the sample size in advance |
| Optimizing for clicks | A curiosity headline brings in visitors who don't buy | Decide by conversions or CPA |
| Letting the platform decide | One variant gets 90% of impressions on day one | Rotate evenly or use Experiments |
Any figure that looks interesting or different is usually wrong.
Twyman's law, a rule of thumb from data analysis that experimentation teams often quote
Keep this in mind when an early result shows a 40% lift. It's far more likely to be noise or a tracking error than a real discovery. Check the tracking and wait for the planned sample before you believe it.
What to test, in order of impact
- Offer. "50% off the first box" vs. "free grinder with your first order". This often makes the biggest difference.
- Message or angle. Saving time vs. saving money vs. quality. It's the same product described from a different side.
- Visual idea. Product close-up vs. a scene with the product in use vs. an abstract background.
- Headline wording, for the same message.
- CTA wording. "Start free trial" vs. "Try it free".
- Layout, colors and logo placement. The differences here are usually small. Test these only when you have a lot of traffic.
Write the hypothesis first
Write down what you expect before the test starts. It keeps the test about one change and stops you from explaining the result afterwards to fit whatever happened.
Because [what we observed or believe],
changing [one element] from [A] to [B]
will increase [primary metric] for [audience/placement].
We'll call it after [N impressions or N conversions per variant].For example: "Because our buyers say they compare prices, changing the headline from 'Fresh beans every week' to 'First box 50% off' will increase conversion rate for prospecting audiences. We'll call it after 100 conversions per variant."
Pick the metric that matches the goal
| Metric | What it tells you | Use it when |
|---|---|---|
| CTR | Whether the banner gets noticed and clicked | You're testing visuals or headlines and conversions are too few to compare |
| Conversion rate | Whether the people who click do what you want | You're testing offers or messages that set an expectation |
| Cost per acquisition | What a result costs in the end | You have enough conversions and the variants get different bids or placements |
Decide by conversion rate or CPA, and use CTR as a check. If a variant gets more clicks but fewer conversions, its message probably promises something the landing page doesn't deliver.
How much data you actually need
Display CTRs are often well below 1%, so small differences need a lot of impressions to prove. The chart shows how many impressions each variant needs, starting from a 0.5% CTR, at 95% confidence and 80% power:
The calculation behind the 20% bar uses the standard two-proportion formula:
n ≈ (1.96 + 0.84)² × (0.005 × 0.995 + 0.006 × 0.994) ÷ (0.001)² ≈ 86,000
Two things follow from it:
- Test big differences. A test of button colors on a normal budget rarely finishes. A new offer or a new visual idea has a real chance of showing a difference you can measure.
- Run tests long enough. Run each test for at least one full week, and preferably two, so weekday and weekend behavior both count. Set the stopping point before the test starts.
For conversion-rate tests, count conversions, not impressions. A common rule of thumb is to wait for at least 100 conversions per variant before you trust a difference.
Running the test in Google Ads
| Setup | How | Good for |
|---|---|---|
| Same ad group | Variants side by side. If the campaign has an ad rotation setting, choose "Do not optimize: rotate ads indefinitely" | Headline, CTA and visual tests on uploaded banners |
| Experiments | Split traffic between the original campaign and a copy with the new creatives | New visual ideas, bigger changes |
| Asset ratings | Responsive display and PMax rate each asset (Low, Good, Best) | Deciding which assets to keep. This is not a controlled test |
Whichever setup you use, don't change bids, audiences or budget while the test runs. Any change that affects one variant more than the other makes the result unreliable.
Making clean variants
A clean test needs variants that are the same except for the one thing you're testing. If you generate each variant from scratch, that's hard: change the headline and the AI also gives you a new background. In AdflowKit you can work in two ways:
- Testing the visual idea: generate up to three creative options for the same brief, at one credit each, with the same headline, description and CTA. The scene is the only difference.
- Testing copy, CTA or layout: open a banner in the editor and change only the headline, the CTA, the colors or the logo position. The background scene stays the same, the change applies to every size in the pack, and it doesn't spend a credit.
Make sure the variants are complete before the test starts. If A has all five core display sizes and B has only three, they show on different sites and the comparison is unfair. For ideas on what to put in the variants, see how to write a banner prompt.
Keep a test log
One test teaches you about one banner. A log of fifty tests shows you what works for your audience. Keep a simple table:
| Dates | Variable | A | B | Metric | Result | What we learned |
|---|---|---|---|---|---|---|
| Oct 1–14 | Offer | Fresh beans weekly | First box 50% off | Conv. rate | B +31%, 95% conf. | Prospects respond to price, not freshness |
The row above shows the format. It's an illustration, not real campaign data.
Add the date each winner stopped working, too. Winning banners wear out after a few weeks in front of the same audience, and a CTR that slides long after the test ended is usually a sign of ad creative fatigue, not a wrong test result.
FAQ
How many variants should I test at once?
Two, unless you have a lot of traffic. Each extra variant needs about the same amount of data again, so three variants take roughly 50% longer than two.
How long should an ad creative test run?
At least one full week, preferably two, and until each variant reaches the sample size you set in advance. Stopping early because one variant is ahead is the most common mistake.
Is CTR a good metric for display ad tests?
Use it to check that people notice the banner, but don't pick the winner by CTR alone. A banner with a higher CTR can still bring in fewer customers.