Offer Rotator Split Test: Share Splits Without Skewed Stats

If you want to compare two landing pages or two offers, you send half of the traffic to each and look at which one converts better. That part is easy. The hard part is doing it so the numbers you read at the end mean something. Most "winners" in affiliate split tests are noise, a changed setting or a bot burst.

This post covers the two ways to split traffic by shares, when each fits, how many clicks you need before you believe a result, and the mistakes that quietly ruin a test.

Two ways to split: equal chance and weighted shares

In TDS.SO there are two tools for splitting, and they answer different questions.

Several URLs in one rule, separated by |. Every click picks one of the URLs at random with equal chance. With two landers you get about 50/50, with four about 25% each. You can't set the proportions, which is exactly what you want in a test: no variant gets a privileged share.

The link rotator. You list destinations and give each a share, for example 50/30/20. Shares can be changed at any time and the link in the ad stays the same. This is the tool for distributing volume between offers: the strongest one gets half, the second a third, and a small share goes to a newcomer you are still checking. More on the feature is on the link rotator page.

What "random" gives you and what it doesn't

Random assignment means each click is independent. With 1,000 clicks and a 50/50 split you will not get exactly 500 and 500. You might see 487 and 513, and that is normal. The gap shrinks in relative terms as volume grows, so do not panic over a small imbalance in the first hundred clicks.

What randomness does not give you is a repeat of the same visitor on the same variant. A person who clicks twice may see landing A the first time and landing B the second. For most paid traffic this is rare enough to ignore, but keep it in mind if your offer has a long consideration cycle.

How many clicks before you believe the result

Say you buy traffic at $0.10 per click and landing A converts at 2.5%. You build landing B and hope for 3.5%. After 300 clicks on each, A has 7 conversions and B has 11. B looks 50% better. It is not a result you can bank on, because a swing of a few conversions in either direction is easy to produce by luck.

A common statistics rule of thumb (about 80% power at the usual 5% significance level) says you need roughly 16 × p × (1 − p) / d² clicks per variant, where p is the base conversion rate and d is the difference you want to detect. These numbers are hypothetical, so check them with your own:

• Base rate 2.5% (0.025), target 3.5%, so d = 0.01: 16 × 0.025 × 0.975 / 0.0001 is about 3,900 clicks per variant.
• At $0.10 per click that is around $390 per variant, $780 for the whole test.
• If you only care about a bigger jump, say 2.5% to 5%, d = 0.025 and you need around 620 clicks per variant.

The takeaway for planning: small improvements need large volume, and large improvements show up quickly. Decide the minimum difference that would change your decision before you start, work out the volume, and check whether the budget covers it. If it doesn't, test something bolder, such as a different angle or offer, not a button colour.

This is a general statistics guideline, not a feature of any tracker. TDS.SO splits the traffic and logs the clicks; the judgement of significance is yours.

Mistakes that ruin a split test

Changing shares in the middle

Because shares can be edited any time, it is tempting to nudge them when one variant looks ahead after two days. Don't. If A had 30% of the traffic on Monday and 70% on Wednesday, and your audience differs by day of week, the two variants have seen different people. Either keep the shares fixed until you have the volume you planned, or start a fresh test with a clean period and note the date.

Mixing geos

Say landing A gets more clicks from a high-converting country by pure chance. It will win for reasons that have nothing to do with the page. Keep one geo per test, or put a country condition on top and split inside each country separately. The rules are checked top to bottom and the first match wins, so a rule for one country sits above the general rule. The approach is the same as in geo redirects without a headache.

Letting bots into the sample

Bots click but almost never convert, and they don't spread evenly across variants. If a burst of data center traffic lands on B, B's conversion rate drops and you throw away a good landing. Turn on a filter level that fits your source and tick the signals you need, such as data centers or an empty referrer. Filtered clicks go to a page you set and stay in the log with the reason, so you can see how many were removed. Details are on the anti-bot page, and the symptoms are covered in signs of bot traffic.

Judging on clicks instead of conversions

Clicks per variant tell you only that the split worked. A landing with a high click-through to the offer can still produce fewer sales if it warms the visitor wrongly. Compare conversions per click, and if you can, revenue per click. In TDS.SO the tracker with postbacks, conversions, revenue and ROI is in beta; revenue is counted from approved conversions and cost from all clicks. If you are not on postbacks yet, take approved leads from your network's report and match them by variant.

Stopping at the first lead

The moment one variant is ahead is not the moment to stop. Set the volume in advance, and read the result once you reach it.

A checklist before you start

1. One hypothesis, one change. If you change the headline and the offer together, you won't know which did the work.
2. One source, one geo, one device group per test.
3. Equal split with | for the comparison.
4. Bot filter on, with signals chosen for your source.
5. Planned volume per variant written down.
6. A way to see conversions per variant: postbacks, or the network report matched by variant.
7. No edits to shares or pages until you hit the volume.

After the test: move to weights

When the test is over, you can switch to the rotator and keep the winner at a high share, for instance 80%, with the rest on a challenger. This keeps a small stream of fresh data so you notice when the winner fades. The ad link doesn't change, so the campaign is not touched.

FAQ

Can I split traffic 70/30 for a test?

You can with the rotator, but the smaller variant needs more total time to reach the same volume. Use 50/50 unless you have a reason, such as protecting a strong offer from a risky change.

Do I need a tracker for this?

To split the traffic, no. To know which variant converts better you need conversions tied to the click, which means a postback or a network report you can match by variant. See what a postback URL is.

How long should a split test run?

Until you reach the planned volume and cover at least a full week if your audience behaves differently on weekdays and weekends. Stats days in TDS.SO follow your time zone, which helps when you compare whole days.

Should I test landings or offers first?

Offers usually differ more than landings, so differences show up with less traffic. Landing tests need more volume for smaller gains.

What if my traffic is too small to reach the volume?

Test one big change instead of several small ones, or run the test longer at lower daily spend.

To set up a rotator or an equal split on one link, see the link rotator and the plans.