Why We Aligned Our A/B Test Allocation Design with Industry Standards — Separating "Traffic Allocation" from "A/B Split"

Summary of this article

• A/B test allocation settings tend to conflate two distinct concerns: "traffic allocation" and "A/B split"
• We separated the two into independent parameters
• This separation lets us handle "risk management" and "fair comparison" cleanly, each on its own terms

When implementing the A/B testing feature in HeatMapX, we faced a design decision that was subtle but important: how to model "allocation." In the end, we separated "the share of visitors included in the experiment (traffic allocation)" and "the split between A and B" into two independent parameters. This article explains why.

Two concerns that are easy to conflate

The "allocation" in an A/B test actually bundles together two concerns of a fundamentally different nature.

• How many visitors to include in the experiment (risk management)
• How to split those included visitors between A and B (fairness of comparison)

Early, naive designs tend to combine these into a single slider. But doing so tangles up two separate goals — "I want to test safely, starting small" and "I want to compare A and B fairly" — making things harder to understand for both users and implementers.

The separated model

So we made the two independent.

• traffic_allocation (share included in the experiment): the proportion of all visitors who are made eligible for the experiment. 0-100%.
• A/B split: the weighting between Control and Variant B among visitors who are eligible.

In pseudocode, the assignment logic looks like this:

1. First, determine whether a visitor is "eligible for the experiment" based on the traffic allocation.
2. Only visitors who are eligible are then assigned to Control or Variant B according to the A/B split.

Visitors who are not eligible simply see the normal page (unchanged).

Benefits of the separation

• Risk can be scaled up gradually: you can move traffic allocation from 10% to 30% to 100% while keeping the comparison fair (50:50) throughout, adjusting only the risk.
• Settings are clearer: "how much to include" and "how to present it" become separate UI elements, reducing confusion.
• Consistency with industry standards: major A/B testing tools also treat traffic allocation and in-experiment assignment as separate concerns.

Reflected in the UI

This design decision carries straight through to the UI. On the experiment settings screen, you can specify "share included in the experiment" and "A/B split" separately, and the experiment detail view shows both as a summary.

Summary

Even a seemingly simple element like "allocation" hides two distinct concerns: risk management and fairness of comparison. By separating them into distinct models, we made A/B testing clearer for users and easier to operate. It's a small design decision, but choices like this add up to shape the overall usability of a product.