Choosing the Unit of Randomization in Experiments: An A/B Testing Design Guide

Why the Unit of Randomization Matters

The unit of randomization is the entity that gets independently assigned to either the control or treatment group. Whether that's an individual user, a single session, a particular device, or an entire organization — this choice is not trivial. It determines whether you can maintain statistical independence between observations. When independence breaks down, the statistical tests themselves become unreliable, and conclusions about the range of effect sizes become shaky as well.

Choosing the wrong unit of randomization causes problems like:

1. Increased Type I error (false positives): Mistakenly concluding an effect exists when it doesn't.
2. Reduced statistical power: Failing to detect an effect that actually exists, missing an opportunity.
3. Biased effect estimates: Misjudging the size or direction of an effect, leading to bad decisions.
4. Reduced generalizability: The sample context (scope) ends up too narrow or poorly defined, making it hard to apply the findings broadly.

At the root of the problem is a violation of statistical independence. When the treatment assignment of one unit affects the outcome of another unit — whether within the same group or across groups — it breaks the assumptions underlying standard statistical tests. This often shows up as "leakage" or "contamination."

Common Units of Randomization Used in Experiments

1. User-Level Randomization

Definition: Individual users are randomly assigned to the control or treatment group, and remain fixed in that group for the duration of the experiment regardless of device, session, or activity.

Pros:

• High statistical independence: When implemented properly, user-level randomization generally offers the highest degree of independence. Since one user's behavior rarely directly affects another user's treatment experience, it's easier to robustly estimate the range of effect sizes and build confidence in the results.
• Consistent user experience: Because the user's experience stays stable, this is advantageous for measuring long-term engagement or retention and avoiding user confusion.
• Strong for cumulative metrics: Well-suited to measuring metrics that involve a user's entire journey, such as conversion rate, retention, LTV, or cumulative feature usage.

Cons:

• Leakage risk (multiple devices / shared accounts): If a user accesses the product from multiple devices or shares an account, they may experience both conditions unless assignment is based on a consistent user ID.
• Requires persistent user identification: Depends on robust tracking of a unique ID across sessions and devices.
• Not well suited to observing short-term effects: Because the assignment is fixed, immediate short-term effects that could vary session by session may get buried within the user's overall behavior pattern.

When to use:

• This is the most common and generally recommended choice for experiments where a consistent, long-term user experience matters — product feature tests, pricing experiments, personalization initiatives, and so on.
• When measuring metrics tied to an individual user's behavior over time, such as subscriptions, churn, or repeat purchases.

2. Session-Level Randomization

Definition: Each new session a user starts is independently assigned to the control or treatment group. The same user may experience both control and treatment across different sessions.

Pros:

• Quickly gathers session-dependent effects: Allows you to rapidly collect data on immediate within-session behavior or short-term UI/UX changes.
• Useful for transient interventions: Well suited to experiments where the intervention is extremely short-lived or limited to a single interaction within a session (a one-time pop-up, a specific search-results layout for that query, etc.).
• Less reliant on persistent user IDs: Leaves room to use this approach when robust cross-session user identification is difficult, though this comes with its own risks.

Cons:

• Statistical independence is easily violated: When the same user experiences both control and treatment, it creates significant contamination and bias. What a user experienced in one session can affect their behavior in the next session regardless of the new assignment, making it extremely difficult to isolate the true range of effect sizes.
• User confusion/frustration: Inconsistent experiences for the same user can cause frustration, potentially affecting engagement metrics in both groups.
• Analysis becomes more complex: Since observations are no longer independent, you need more sophisticated statistical methods that account for within-user correlation, which affects your confidence in the results.

When to use:

• Use only in very limited scenarios where you're confident the intervention is genuinely confined to a single session and unlikely to have a lasting effect on subsequent sessions.
• Example: a one-off notification, a specific query-result layout (where the user leaves and returns to a completely independent new search), or within-session banner-ad variations that are assumed to have no persistent effect — though such cases are rare.
• Caution: This is generally not recommended for interventions intended to change user behavior over time, because of the high risk of leakage and lack of statistical independence.

3. Group/Cluster-Level Randomization

Definition: Entire groups or clusters of users are randomized together — teams, organizations, regions, or users of a particular legacy system, for example.

Pros:

• Mitigates spillover/leakage: Essential when an intervention inherently affects an entire group, or when contamination between individuals within a group is unavoidable. For example, when testing a new collaboration feature for teams, the whole team needs to be in the same group.
• Preserves "social" independence: Prevents closely interacting individuals from being split between control and treatment, preserving natural social dynamics that may matter for a particular experiment.

Cons:

• Reduced statistical power: Achieving sufficient power often requires substantially increasing the number of clusters rather than individuals, since the "effective sample size" becomes the number of clusters, not the number of individuals. This directly affects your ability to detect a meaningful range of effect sizes.
• Increased variance: Variance between clusters tends to be large, making it harder to discern the treatment effect.
• Harder to balance: Because the units are fewer and larger, it's more difficult to balance attributes or usage patterns between the control and treatment groups.
• Harder to generalize: The sample context can end up heavily dependent on the particular type of clusters chosen.

When to use:

• When the intervention affects groups rather than individuals (e.g., a new team communication tool).
• When leakage or "network effects" are a major concern and individual-level randomization can't ensure independence (e.g., A/B testing a product feature with viral characteristics).
• When testing pricing or features for enterprise accounts, where the entire company needs to see the same experience.

The Danger of Leakage and Contamination

Leakage (also called spillover or contamination) occurs when the treatment status of one unit unintentionally affects the outcome or experience of another unit, undermining statistical independence. This is a major threat to the validity of your experimental design.

Examples of leakage:

• Shared devices/accounts: A user is assigned to the control group, but a family member who shares the same device or account is exposed to the treatment.
• Communication between users: A treatment-group user tells a control-group friend about a new feature, influencing that friend's behavior.
• Platform-wide changes: An optimization aimed at a specific user segment (treatment) unintentionally affects the performance or experience of the entire system, including control users.
• Flaws arising from session-level randomization: As mentioned above, if what happened in one "treatment" session affects behavior in a later "control" session, the observations are no longer independent.

Impact: Leakage generally biases effect estimates toward zero, making it harder to detect a real effect (reduced power) or masking its true magnitude. In some cases, it can bias effects away from zero, producing false positives. Either way, it distorts your understanding of the range of effect sizes and undermines confidence in decision-making.

A Process for Choosing the Unit of Randomization

Choosing the optimal unit of randomization is a systematic process. Follow these steps for a robust experimental design:

1. Define the causal question and the intervention:
2. Specifically, what change are you introducing?
3. What is the smallest unit directly exposed to this change?
4. What is the primary outcome metric you want to measure?

5. Example: If you're testing a new onboarding flow, the intervention applies to new users. If you're testing a button color, the intervention applies to the user who sees that button.

6. Identify the natural unit of behavior:

7. Whose behavior do you primarily want to influence and measure — an individual's overall engagement (user), or a discrete interaction within a short window of time (session)?

8. Tip: If the effect is expected to persist beyond a single interaction, or if memory or learning is involved, user-level randomization is generally safer.

9. Assess the potential for interference or leakage:

10. This is the most important step. Map out every pathway by which the treatment assignment of one unit could affect the experience or outcome of another unit.
11. Consider shared resources, social networks, multi-device usage, and the temporal nature of the intervention.
12. If you determine that a smaller unit (such as a session) risks significant leakage, you need to move to a larger, more independent unit (user-level, or even group-level).

13. Reference: OKPy Data Analysis's insight cards: Correctly understanding the range of effect sizes and sample context depends heavily on whether you've correctly isolated the treatment effect. Leakage obscures this.

14. Assess the impact on statistical power:

15. The unit of randomization you choose changes how many observations you need to detect a statistically significant effect. Cluster-level randomization in particular often requires a much larger overall sample size to compensate for within-cluster correlation.
16. Estimate the variance at the level of the unit you've chosen. Assuming independence when it doesn't actually exist will leave your experiment underpowered.

17. Use OKPy Data Analysis's insight cards: Check confidence and the range of effect sizes to understand how your chosen unit affects sample-size requirements and the precision of your effect estimates.

18. Consider implementation and tracking feasibility:

19. Can your analytics system reliably assign and track the unit you've chosen (a consistent user ID, a precise session definition, etc.)?
20. Are there technical constraints that make a particular unit impractical or error-prone?
21. Robust implementation is essential for data quality and the validity of your experimental design.

Practical Examples

Summary

The choice of the unit of randomization is a foundational decision in experimental design that directly affects the validity of your results. Prioritizing statistical independence and actively eliminating leakage are key to obtaining trustworthy findings. By carefully considering the nature of the intervention, the user experience, and the potential for contamination, you can choose the right unit and increase your confidence in the measured range of effect sizes. That precision is what grounds business decisions in robust data rather than statistical artifacts.

Related guides:

• How to read A/B test results: /guide/how-to-read-benchmarks
• Calculating experiment power and sample size: /tools/benchmark-calculator