What Is Holdout group?

A comparison protected from an intervention

A holdout is intentionally kept outside a specified change so outcomes can be compared with an appropriate exposed group. The withheld intervention might be a new scoring rule, an additional channel or an entire program. Define what the group continues receiving: previous practice, another baseline or no treatment. Withholding one change is different from stopping every service or ignoring an existing customer request.

Some platforms use holdout for a persistent group spanning several experiments, while other workflows use a narrower campaign comparison. State the intended duration and scope. A control in one A/B test and a program-wide holdout answer different questions, so the name alone should not imply a particular experimental design or a universal measurement feature.

Assign the right unit

Choose whether the unit is a person, account or another entity before allocating records. In account-based outreach, assigning colleagues at the same company to different groups can expose the nominal holdout indirectly. Existing campaigns, seller activity and shared marketing can also contaminate the comparison. Track actual exposure and preserve the assignment rather than moving inconvenient outcomes between groups afterward.

Decide outcomes before looking

Specify the business measure, observation period and analysis plan. Use random assignment where appropriate for the question and ensure the groups have enough evidence for the intended comparison. Small differences in sparse meeting counts can be noisy. A holdout label does not make an underpowered or biased test reliable, and withholding a change does not guarantee that all other conditions stay equal.

Keep mandatory exclusions and contact restrictions outside the treatment decision. No experiment makes an otherwise prohibited outreach action acceptable. A well-designed holdout supports an estimate of incremental effect under stated conditions; it is not automatic proof that every observed difference came from the program or that the same effect will persist in a different market or period.