Within-Subject Design: A Solution for Small Sample Experiments?

When population size is low and the expected effect size from a treatment is small, experimenters feel powerless in detecting statistically significant results.

One approach, still be able to conduct hopefully a conclusive experiment that I’ve recently come across is the Within-Subject-Design (WSD), which to an extent feels falls under a broad category of quasi-experimental methods(?).

Unlike a traditional randomized controlled A/B test—where participants are split into treatment and control groups for the full duration—WSD exposes the same participants (or units) to all conditions/treatments, and compares their responses across those conditions. The key advantage of this setup is that it naturally accounts for individual differences, since each participant serves as their own control. This often means fewer participants are needed to detect an effect compared to between-subject designs.

However, the approach isn’t without trade-offs. WSDs can suffer from carryover effects (where experience in one condition influences the next) and order effects (such as fatigue or learning), albeit are manageable risk.

My bigger concern lies with time effects. In many real-world settings—like customer experiments—different weeks or periods of time can mean very different contexts for the same person. This makes time a potential confounding variable that isn’t naturally controlled for in WSD. Handling this properly requires additional design choices or statistical adjustments, which makes me cautious about applying WSD blindly.