Determining the Optimal Size of Experiments and Surveys in Empirical Research

In empirical research, two activities carry most of the weight: experiments taken in the strict sense, and surveys. For the purposes of this discussion I group both under a single label, calling them experiments in a broader sense. Together they are the main instruments through which empirical questions get answered.

Research never has unlimited resources. Money, time, material, and the number of subjects available are all finite, so the only responsible approach is to spend them as efficiently as the problem allows. That efficiency has to be built into both halves of the work: the way an experiment is designed before any data appears, and the way the resulting data gets analysed afterward.

A sound sequence begins with stating the experimental question exactly. Vagueness at the start tends to multiply later, and the question is worth revisiting after each of the subsequent steps rather than treating it as settled on the first try.

Several decisions follow from that opening question. One is settling on a statistical model that actually suits the situation. Another is specifying how much precision the result needs to reach, together with the criterion by which one design counts as better than another.

Once those pieces are fixed, the experiment can be planned so that it meets the stated requirements with the least expenditure, then carried out, and then analysed in a manner consistent with the plan that produced it. The planning and the analysis are not separable concerns; the choices made before data collection determine what the later analysis can legitimately claim.

This article walks through that procedure in a set of practical cases, drawing on both linear and non-linear statistical models to show how sample size and design questions get resolved before a single observation is recorded.