No matter what their area of expertise, statisticians are asked certain questions with such predictability that it borders on the deterministic. No question is asked more often than:

How many samples do I need?

Most statisticians wish they could answer the sample size question definitively instead of mumbling about effect sizes and whatnot. It’s just not that simple.

One way to look at how many individual samples (i.e., observations, cases, records, subjects, survey respondents, organisms, or any other object or entity on which you collect information) you need for an analysis is in terms of how much resolution you want. Think of the resolving power of a telescope or a microscope, or the number of pixels in a computer image. The greater the resolution, the more detail you’ll see.

Consider this picture. You couldn’t make out the image with a resolution of 9 pixels per inch and maybe not even with a resolution of 18 pixels per inch. At 36 pixels per inch, you can tell it’s an image of a kitten, even if it’s a bit fuzzy. At 72 pixels per inch, the image is sharp and you can tell that the kitten is Kerpow. Doubling the resolution again adds little to your perception of the image; it’s a waste of the additional information. Likewise with statistics, the greater the number of samples, the more precise your results will be. But beyond a certain point, adding samples adds little to your understanding. In fact too many samples can have negative consequences. So, the trick is to collect the fewest samples that will achieve your objective.

Deciding how many samples you’ll need starts with deciding how certain your answer needs to be given your objective. Now here’s the bad news. There’s no way to know exactly how many samples you’ll need before you conduct your study. There are, however, formulas for estimating what an appropriate number of samples might be. In the situations in which the formulas don’t apply, there are rules-of-thumb or other ways to come up with a number. Unfortunately, it seems that no matter how many samples you estimate you’ll need, the number is always a lot more than your client wants to collect. After all, they are the ones who have to pay for collecting and analyzing the samples. So, any estimate of the number of samples that you tell your client ends up being a subject of negotiations. But here are a few places to start.

How Many Samples for Describing Data?

Say all you want to do is to collect enough samples to calculate some descriptive statistics. Maybe you want to characterize some condition, like the average weight of a litter of kittens or the average age of your favorite professional sports team. How many samples do you need? Well if your population is small enough, like five kittens or 25 baseball players, you simply use all the members of the population, a census.

But what if you want to calculate descriptive statistics to characterize a large population? The number of samples you’ll need to describe it will depend on the precision you want, not the accuracy. The greater the number of samples, the more precise your estimate will be. More specifically, the precision will be proportional to the variance divided by the square root of the number of samples. So maximize the number of samples if you can (by a lot, remember, precision is proportional to the square root of the number of samples), but if you can’t, try to control the variance.

How Many Samples for Detecting Differences?

Often, the point of calculating statistics is to make an inference from a sample to a population. You can estimate how many samples you might need to conduct a statistical test of one or more populations by rearranging the equation for the test you plan to use and solve for the number of samples. To take this approach, there are usually two other things you need to know—the difference you want to detect and the population variance.

You should have some idea of the size of the difference you want to detect, called a meaningful difference. Say you want to compare how long it takes you to commute to work via two different routes. Differences of a few seconds probably aren’t meaningful but differences of a few minutes probably are. If you work as a NASCAR driver, go with seconds. The smaller the difference you want to detect the more samples you’ll need.

Knowing the population variance is the Catch-22 of statistics. You can’t calculate the number of samples you’ll need without knowing the population variance and you can’t estimate the population variance without already having samples from the population. Now, there are maybe a half dozen ways to try to get around this problem but they all require you to know or guess at some aspect of the population. The approach is often used after a preliminary study (called a pilot study) is done in part to estimate the population variance.

How Many Samples for Opinion Surveys

If you’re going to survey a small population, like your colleagues at work, send surveys to everybody and hope you get a representative sample from the people who do respond. If the size of your population is large compared to the number of surveys you might take, a quick way to estimate the sample size is:

sample size = 1 / (approximate percent error you want)²

So if you want a ±5% error with 95% confidence, you would need about 400 surveys to be completed (i.e., 1/0.05²). With 1,000 surveys, the error drops to about 3%, but to get to 2% error, you would have to collect 2,500 surveys. It’s more complicated than this of course. If your sample will be a sizable proportion of your population or if the opinions aren’t evenly divided, the short-cut formula will overestimate how many surveys you might need. If you plan to subdivide the population to look at demographics, you’ll need more surveys to get to your desired error rates.

How Many Samples for Evaluating Trends?

Say you plan to do a regression analysis to evaluate the relationship between two sets of measurements. How many samples do you need? There are two ways to answer this question, a difficult way and an easy way. The difficult way is to calculate it the same way as you would if you were looking to detect differences. This approach requires a sophisticated understanding of statistical tests and the populations being tested. It is most often used in experimental situations.

The simpler approach is to base the number of samples on a rule-of-thumb based on the number of independent variables. The more independent variables (i.e., predictor variables) there are, the more samples are needed to define their relationship to a dependent variable. The guidelines are not hard and fast but they boil down to these:

• 10 samples per predictor variable—the bias may be large but there are often enough samples to estimate simple linear relationships with adequate precision.
• 50 samples per predictor variable—the bias is relatively small, linear relationships can be estimated with good precision, and there are usually enough samples to determine the form of more complex relationships.
• 100 samples per predictor variable—the bias is insignificant, linear relationships are estimated precisely, and complex nonlinear relationships can be estimated adequately.
• 250+ samples per predictor variable—the bias is insignificant and most complex relationships can be estimated precisely.

How Many Samples for Forecasting Time Series?

Deciding how many samples to use for analyzing a time series can be a challenge. Here are two popular rules-of-thumb:

• Collect samples at regular intervals from at least three or four consecutive cycles or units of any pattern in which you might be interested. For example, if you are interested in seasonal patterns (i.e., a pattern lasting a year) collect data for at least three or four years.
• Collect samples at time units smaller than the duration of the pattern in which you might be interested. For example, if you are interested in seasonal patterns, collect data weekly, biweekly, or at least, monthly.

How Many Samples for Identifying Targets?

Sometimes the goal of sampling is to find one or more targets. For example, in World War II, destroyer captains needed to know how many depth charges to drop to be reasonably certain of destroying an enemy submarine. Likewise, adventurers looking for sunken ships, like the Monitor and the Titanic, use statistical sampling to find their targets. In the environmental field, sampling is often done to look for “hot spots” of contamination in soil. There are two ways this type of problem is typically handled—judgment sampling and search sampling.

The strategy behind judgment sampling is that an expert picks locations for sampling that he or she believes are most likely to reveal a target. With this approach, it is assumed that the expert has some preternatural ability to find the targets. Judgment sampling (a.k.a. judgmental sampling, biased sampling, haphazard sampling, directed sampling, professional judgment) has the advantage of involving far fewer samples than search sampling. The disadvantage is that there is no way to quantify the uncertainty of the result.

Search sampling involves sampling on a regular grid so that it is possible to estimate the probability of finding randomly located targets. In essence, the probability of finding a target depends on the size and shape of the target and the size and shape of the cells of the sampling grid. The downside of this sampling approach is that it usually involves many more samples than the judgment sampling approach and the results do not always sound very reassuring. For example, you would need over 10,000 samples taken on a 100-foot square grid in a 1,000,000 square-foot search area to have an 80% probability of finding a circular hotspot 100 feet in diameter. In search sampling, a large number of samples is the price you pay for being able to quantify uncertainty. But if you understand the uncertainty, you are one giant step closer to controlling adverse risks. That’s the resolving power of statistics.

Read more about using statistics at the Stats with Cats blog. Join other fans at the Stats with Cats Facebook group and the Stats with Cats Facebook page. Order Stats with Cats: The Domesticated Guide to Statistics, Models, Graphs, and Other Breeds of Data Analysis at Wheatmark, amazon.com, barnesandnoble.com, or other online booksellers.