To err is human, to purr feline.        Robert Byrne

You can’t understand your data unless you control extraneous variance attributable to the way you select samples, the way you measure variable values, and any influences of the environment in which you are working. Using the concepts of reference, replication and randomization, you can control, minimize, or at least be able to assess the effects of extraneous variability in five ways:

• Procedural Controls — used primarily to prevent measurement variability by reference.
• Quality Samples and Measurements — used primarily to identify sources of measurement and environmental variability by reference.
• Sampling Controls — used primarily to correct sampling variability by reference and randomization.
• Experimental Controls — used primarily to correct measurement and environmental variability by reference and randomization.
• Statistical Controls — used primarily to correct environmental variability by reference.

Procedural Controls

Procedural controls include the field, office, and laboratory procedures that specify how activities related to data generation should be carried out. Forms of procedural control include Standard Operating Procedures (SOPs), chain-of-custody (CoC) procedures, survey scripts and instructions, checklists, calibration procedures, training courses and so on. Procedural controls are the main way to minimize extraneous variation before it occurs.

Quality Samples and Measurements

When a physical sample is collected, especially for laboratory analysis, there are a variety of samples that can be collected to verify that the analyses produce valid results. The aim of quality tests is to identify quality problems and sources of undesirable variability. Some quality samples and tests allow corrections to data points to correct bias. Quality samples used to assess variability in the collection and measurement of the experimental samples include:

• Replicate samples — multiple samples collected of the same subject to assess variability. Replicate samples may be collected by splitting a large sample into two or more subsamples (called split samples). Alternatively, discrete samples may be collected sequentially (called co-located samples). Most replicates are co-located samples because they are less time-consuming to collect. Co-located samples of mixable media (e.g., fluids like water and air) will usually be very similar compared to co-located samples of non-mixable media (e.g., solids like soil). Duplicate is a generic term for a replicate sample that may be either a split sample or a co-located sample.
• Blind samples — split samples submitted with unique sample identifiers so they appear to be from different sources. Blind samples are an independent check of laboratory methods.
• Trip blanks — distilled water placed in sample containers and sealed in the lab. Trip blanks are carried by the field sampling team and returned to the lab with the test samples. If any aspect of the sample handling and transport process might have compromised the quality of the samples within their containers, the effect should be detectable in the trip blank. It is very rare for trip blanks to contain contaminants.
• Rinse blanks — distilled water that has been poured over reusable sampling equipment that has been decontaminated. Rinse blanks are sometimes called field blanks, rinsate blanks, equipment blanks, or decontamination blanks because they are a test of the cleanliness of the sampling equipment.
• Atmosphere blanks — distilled water placed in sample containers that are opened during sample collection. Atmosphere blanks are a test of whether any air pollutants or windblown particulates might have contaminated test samples during the process of collecting the samples.
• Water blanks — potable water used for site activities, particularly rock drilling. Water blanks are also collected from wells when domestic plumbing is being assessed for lead contamination.
• Preservative blanks — preserved samples of distilled water. Preservative blanks are used to assess possible contamination of the acids, bases, and other reagents used to preserve test samples.

Analytical results from these samples are usually checked during data validation and exploratory data analysis. Except for replicates and blind samples (which are usually averaged), they are not included in datasets to be used for statistical analysis. Other QA/QC samples used to check laboratory procedures include:

• Laboratory blanks — distilled water or purified solid prepared in the lab to assess contamination from reagents, glassware, and analytical hardware. These samples are sometimes called method blanks because they are a test of the laboratory method.
• Performance evaluation samples — samples containing known concentrations of specific compounds. PE samples are usually used to certify laboratories for a particular type of analysis rather than check quality on a specific data collection task.
• Calibration samples — samples containing known concentrations of an analyte similar in chemical behavior to the analytes of interest. These samples are used to calibrate instruments and assess method bias.
• Matrix spikes — samples of the media being analyzed that have been spiked with known concentrations of a representative analyte. These samples are used to check for interferences between the analytes being tested and the sample matrix. Matrix spike duplicates are routinely analyzed by laboratories to assess measurement variability.

Analytical results from these samples are usually checked during data scrubbing. They are not included in datasets to be used for statistical analysis. Most of these QA/QC samples are used only for chemistry laboratories. Quality samples for other types of laboratory analyses (e.g., geotechnical, radiological, biological) are usually limited to replicates.

These are commonly used quality samples but there are many others possible. You create the samples to fit the experimental situation. Don’t feel limited to laboratory samples. You can use the same approach to create tests or other methods of variance assessment. If you understand the possible sources of variation in your data, you can create relevant quality tests for survey questions, industrial processes, or whatever your study will involve.

Sampling Control

Some types of samples have inherent properties that may introduce extraneous variability into data being generated. For instance, differences in sex, age, and social class, may introduce extraneous variability in sociological surveys. Environmentally related examples might include: soil type, geologic strata, species, location and depth, and season (or other time unit). Applying sampling controls involves grouping the data by the control factor and calculating statistical analyses for each homogeneous group. Stratified sampling designs are one form of sampling control.

Sampling controls can be used to prevent, identify, or correct all kinds of extraneous variation. As a consequence, they are used to some extent in most statistical studies.

Experimental Control

Statistical studies can be categorized into two types — observational and experimental. In an observational study, the phenomenon under investigation is a characteristic of the objects being sampled. The concentrations of arsenic in the soils of a waste management facility would be an example. Variability and bias in observational studies can be assessed through the use of control samples. Control samples (like quality samples) are groups of samples that don’t have the condition being tested but are otherwise identical to the experimental group. Adding an offsite (or background) area to the study of the waste management facility would be an example of a control group.

In an experimental study, the phenomenon under investigation is assigned to the objects being sampled. Testing a pollution cleanup technique on several plots of contaminated soil would be an example of an experimental study. In such a study, the cleanup techniques would be randomly assigned to the plots in a manner that would help control some of the variation in the plots.

So, the difference between experimental and observational studies is that in an:

• Experimental study, samples are randomly assigned to controlled conditions.
• Observational study, samples are randomly selected from preexisting conditions.

I’m not telling who got the placebo. (Cat stays in bag.)

Another way to control variability and bias is through the use of placebos. A placebo is usually thought of as a faux drug because of its association with statistical tests of pharmaceuticals, but it can be any item or action that gives the appearance of being a valid treatment. For example, give two patients blue pills, one of which has some active ingredient and the other does not. To test a new pesticide, spray two agricultural test plots, one of which contains the pesticide and the other contains just water. The key is that the subject or data generator can’t tell the difference. In Season 5, episode 14 of the television series House, the nurse can tell which patients are being given the placebo during Dr. Foreman’s drug trial because the real pharmaceutical has a strong odor. This would not be a good example of an effective placebo.

Placebos are a type of blinding. In any experiment, there are subjects or samples, experimenters, data collectors or generators, and data analysts. Sometimes, one individual will fill several roles, such as the experimenter who designs the experiment and then generates and analyzes the data. But whenever there are humans involved, there exists the possibility of intentional or unintentional bias. Blinding is simply the act of denying one or more of the study participants with information that might induce them to behave differently.

To test the effects of a pharmaceutical, for example, a single-blind study might involve not telling subjects whether they are receiving the active drug or a placebo. A double-blind study might involve not telling either the subjects or the data generators (the nurses who collect physical measurements on the subjects or the lab technicians who analyze blood samples) who received the drug and who received the placebo. A triple-blind study might involve not telling the subjects, the data generators, or the data analysts who received the drug and who received the placebo. It’s ironic that the less informed the participants are, the better the statistics. Too bad government doesn’t work that way.

Statistical Control

Special statistics and statistical procedures can be used to partition variability shared by measurements so that extraneous variability can be assessed. For example, partial correlations quantify the relationship between two variables while holding the effects of other variables constant. A covariate is a continuous variable that is incorporated into an analysis of variance design to eliminate extraneous variability so that tests of the grouping factors will be more sensitive. Statistical controls are typically used when variation cannot be controlled adequately through other means.

Read more about using statistics at the Stats with Cats blog. Join other fans at the Stats with Cats Facebook group and the Stats with Cats Facebook page. Order Stats with Cats: The Domesticated Guide to Statistics, Models, Graphs, and Other Breeds of Data Analysis at Wheatmark, amazon.com, barnesandnoble.com, or other online booksellers.