Scientists may work to prevent bias, but they don’t always say so
For a scientist, conducting a scientific study is walking into a minefield of potential biases that could detonate all over the results. Are the mice in the study randomly distributed among treatment groups? Does the person evaluating an animal’s behavior know what treatment the mouse got — and thus have an expectation for the outcome? Are there enough subjects in each group to reduce the odds that the results are due to chance?
“I think we’re getting increasingly better at identifying these risks and identifying clever and practical solutions,” says Hanno Würbel, an applied ethologist at the University of Bern in Switzerland. “But it’s not all obvious, and if you look back at the history of science you find that these methods have accumulated through a learning process.”
In theory, every time scientists design an experiment, they keep an eye out for these and other potential sources of bias. Then, when scientists submit the study design for approval or write journal articles about the work, they share that research design with their colleagues.
But scientists may be leaving some rather key bits out of their reports. Few animal research applications and published research reports include specific mentions of key factors used to eliminate bias in research studies, Würbel and colleagues Lucile Vogt, Thomas Reichlin and Christina Nathues report December 2 in PLOS Biology. The results suggest that the officials who approve animal research studies — and the scientists who peer review studies before publication — are trusting that researchers have accounted for potential biases, whether or not there is evidence to support that trust.
The team gained access to 1,277 applications for animal experiments submitted to the Swiss government in 2008, 2010 and 2012. The researchers examined the applications for seven common measures used to prevent bias: Randomization, calculations to make sure sample sizes were large enough, not telling the experimenter what treatment will be administered to the next animal, blinding the experimenter during testing to which animals received which treatment, criteria used to include or exclude subjects (say, the animal’s age or sex), explicitly stating the primary outcome to be measured and plans for statistical analysis of the data.
Most of the time, the applications didn’t mention how or whether any of those measures were considered. Scientists included statistical plans 2.4 percent of the time and sample size calculations only 8 percent of the time. Even the primary outcome variable — the main objective measured in the study — was mentioned in only 18.5 percent of the applications.
Würbel’s group also looked at 50 publications that came out of some of those animal research applications that were ultimately approved. Here, scientists were better about reporting their effort to stave off bias in their studies. If scientists mentioned one of the seven efforts to combat bias in their animal experiment applications, they were more likely to mention it when their final papers were published. But they still only reported the statistical plan 34 percent of the time, and none of the 50 papers reported sample size calculations.
Switzerland’s animal research application process didn’t actually require that any of these measures of bias be disclosed, Würbel notes. But “unless [the licensing officials] know the studies have been designed rigorously, they can’t assess the benefit.” The implication, the researchers suggest, is that authorities approving the studies trusted that the scientists knew what they were doing, and that peer reviewers and editors trusted that the authors of journal articles took those forms of bias into account.
Was that trust well placed? To find out, Reichlin, Vogt and Würbel surveyed 302 Swiss scientists who do experiments on living organisms, asking about the efforts they made to combat bias, and how often they reported those efforts. When asked directly, scientists said that of course they control their studies for certain risks of bias. The vast majority — 90 percent — reported that they included the primary outcome variable, and 82 percent included a statistical analysis plan. A full 69 percent reported that they calculated their sample sizes. Most also reported that they wrote these antibias efforts into their latest published research article.
But when the team probed deeper, asking scientists specific questions about what methods they used to combat bias, “you find out they don’t know much about methods,” Würbel notes. Only about 44 percent of the researchers knew about the guidelines for how to report animal experiments, even though 51 percent of them had published in a journal that endorsed those guidelines, the researchers report December 2 in PLOS ONE.
“This is a type of empirical work that we need, to see how people think and what they do,” says John Ioannidis, a methods researcher at Stanford University in California.
Just because scientists aren’t reporting certain calculations or plans doesn’t mean that their research will be subject to those biases. But without efforts to rigorously prevent bias, it can sneak in — subjects that aren’t randomly assigned properly, or an experimenter who unconsciously leans toward one result or another. Too small of a sample size and a researcher could detect a difference that disappears in a larger group. If a researcher knows which animals got which treatment, they may unconsciously focus more carefully on some aspects of the treated animals’ behavior — ignoring similar behavior in the control. And that can result in studies that are tougher to replicate — or that can’t be replicated at all.
None of this means that scientists are ill-educated or performing science badly, notes Malcolm Macleod, a neurologist at the University of Edinburgh. “I think there’s a temptation to make this binary, [to say] people don’t know so we need to train them,” he says. “The fact is most scientists know a bit … [but] everyone has something they can do better.”
How do you make scientists take more actions against bias, and then report what they’ve done? Journals, funding bodies and agencies approving animal research projects could require more information of scientists. Journals could require checklists for reporting methods, for example. Journals or funding bodies could also require full preregistration of animal studies, where a scientist gives all the details of a study and how it will be analyzed before the experiments are ever performed. (Such pre-registration became mandatory for clinical studies in humans in the United States in 2000.) Detailed reporting isn’t complete insurance against an irreproducible result, but “the more information you have, the easier it is to reproduce,” says Vogt, who studies animal welfare at the University of Bern.
Some scientists might worry that preregistration is too onerous, or that it could straitjacket researchers into unproductive studies. It would be frustrating, after all, to be stuck with a hypothesis that is clearly not bearing out, when the data provide tantalizing hints of another path to pursue. But it’s possible to provide the flexibility to pursue interesting questions, while still making sure the studies are rigorous, Ioannidis says.
But when scientists don’t know sources of bias in the first place, more education might be a good place to start. “When I first came into the lab for my master’s thesis, I had a lot of information [about research design] but I wasn’t ready to apply it,” Vogt explains. “I needed guidance through the steps of how to plan an experiment, and how to plan to report the experiment afterward.” Education doesn’t stop when graduate students leave the classroom, and more continuing education might help scientists — students and emeriti alike — recognize unfamiliar sources of bias and provide tools to combat it.