Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials
Accepted 8 October 2004. published online 19 April 2005.
Abstract
Background and Objective
Divergent results on the same scientific question generate controversy. We hypothesized that controversial data are attractive to investigators and editors, and thus the most extreme, opposite results would appear very early rather than late, as data accumulate, provided data can be generated rapidly.
Methods
We used data from MEDLINE-indexed meta-analyses of case–control studies on genetic associations (retrospective, hypothesis-generating research with usually rapid turnaround) and meta-analyses of randomized trials of health care interventions (prospective, targeted research that usually takes longer) sampled from the Cochrane Library. Using cumulative meta-analysis, we evaluated how the between-study variance for studies on the same question changed over time and at what point the studies with the most extreme results ever observed had been published.
Results
The maximal between-study variance was more likely to be recorded early in the 44 eligible meta-analyses of genetic associations than in the 37 meta-analyses of health care interventions (P=.013). At the time of the first heterogeneity assessment, the most favorable-ever result in support of a specific association was more likely to appear than the least favorable-ever result (22 vs. 10, P=.017); the opposite was seen at the second heterogeneity assessment (15 vs. 5, P=.031). Such a sequence of extreme opposite results was not seen in the clinical trials meta-analyses. The estimated between-study variance decreased over time in genetic association studies (P=.010), but not in clinical trials (P=.30).
Conclusion
In contrast to prospective trials, a rapid early sequence of extreme, opposite results is frequent in retrospective hypothesis-generating molecular research.
aInstitute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, MA 02111, USA
bClinical and Molecular Epidemiology Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece
cBiomedical Research Institute, Foundation for Research and Technology-Hellas, Ioannina 45110, Greece
Corresponding author: Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece. Tel.: +3026510-97807; fax: +3026510-97867.
ABSTRACT