Heroes Not Zombies

There’s a strange presumption in some quarters that “science” (as if there was a “thing” called “science”) is the only way to know the TRUTH about reality. Research is a fascinating phenomenon to consider, and the more you consider it, the less straight forward it appears! One of the issues in medical research is the issue of bias produced by the funds for a study being put up by the pharmaceutical company which makes the drug. This is an issue which has been, and will continue to be, discussed and debated extensively. I’m not going to go down that road in this post. What I would like to draw your attention to however is an article published in PloS Medicine. The authors argue that the sheer size of the output from labs and research groups so outweighs the number of outlets for publication that there is an intense publication bias through the necessary highly limited selection process of the journals, that what is published amounts to a serious distortion of the reality of scientific endeavour.

The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality. Science is subject to great uncertainty: we cannot be confident now which efforts will ultimately yield worthwhile achievements. However, the current system abdicates to a small number of intermediates an authoritative prescience to anticipate a highly unpredictable future. In considering society’s expectations and our own goals as scientists, we believe that there is a moral imperative to reconsider how scientific data are judged and disseminated.

An example of the kind of distortion of reality this selection process produces was highlighted in the New England Journal of Medicine which found this –

Results Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published (22 studies) or published in a way that, in our opinion, conveyed a positive outcome (11 studies). According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall.

The PloS article makes some suggestions for improving the system, not least recommending the digital publication of any research without significant errors rather than selection on the basis of what someone considers “important”. That would be a start! But the most important point here is that published research is not equivalent to Truth. We should consider it, but remain wary of claims to certainty.