Heroes Not Zombies

Here’s two reasons why I’m concerned about using “Evidence Based Medicine” to make decisions about what should not be available on the NHS.

The first reason is we actually don’t have good convincing evidence for most of the treatments we currently deliver. The BMJ’s “Clinical Evidence” highlights this on its front page.

They categorise treatments into “beneficial, likely to be beneficial, trade-off between benefits and harms, unlikely to be beneficial, likely to be ineffective or harmful, and, unknown effectiveness. This in itself is controversial –

Dividing treatments into categories is never easy hence our reliance on our large team of experienced information specialists, editors, peer reviewers and expert authors. Categorisation always involves a degree of subjective judgement and is sometimes controversial.

In other words, the categorisation process is a matter of opinion and the opinions will be influenced by the authors’ values, beliefs and personal experiences.

Actually, it gets worse, because earlier on the same page, “Clinical Evidence” says

we wish to highlight treatments that work and for which the benefits outweigh the harms, especially those treatments that may currently be underused. We also wish to highlight treatments that do not work or for which the harms outweigh the benefits.

The thing is, in the real world, you can’t divide treatments into two such categories – those which work and those which don’t. There are NO treatments which “work” for every single patient. We are all individuals. It would be much better to publish the spread of responses experienced, and to resist this simplistic and delusional two-value “works/doesn’t work” categorisation.

And here’s the stunning statistic – 51% of the treatments are rated as “unknown effectiveness” (sometimes referred to as “insufficient evidence”). Were we to remove ALL of these treatments from the NHS, not only would we throw out a lot of babies with the bathwater, but does anyone seriously think we’d have a better health service?

The second reason to worry is just how appallingly distorted the evidence base is by the failure of companies to publish research which does not demonstrate positive effects of their drugs.

The German Institute for Quality and Efficiency in Health Care has published a study in the journal Trials.

they assessed hundreds of articles from journals and other sources, which covered areas including treatment for psychiatric disorders, pain, heart and circulatory disease, skin disease, cancer, and infectious diseases. A wide range of interventions was affected: from drugs and vaccines to medical devices such as ultrasound or devices for wound care.

What did they find?

an analysis of 90 drugs that had been newly approved in the US showed that they had been tested in a total of 900 trials. However, even 5 years after approval, 60% of these studies were unpublished. On the second level only selected outcomes from studies are published

Comparisons of protocols and journal articles of studies showed that in 40% to 60% of studies, results had either been completely omitted or analyses changed.

This does not only affect studies sponsored by the pharmaceutical industry. In their paper, the IQWiG authors also cite an analysis in which 2000 studies on cancer topics were analysed according to sponsorship. The proportion of published studies was extremely low: of the industry-sponsored studies, 94% were unpublished; however, even 86% of university-sponsored studies were also unpublished.

Irving Kirsch found similar significant distortions of the evidence base when he got hold of unpublished trials of anti-depressants.

The claims to certainty by the advocates of those who seek to make decisions about health care solely based on published research are misguided.

What’s the answer? Well, first of all bringing the current advice up to date by doing what these researchers have done, getting hold of the unpublished research and seeing how that changes the picture. Secondly, a bit of humility would help! It’s delusional to claim certainty about treatments for individual patients. Human beings just aren’t that predictable. They’re not machines. Thirdly, developing a greater wariness about categorisation. It doesn’t make sense to pretend there are treatments “which work” when we know there are no treatments which work for everyone. As Dr Roses of Glaxo said –

The vast majority of drugs – more than 90 per cent – only work in 30 or 50 per cent of the people,” Dr Roses said. “I wouldn’t say that most drugs don’t work. I would say that most drugs work in 30 to 50 per cent of people. Drugs out there on the market work, but they don’t work in everybody.

Finally, let’s put people back at the heart of our decision making. Health care can only be individual. It’s a mistake to consider statistics more useful than a patient’s personal experience.

Read Full Post »