Here’s two reasons why I’m concerned about using “Evidence Based Medicine” to make decisions about what should not be available on the NHS.
The first reason is we actually don’t have good convincing evidence for most of the treatments we currently deliver. The BMJ’s “Clinical Evidence” highlights this on its front page.
They categorise treatments into “beneficial, likely to be beneficial, trade-off between benefits and harms, unlikely to be beneficial, likely to be ineffective or harmful, and, unknown effectiveness. This in itself is controversial –
Dividing treatments into categories is never easy hence our reliance on our large team of experienced information specialists, editors, peer reviewers and expert authors. Categorisation always involves a degree of subjective judgement and is sometimes controversial.
In other words, the categorisation process is a matter of opinion and the opinions will be influenced by the authors’ values, beliefs and personal experiences.
Actually, it gets worse, because earlier on the same page, “Clinical Evidence” says
we wish to highlight treatments that work and for which the benefits outweigh the harms, especially those treatments that may currently be underused. We also wish to highlight treatments that do not work or for which the harms outweigh the benefits.
The thing is, in the real world, you can’t divide treatments into two such categories – those which work and those which don’t. There are NO treatments which “work” for every single patient. We are all individuals. It would be much better to publish the spread of responses experienced, and to resist this simplistic and delusional two-value “works/doesn’t work” categorisation.
And here’s the stunning statistic – 51% of the treatments are rated as “unknown effectiveness” (sometimes referred to as “insufficient evidence”). Were we to remove ALL of these treatments from the NHS, not only would we throw out a lot of babies with the bathwater, but does anyone seriously think we’d have a better health service?
The second reason to worry is just how appallingly distorted the evidence base is by the failure of companies to publish research which does not demonstrate positive effects of their drugs.
The German Institute for Quality and Efficiency in Health Care has published a study in the journal Trials.
they assessed hundreds of articles from journals and other sources, which covered areas including treatment for psychiatric disorders, pain, heart and circulatory disease, skin disease, cancer, and infectious diseases. A wide range of interventions was affected: from drugs and vaccines to medical devices such as ultrasound or devices for wound care.
What did they find?
an analysis of 90 drugs that had been newly approved in the US showed that they had been tested in a total of 900 trials. However, even 5 years after approval, 60% of these studies were unpublished. On the second level only selected outcomes from studies are published
Comparisons of protocols and journal articles of studies showed that in 40% to 60% of studies, results had either been completely omitted or analyses changed.
This does not only affect studies sponsored by the pharmaceutical industry. In their paper, the IQWiG authors also cite an analysis in which 2000 studies on cancer topics were analysed according to sponsorship. The proportion of published studies was extremely low: of the industry-sponsored studies, 94% were unpublished; however, even 86% of university-sponsored studies were also unpublished.
Irving Kirsch found similar significant distortions of the evidence base when he got hold of unpublished trials of anti-depressants.
The claims to certainty by the advocates of those who seek to make decisions about health care solely based on published research are misguided.
What’s the answer? Well, first of all bringing the current advice up to date by doing what these researchers have done, getting hold of the unpublished research and seeing how that changes the picture. Secondly, a bit of humility would help! It’s delusional to claim certainty about treatments for individual patients. Human beings just aren’t that predictable. They’re not machines. Thirdly, developing a greater wariness about categorisation. It doesn’t make sense to pretend there are treatments “which work” when we know there are no treatments which work for everyone. As Dr Roses of Glaxo said –
The vast majority of drugs – more than 90 per cent – only work in 30 or 50 per cent of the people,” Dr Roses said. “I wouldn’t say that most drugs don’t work. I would say that most drugs work in 30 to 50 per cent of people. Drugs out there on the market work, but they don’t work in everybody.
Finally, let’s put people back at the heart of our decision making. Health care can only be individual. It’s a mistake to consider statistics more useful than a patient’s personal experience.
Heroes Not Zombies 
amen to that! not all treatments work for everyone. how many drugs were deemed “safe” by the FDA only to be recalled a few years later due to horrendous side effects and even death?? it needs to be up to the individual as to what treatment they want to pursue.
There is so much that contradicts the claims of ConMed that if those standards would rigorously be applied, more that half the treatments would be thrown out. Then we are left with what?
Evidence-based? The evidence shows that 90% of drugs do not work for the conditions they are prescribed for. An example is the use of antibiotics to treat a cold.
Recently, industry captains cynically admitted that 90% of drugs don’t work for what they are prescribed for, yet these same people continue to push them. If your local crack dealer does the same, he gets arrested.
The Pharmaceutical industry may have legal sanction to sell drugs, but they have no moral sanction to deliberately poison the people.
An anecdote (I think it’s a true story, but can’t find source): For his retirement present, the scientists at Cold Spring Harbor decided to give Dr. Jim Watson his own genome, fully mapped. When the analysis was done, however, it turned out that his genes coded for over 20 diseases/conditions that should have killed him years ago. Fortunately he didn’t know about it then, and just kept on living 🙂
Here’s to human unpredictability!
As for individualized medicine, my father works as a Senior Scientist at the Cedar-Sinai Hospital in LA, where they have access to a huge DNA database and medical history of thousands of patients, and are working on this very issue. He echoes your sentiments on the benefits and even necessity of individualized medicine, but agonizes over the fact that pharmaceutical companies just aren’t buying it. Literally. For them it is much more profitable to stamp out a thousand “heal-all” pills that won’t work for half the population, than to do all of the research/testing/production for 20 different types of medications, where each would target only a very specific (and small) population.
The trick is to make individualized medicine profitable for the industry (unfortunately, the health of the people buying the treatment is not sufficient reason to move in that direction). My father proposes to go to insurance companies, and convince them that with individualized medicine you will have fewer recurrent conditions, relapses, complications, etc. (which means less claims sent to the agencies), and that the insurance companies in turn should be the ones pressuring the pharmaceutical industry to buy into this idea…
Excellent analysis. Just yesterday I wrote a post on Evidence Soup about how easy it is to love the “evidence-based” concept. But turning it into a repeatable process? Not so much.
Great post Bob really resonates with me working in a busy orthopaedic clinic where weak so called evidence based protocols and social mobility swamp individual narratives. Hummility ? well not part of the training…… Delusion ? if it doesnt work its the patients fault , very little need for self reflection with all that evidence around !!
Well said Bob! The good thing is that it seems these arguments about the limits of EBM are coming from across medicine…can we hope that decision makers might take notice? I guess we are all tying to make them take notice. Cheers on keeping up the fantastic dialogue, it is energising to read.
[…] Many researchers have found that, using Freedom of Information requests, it turns out that drug companies just don’t publish all the data. (see my second point in that link to an earlier post). If only the studies showing better results […]