Heroes Not Zombies

Sir Michael Rawlins, Chair of the National Institute for Health and Clinical Excellence, has recently criticised our use of RCTs (Randomised Clinical Trials). This particular type of research study is promoted by many decision makers as a “gold standard” but Sir Michael listed a number of objections to this practice. He gave examples of situations where RCTs are either unneccessary or impossible, questioned the practice of failure to complete studies (30% of recent trials in oncology have been stopped early because of apparent benefit from the treatment), and expressed his concern about the high costs of conducting such research. However, perhaps his most interesting remarks related to the generalisability of RCT findings and the elevation of this methodology above others.

RCTs are often carried out on specific types of patients for a relatively short period of time, whereas in clinical practice the treatment will be used on a much greater variety of patients – often suffering from other medical conditions – and for much longer. There is a presumption that, in general, the benefits shown in an RCT can be extrapolated to a wide population; but there is abundant evidence to show that the harmfulness of an intervention is often missed in RCTs. Sir Michael argues that observational studies are also useful and, with care in the interpretation of the results, can provide an important source of evidence about both the benefits and harms of therapeutic interventions. These particularly include historical controlled trials and case-control studies but other forms of observational data can also reveal important issues. Sir Michael rejects the trend to grade various kinds of clinical trials and studies on scales of merit which he says has come to dominate the development of some aspects of clinical decision making. “Hierarchies attempt to replace judgement with an oversimplistic, pseudo-quantative, assessment of the quality of the available evidence.” Sir Michael believes that arguments about the relative importance of different kinds of evidence are an unnecessary distraction. What is needed instead is for “investigators to continue to develop and improve their methodologies; for decision makers to avoid adopting entrenched positions about the nature of evidence; and for both to accept that the interpretation of evidence requires judgement.

The generalisation issue has been raised so often it’s surprising that it is dismissed so easily by decision makers. Not only are trials artificially created clinical experiences but the entry and exclusion criteria for patients admitted to such trials always excludes “co-morbidity”. In other words if somebody had more than one clinical problem they won’t be admitted to the study. However, in actual clinical practice patients presenting with more than one disease at a time occurs so frequently that it is considered usual. Other criteria also make it difficult to generalise the results of RCTs. But, Sir Michael’s main criticism is about safety. The problems with a new drug are rarely picked up by RCTs. The problems indeed are not usually evident until many more patients have taken the drug in real life circumstances. So if we want to understand the real life potential of a drug, then we need larger, observational studies to pick up the side effects and harms which the drug can cause.

His final point is about the use of “hierarchies of evidence” which give greater credence to some methodologies over others. His contention that this distorts both the nature of evidence and that it is used as a substitute for clinical judgement is an important one