The impact of such differences from real life presented by the trial setting will depend on the aim of the trial. Ultimately the purpose of any trial will be to provide an estimate of some aspect of the intervention’s usefulness in achieving its therapeutic goal in the population being studied: the estimate of interest may be the treatment effectiveness (aiming to mirror how it will be used in practice) or efficacy (use under ideal circumstances as prescribed). The importance of a trial’s external validity will therefore depend entirely on its research question, which may be broadly classified as either pragmatic or explanatory.
3.3.1. Efficacy trials
‘Explanatory’, ‘efficacy’ or ‘experimental’ trials aim to determine the biological or pharmacological efficacy of an intervention, in order words whether or not it produces
57
the expected result under ideal circumstances (48), with the primary purpose of furthering scientific knowledge (50). Efficacy, also referred to “proof of principle” (51) or “method-effectiveness” (52), measures the effect of the treatment actually administered.
Haynes (11) defines an efficacy trial as one whose aim is to determine whether a “treatment does more good than harm to those who take it”, whereas an effectiveness trial assesses this in those to whom it is offered. As such, a trial which aims to demonstrate the explanatory effect of treatment requires highly controlled conditions; thus explanatory trials tend to be designed to prevent blurring of the treatment effect by extraneous factors.
In seeking to determine whether a treatment works in terms of biological processes, an explanatory trial will therefore typically restrict patient eligibility criteria to ensure recruitment of those thought most likely to respond and adhere to study treatments, potentially incorporating pre-randomisation screening using a placebo (thus assuming that non-compliance is a general characteristic of the person rather that the drug) to weed out non-compliant individuals (32, 44). Thus, such trials typically involve a homogeneous population without concomitant medical conditions who have demonstrated a degree of compliance prior to recruitment into the trial and are therefore likely to remain in the study (42). As such, patients recruited into an explanatory trial may not be wholly representative of the general population of interest.
3.3.2. Effectiveness trials
The purpose of a “pragmatic” or “effectiveness” trial, on the other hand, is to ascertain whether a treatment which has previously been shown to be efficacious actually works in real life conditions. Pragmatic trials are therefore said to measure the effectiveness of
58
an intervention and to inform choices between treatments (50). Effectiveness is also known as “proof of practice” (51) or “use-effectiveness” (52) because it measures the benefit observed under the usual conditions of use. The participants of a pragmatic trial will therefore tend to be more representative of the entire intended target population, and as such may demonstrate varying degrees of compliance and be taking concomitant medications for other medical conditions (42).
Effectiveness of treatment incorporates not only the true efficacy of treatment but also the compliance with the treatment, and thus regardless of the actual efficacy of treatment, effectiveness of treatment will decrease as compliance decreases (47). The pragmatic estimate inherently includes the effect of the acceptability to the patient, which is regarded as an inseparable part of the evaluation of treatment effectiveness (53).
Schwartz and Lellouch (54) were the first to make a “distinction” between explanatory and pragmatic trials. Schwartz explains that treatments are administered within a particular context (mode of administration, side effects (and the consequential treatments), diet, auxiliary care, associated treatment). An explanatory approach requires that these contextual factors are balanced across randomised groups in order that the only difference between groups is the exact treatment received. In a pragmatic trial, however, optimal levels of these factors are set separately for each treatment group in order to inform a choice between two treatment modalities, such that these contextual factors become part of the treatment package.
Although trials may be broadly classified as pragmatic or explanatory, efficacy and effectiveness are not discrete, independent quantities, but instead exist on a continuum, precluding any sensible dichotomisation (48). Rarely will a trial adopt an entirely pragmatic or explanatory approach; for example, even the most pragmatic trial design
59
will typically include collection of more outcome data than in usual practice (55). The degree to which a trial demonstrates evidence of efficacy or effectiveness should therefore be measured on a sliding scale; there are potential characteristics that help to distinguish pragmatic and explanatory elements of a trial and a number of tools exist for determining where on this continuum a trial lies.
Gartlehner et al (48) proposed the use of a simple, validated tool composed of seven criteria to distinguish explanatory and pragmatic trials, namely the population and care facilities, eligibility criteria, principal outcomes, treatment modality and study duration, adverse event assessment, sample size considerations and analysis population. Alternatively, the PRECIS (PRagmatic-Explanatory Continuum Indicator Summary) wheel is based on ten domains (outcomes, patient adherence, practitioner adherence, primary analysis, eligibility criteria, flexibility of experimental/control interventions, practitioner expertise with interventions, blinding) and is intended to aid researchers in designing trials in line with their purpose (55).
Trial design features therefore determine the balance between efficacy and effectiveness, ideally at a point which provides satisfactory internal and external validity relative to the trial aims. However, a satisfactory level of interval validity is required in order to achieve external validity (48); thus all trials require at least the basic elements of trial design, such as randomisation and allocation concealment, to ensure validity of its conclusions. As such, the setting of even the most pragmatic of trials will inevitably be artificial in some respects.