This randomised controlled crossover trial, into the effectiveness of structured reply letters, has provided a high level of evidence on which to base our findings (Evans, 2003).
Randomised controlled trials are considered by most to be the most reliable and robust form of scientific evidence to influence healthcare policy and practice because they have the potential to reduce bias. Results of randomised controlled trials may be combined in systematic reviews which are increasingly being used in the conduct of evidence-based medicine.
Despite the known advantages of randomised controlled trials, one should not consider them to be impervious to predicament and it is important to note that this trial had several shortcomings which are outlined below.
The extent to which the results can be generalisable to outside the sample population is uncertain since those practitioners outside of the Mersey region were excluded. What is true in one region of the United Kingdom is not necessarily true for every region. In a different Dental Hospital, with different consultants and referring practitioners, with different needs, the results may have been different.
Single centre randomised controlled trials provide results that are low risk of error bias. However this is ranked behind multicentre randomised control trials and systematic reviews because it is based upon a single population. Skill mix, staffing, resources and expertise unique to this region will impact on the findings (Evans, 2003).
Many of the participants recruited into the trial were likely to have been involved in the previous regional audit conducted by Waring (2007). This means that participants recruited
108
into this study may have already been sensitized to the idea of structured reply letters and therefore could potentially make them unrepresentative of the population as a whole. However, the audit was conducted in 2005 so some 23% of practitioners would not have qualified at this point.
The demographics of participants were not necessarily representative of the wider dental population as 23% had undergone further specialty training. This compares to 10.5% of specialists in the UK dental workforce and equates to a two fold increase (General Dental Council Annual Report and Accounts, 2010). However, the results suggested that their responses were similar to the General Dental Practitioners.
Despite randomisation and stratification of the practitioners by consultant, into control and intervention groups there were a greater number of referrals to one consultant. The uneven distribution between consultants may have potentially biased the results since this consultants‟ letters may have been better or worse than their counterparts. However, the findings suggested they were similar on both letter formats with regard to letter preference. In future trials one could consider ceasing recruitment at a predetermined number depending on the sample size.
The trial was unable to adjust to the possibility that the results may have been biased at the level of the individual letter, because the quality of the letter may have been dictated to by the complexity of the case. For example, a practitioner may perceive a letter to be favourable because the problem, solution and action required was relatively simple and easy to understand, while a more complex set of issues may be perceived less favourably due to the difficulty of relaying the information and the practitioners‟ grasp and understanding of the case. However, this potential effect would have been diluted by practitioners receiving both letter formats based upon the same individual case.
The study was conducted within the dental specialty of orthodontics therefore the results may only be applicable to letters generated by an orthodontic consultant to referring
109
practitioners. It would however be interesting to see how the results are perceived amongst practitioners in other dental and medical disciplines.
Although this randomised controlled trial could be considered the “gold standard” in evidence what it cannot control for is the Hawthorne effect (Landsberger, 1958). This is a form of reactivity, whereby subjects improve or modify an aspect of their behaviour being measured, in response to the fact that they are being studied, not in response to a manipulation. Simply by testing the effectiveness of the structured consultant reply letter will in part improve its performance.
Despite the aforementioned weaknesses I believe this study still adds a considerable level of high evidence to the research base.
8.7
Suggested Improvements
One method of obtaining a true representative population sample would be to undertake a multi-centre randomised controlled crossover trial, including a broad range of dental and medical specialties, to assess the effectiveness of the structured consultant reply letter in a “real world” setting (Evans, 2003). Organising such a study would need a greater level of financial support and time commitment so was considered beyond the level of a specialist registrar‟s research thesis.
Recruitment of participants could have been made more equal by putting a limit on the number of participants assigned to each consultant. In this trial this would have meant no more than 25 practitioners could have been assigned to each of the three consultant orthodontists. Introducing such a limit to this trial may have affected the length of the data accrual period beyond what was achievable within the research component of the DDSc since our department has an unbalanced pattern of referrals to orthodontic consultants.
110
However, a perceived disadvantage of placing such a limit may have reduced the “real world” effect.
The trial could have been improved further by providing a greater level of structure and time to train the consultants in dictating, the structured letter format and the secretaries in using the structured letter templates. Such training programmes have been shown to be more effective at producing structured letters of greater quality and could have enhanced the overall standard and consistency of letter writing between individual consultants (Hook, 2006).
The letter could be further developed with appropriate computer software to allow for automatic text box re-sizing and improved page layout design on a case by case basis. Having had discussions with the secretaries on their perception of the letter, they felt this to be advantageous and would have saved more time.