Evaluation approaches vary across reported telehealth case studies but they generally combine clinical outcomes and patient satisfaction. Many were essentially pilot studies intended to explore whether or not a particular telehealth idea would work in practice. These often had to deal with sceptical clinical participants as well as volunteer patient participants and the case study by Boulanger and colleagues is an example (Boulanger, Kearney et al. 2001). They saw 19 patients for a post-discharge follow-up after hospital treatment for trauma (gunshot wounds, blunt trauma) over 22 appointments. Their paper presents the average duration of consultation and lists examination instruments and imagery used. In 15 appointments there was no further follow-up. The other seven appointments generated referrals to further telemedicine follow-up or other face-to-face treatment. One author performed 20 of the 22 telemedicine assessments and
acknowledged that this produced a “biased assessment of physician satisfaction” with much prior “scepticism as to the use of this new technology” amongst his colleagues. The exit interviews with the patients showed high levels of satisfaction. In particular, all patients gave the maximum positive rating to the assertion that “Telemedicine makes it easier to get medical care” and several said that they would not have made the effort to travel the long distance back to the hospital where they had been treated to attend this follow-up appointment.
A second example concerns delivery of Cognitive Behaviour Therapy (CBT) to rural and remote mental health clients (Griffiths, Blignault et al. 2006). A psychologist based in a regional city conducted between six and eight Cognitive Behaviour Therapy
sessions with each of 15 mental health clients using a videoconference system, with the clients’ case managers present at the remote site. Using a standard Mental Health Inventory they found significant improvement in their clients’ scores before and after the treatment. The clients and case managers rated the experience as “average” to “much better than average” on a five-point scale. The authors noted that their study
“demonstrated that it was possible to deliver CBT via videoconference” for this range of patients but that, in the absence of a control group, they could not quantify the
improvement. They noted that the case managers had not had any prior experience with videoconferencing (and by implication, though they did not say so, no experience with telehealth) and they conducted a single training session for each case manager prior to this trial. They did not describe any prior telehealth experience on the part of the psychologist. The PC-based videoconferencing system used 128 kilobits/second telephone lines and this low bandwidth caused “a slight delay in speech and movement”.
In his review of telemedicine and health care, Bashshur discussed systemic
organisational barriers to traditional evaluation methods for a deployed technology (Bashshur 2002). He noted that “many projects have been funded for the short term” and have “incomplete or nonexistent plans for long-term sustainability”. This lack of mature telemedicine programs prevents “adequate and definitive cost-benefit analysis, particularly in terms of health outcomes, patient-borne costs and total costs”. He continued this discussion in a paper on telemedicine evaluation (Bashshur, Shannon et al. 2005) where he looked at public policy on investment in telemedicine programs, noting that it should be based on “scientific evidence of their benefits and costs as compared with alternatives”.
Pawson and Tilley noted in their book on evaluation that program evaluation of large- scale, publicly funded interventions is very difficult (Pawson and Tilley 1997) and that accumulation of results across studies is difficult because each study typically has its own foundations, assumptions and methodologies leading to incompatible descriptions of outcomes. They also noted that the high-level reporting typical of such program evaluations tends to mask the underlying mechanisms driving the phenomena being evaluated and that knowledge of these mechanisms is often the key to successful replication of the program elsewhere.
Klecun-Dabrowska and Cornford addressed the role of public policy in the emergence of telehealth using the United Kingdom as their case study (Klecun-Dabrowska and Cornford 2000). They observed that policy documents set the context for the
development of new technologies, such as those involved in telehealth, and that these policies provided the framework in which telehealth would be judged a success or a failure. As an example, if “policy emphasises efficiency and cost-cutting, services that do not contribute to these goals, even if offering a better service for the population, may be deemed as failures” (ibid.). This is an important point to keep in mind when
considering the cost-benefit analyses that Bashshur and colleagues advocated (Bashshur 2002; Bashshur, Shannon et al. 2005) because the outcome of a cost-benefit analysis depends entirely on the values that are allocated to those costs and benefits.
In medical informatics the gold standard for evaluation is the randomised controlled trial (Friedman and Wyatt 1997). This standard can indeed be applied to health
situations that have a heavy use of Information and Communications Technologies. An example from the use of simulation technologies for surgical skills training, described in Section 2.4.1 above, illustrates this point (Seymour, Gallagher et al. 2002). In this study, 16 trainee surgeons were divided into two groups which were controlled for potentially confounding factors. One group received standard laparoscopic training for a basic procedure and the other group received simulator-based training for the same procedure. In a double-blind trial in the operating room, the simulator-trained group outperformed the control group in all aspects of the evaluation of their surgery. The
study produced results strong enough to alter the attitude of the peak surgical body for the USA towards simulator-based training (Gallagher, workshop presentation, Royal Australasian College of Surgeons Annual Scientific Congress, Brisbane, 2004)
In one review of the methodology of telemedicine evaluations the authors said that “the focus of the evaluation is on the patients’ health status” and they advocated that these trials should be conducted in a randomized controlled manner comparing conventional and tele-treatment (Huis in 't Veld, van Dijk et al. 2006). However, as MacFarlane and colleagues had earlier pointed out, a randomised controlled trial “can determine only whether an intervention works according to predefined criteria. It cannot explain why or how an intervention succeeds or fails”. They argued for the benefits of a qualitative approach to telemedicine research (MacFarlane, Harrison et al. 2002). They studied telehealth consultations between primary and secondary health providers by
interviewing 15 hospital specialists, 24 general practitioners and 30 patients and by recording 60 tele-consultations. Their results showed that the participants (specialists, general practitioners and patients) had “different perceptions of the same tele-
consultations” which in turn were different from the perceptions formed by the researchers conducting the evaluation. They found that some groups thought a consultation went well but others thought it went not-so-well. Awareness of this difference only emerged during the data collection.
Nelson and Palsbo discussed the issue of “diagnostic equivalence” studies in
telemedicine using data from five specialty telehealth clinics (Nelson and Palsbo 2006). The issue here was whether the clinicians were able to make a correct diagnosis of a patient’s condition using telehealth systems. The authors focused on issues of study design, patient and clinician involvement, measurement issues and the role of the “presenter” (the person presenting the patient to the distant clinician). Their model of telemedicine was one of outreach to “underserved populations who would otherwise not receive specialty medical services”. They looked at the initial diagnostic stage of the patient engaging with the health system, i.e. at the start of the patient’s health trajectory (Corbin 1998).
Smith and colleagues presented a study which compared diagnostic outcomes between telehealth and face-to-face consultations (Smith, Dowthwaite et al. 2008) in paediatric otolaryngology. A total of 68 patients received an initial diagnostic consultation with a telehealth system (videoconference plus specialist camera for ear, nose and throat examination) followed some time later by a second full examination at the hospital. They mentioned in passing that “outcomes, including patient satisfaction … are all substantially improved with the use of telemedicine” but their primary focus was on the agreement between the diagnoses arising from the pairs of consultations, which in their case was high.
The work presented in this thesis is located much further along the patient’s health trajectory than those of Nelson and Palsbo or Smith and colleagues. Patient data are already present and the telehealth event can draw on existing patient-clinician
relationships. Nelson and Palsbo made an interesting observation about the role of the “presenters” (their term for the assistants located remotely with the patients) who
“needed to be socialized to the rationale for strong research design in order to encourage strict adherence to the protocol” and they noted the resulting difficulty of using real-life presenters in their research. They also wondered whether the clinician and patient could develop sufficient rapport during the telehealth session to adequately complete the diagnostic interview.
By contrast, in the hospital pilot trial described in this thesis the consultations occurred in the context of an existing relationship between the surgeon and patient/family. The research design was for the consultation to proceed naturally and appropriately for the particular patient’s situation. The role of the presenters (i.e. clinic assistants) was to look after the best interests of the patients according to their own professional
judgement and the research was based on observing what they actually did. Smith and co-authors do not discuss the interactions between the specialist and remotely located patient, family and assistant other than to note that the specialist could “undertake a complete patient history and ask specific questions during the videoconference appointment”.