D ocum ents are an essential aspect o f healthcare fo r clinical, professional, adm inistrative
and regulatory requirem ents. Yin (2 0 0 9 a ) noted th e usefulness o f docum ents fo r
strengthening th e internal validity o f case study research. This is achieved by augm enting
o th e r sources o f evidence, triangulation purposes, and understanding com m unication
channels betw een various professional groups. According to Yin (2009b ), docum ents are
a "stable, unobtrusive and exact" source o f data th a t offers a broad coverage over tim e ,
which also provide insights into key events and settings. H ow ever, th e y can be difficult to
retrieve or access, m ay be incom plete, and m ay contain reporting bias. Access to relevant
docum ents was granted by th e study Trust's relevant senior managers and clinicians.
D ocum ents review ed included letters, clinical audits, Trust reports and policies, clinical
pathw ays and national guidelines, training m aterials, internal m e m o ran d u m , em ail
new sletters and In tran et com m unications, th e study Trust's annual reports and quality
accounts fo r th e relevant periods, and technology strategy briefings (see Appendix 1).
A dditionally, archival docum ents such as historic policy and service reports, lists o f key
people, press releases, previously undertaken surveys and audits w e re also accessed to
gain fresh insights, trian g u late evidence sources and im prove internal validity (Yin, 2 0 0 9 b ).
M o s t archival docum ents w e re publicly available on th e study Trust's w ebsite and in tra n e t
and reports from regulatory bodies. Appendices 1.1 to 1 .4 shows th e range o f relevant
docum ents and related artefacts th a t w e re used to support th e data collection process.
3.4.3 Ethics approval
Before undertaking th e research, ethics approval was sought fro m The Open U niversity
Hum an Research Ethics C om m ittee (HREC). This process involved th e submission o f a
participant in form ation sheet (PIS) detailing th e purposes and conduct of th e research,
h ow data and participants' rights w ould be protected and an inform ed consent fo rm (ICF)
th a t was signed by both th e researcher and each individual participant. The docum ents
th a t w e re subm itted fo r th e HREC ethics application are provided as Appendices 4 and 5.
Following HREC approval (see Appendix 6) all th e docum ents listed above w e re subm itted
to th e study Trust's Research and D evelopm ent d e p a rtm e n t. The researcher also had to
com plete th e Gafrec guidance form as required fo r all research projects th a t involve
collecting data w ithin th e NHS (see Appendix 7). The research project was approved by
th e study Trust's Research and D evelopm ent d e p a rtm e n t on condition th a t th e research
protocol was adhered to. M eetings w ere subsequently held w ith senior m anagers and
clinicians fro m th e relevant dep artm en ts w h o confirm ed th e ir c o m m itm e n t to th e project
and approved access to th e settings w h ere th e selected CDSSs had been ad o p ted .
3.5 Data analysis techniques
As recom m ended by Eisenhardt (1989), data collection and data analysis fre q u e n tly
overlapped to gain fam iliarity w ith th e data and m ake prelim inary assumptions at th e
point o f collection. This process helped to speed up th e data analysis process and allow ed
flexibility, such as changing data collection techniques w h e re necessary. Changes included
adding in terview questions to probe new lines o f enquiry and addressing n e w issues as
th e y em erged (Eisenhardt, 1989). Corbin and Strauss's (1 9 9 0 ) grounded th e o ry approach
was adopted to a llo w th e research erto continually im m erse w ith th e data to gain a d e e p e r
understanding o f each case as th e research developed.
3.5.1 Within case analysis
The case analysis adopted an approach based on Eisenhardt's (1 9 8 9 ) w ith in case analysis.
This tech n iq u e involves com m encing data analysis during th e data collection stage. It
enables th e researcher to m anage large volum es of data collected fro m interview s and
aides a d etailed , descriptive w rite up o f each case as reco m m en d ed by M iles and
H uberm an (1 9 8 4 ). Consequently, a "rich fam iliarity" (Eisenhardt, 1989: 5 4 0 ) can be
established which helps to accelerate cross case analysis. Flyvberg (2011: 3 0 1 ) described
this fo rm o f analysis as m ore d etailed, com plete and richer "depth fo r th e unit o f study
th a n cross unit analysis". W ith in case analysis was im p o rta n t because it allow ed th e
researcher to undertake "intensive analysis" o f each CDSS. Field notes and running
co m m en taries recorded during th e research process w e re also an im p o rta n t p art o f th e
data analysis process (van M aan en , 1988). These notes covered observations, analysis and
sum m aries o f daily activities, such as em erg en t ideas, thoughts, initial impressions and
perceptions w e re im m ed iately recorded in a descriptive m an n er (Eisenhardt, 1 9 89).
In te rv ie w and docum entary data w ere fram ed into th e key th em es id en tified fro m th e
lite ra tu re review as suggested by Yin (2009b ). This was fo llo w ed by ite ra tiv e analysis
(Eisenhardt, 19 8 9 ) o f in terview data using bullet points, spreadsheets and ta b u la r
displays, lists o f issues according to in terview respondents and professional groups and
com paring e x tre m e /p o la r them es (M iles and H uberm an, 1 9 8 4 ). The lists, spreadsheets
and tables w e re also im p o rtan t because th e y allow ed th e researcher an o p p o rtu n ity to
condense several pages o f narrative in terview data into m anageable fo rm ats. Sum m aries
o f field notes w e re also com pared w ith interview data and fo rm al sources to confirm or
re fu te th e researcher's initial impressions. D ocum entary analysis o f process maps and
organisational structures w e re also p erform ed to establish any causal links or gain fresh
Using Yin's (20 0 9 b ) "em bedded m ultiple unit analysis", all th re e CDSSs landed them selves
to th re e levels o f analysis; th e study Trust, th e respective CDSSs, and professional levels.
Data analysis m ainly focused on th e CDSS evaluations undertaken in each case, but also
paid a tte n tio n to organisational, professional and social factors as suggested by Kaplan
(20 0 1 b ) and Greenhalgh and Russell (2 0 1 0 b ). O th e r considerations included external
influences from th e g o vern m en t and regulatory bodies such as NICE, current and previous
evaluations o f related systems w ithin and outside th e study Trust, previously undertaken
clinical audits and surveys relating to th e service supported by th e CDSS. Roger's (1995)
five stage adoption process (awareness, persuasion, decision, im p lem en tatio n , and
confirm ation stages) w e re used to provide structure to data collection and analysis. Each
CDSS was w ritte n up using a descriptive fra m e w o rk [Lynd and Lynd, 1929] to identify
causal links betw een d iffe re n t issues as outlined b elow [Yin, 2 0 0 9 ]. The follow ing fo rm a t
was m aintained as discussed by Eisenhardt (1989):
• Descriptive background o f each CDSS as described by th e stakeholders and
through d ocum entary evidence
• Collating evidence from supporting docum ents and th e e n viro n m en t w h e re th e
CDSS was used, associated health inform ation systems, non-com puterised
systems such as paper-based clinical pathways, protocols and o th e r relevan t
sources o f data
• Perceptions and expectations o f th e key stakeholders ab o u t CDSS evaluation, as
w ell as th e ir know ledge and experiences
• Assessment o f th e interactions and responsibilities b etw een d iffe re n t
stakeholders and professional groups in relation to CDSS usage and evaluation
• A narrative of how th e CDSSs w e re evaluated a t d iffe re n t stages o f th e ir adoption
lifecycle
• Assessment o f opportunities fo r learning fro m evaluations at
practitioner/clinician, d ep artm en tal and organisational levels
• Discussion and analysis o f issues em erging fro m each study
• Cross study analysis o f sim ilarities and differences b etw een identified them es
3.5.2 Cross case analysis
Cross-case analysis com m enced w ith d etailed chronological descriptions o f each CDSS.
Sum m aries fro m each case w e re categorised using d iffe re n t fo rm ats such as lists,
M icro so ft W ord® tables fo r direct comparisons to m atch patterns, establishing divergent
issues and identifying frequency o f issues (See Appendix 8). Com parable issues th a t w ere
identified in th e literatu re, such as perceptions o f CDSS user groups, barriers to evaluation,
evaluation m ethods, and level o f decision support w e re used fo r cross-case
categorisation. Lists o f sim ilarities and differences b e tw e e n th e cases and unique insights
fro m each data collection technique w e re used to corro b o rate evidence. These divergent
techniques (Eisenhardt, 1989) w e re applied to analyse data beyond initial impressions
during data collection and those from th e w ithin case analysis stage. Em ergent issues such
CDSS effects on w o rkflo w patterns, contextual differences and influences fro m national
initiatives such as clinical guidelines supporting individual CDSSs w e re also explored across
th e cases.
To conclude th e cross case analysis, sum m aries o f field notes and running com m entaries
fo r each case w e re com pared fo r similarities and differences, taking advantage o f th e
already established fam iliarity w ith each case as an individual e n tity (Eisenhardt, 1 9 89).
Key com parable dimensions across cases related to CDSS links w ith guidelines and th e