Methods to measure system integration vary considerably and the results have been mixed (Raina et
(Lloyd & Wait, 2006). Ling, Bardsley, Adams, Lewis, & Rolan (2010) proposed a research protocol
for a portfolio of evaluation methods, both quantitative and qualitative, to measure the impact of
integrated care projects in the United Kindgom. Their review of the literature suggests that integration
initiatives improve care but are mixed in their impact on costs. Consistent with other researchers they
caution that integration results are highly contextual, (Kodner & Spreeuwenberg, 2002; Minkman,
Ahaus, & Huijsman, 2009; Shaw et al., 2011).
Some early studies suggest that certain outcomes should result from integration such as
“growth of primary care and multi-specialty practices”, “downsizing of acute care capacity” and
“consolidation of programs and services” (Shortell et al., 1994), however the relevance of those
indicators is heavily influenced by aforementioned context of the health system on which they were
modeled. Wan, Ma, & Lin (2001) examined 100 integrated health networks (IHN) in the United
States which were administratively and financially integrated. Using structure equation modeling,
information integration (a scale variable indicating absence or presence of “integrated information
systems”) was not found to be associated with efficiency, and integration efforts did not impact
profits or efficiency. However, the dependent variable in this study (presence on a list of “top 100
IHNs” determined by hospital utilization, contractual capabilities, service and access, physicians,
financial positions, and system-wide integration) was created by a marketing group with no
explanation of the algorithm by which IHNs were evaluated.
Browne et al. (2004), using a previously developed ordinal scale measuring integration depth,
proposed the Human Service Integration Measure. This indicator is derived from information such as
the number of services in the partnership, the “scope” as measured by awareness of or links between
those services, and “depth” of the integration. The depth is measured by a 5 point Likert scale of non-
Kingston, Grdisa, & Markle-Reid (2007) amended the model to include a measurement of the actual
or observed assessment of the expected structural elements such as scope, depth and extent of
network integration, the functioning of the network, and its outputs. This model appears loosely based
on Donabedian’s structure-process-output model of healthcare quality measurement. Consistent with
this approach, integration measures are grouped by three dimensions:
1. Precursors of integration;
2. Intermediate outcomes or internal process variables; and
3. Outcomes measures that determine the extent to which systems are fulfilling their
purpose.
Other studies use the BSC tool when measuring system integration and its impact (Armitage et
al., 2009). As mentioned in Section 2.1.1. the BSC is used to capture a panoramic view of an
organization or system, and to develop insight into its progress towards strategic goals by analyzing
four perspectives: financial performance, customers, internal processes, and learning and growth.
These perspectives may be adjusted when transitioning the tool to the non-profit sector or other novel
setting but the essential mapping of each perspective to strategic goals does not (Kaplan, 2001).
Balance is achieved through the selection and surveillance of leading (performance drivers) and
lagging (performance outcomes) indicators, and between financial and non-financial indicators. This
notion allows for “feedforward” and “feedback” controls, and to balance retrospective “corrective
actions” with prospective “improvement actions” (Holmberg, 2000), a concept which is crucial to
timely evaluation of complex interventions such as electronic health information exchange.
Identifying leading performance indicators in a system is inherently more difficult as it is
for instance, a retroactive measure of quality emergency room care might be the time it takes for a
patient to be admitted if required. A leading indicator of that quality might be the number of patients
in the hospital who are designated as requiring an alternate level of care. Were they moved to that
care setting they would free a bed and reduce the time it takes for an emergency patient to be
admitted. The former measure informs the hospital they have a problem, the latter indicator gives
them information to prevent it. The more time between indicator and effect, the larger the window of
opportunity to implement corrective action. Leading indicator’s capacity to anticipate outcomes and
provide real-time feedback makes it particularly powerful in large, complex systems such as
healthcare.
Canada’s provincial departments of health are responsible for the delivery of healthcare
services to their constitutents and therefore monitor macro-level or system-level performance
measures including, though not consistently, system integration (Green & Moehr, 2000). The Strategy
and Alignment Branch of Ontario’s Ministry of Health and Long Term Care (2011) has developed a
BSC- inspired mapping of performance indicators to government healthcare system priorities.
Amongst outcomes such as effectiveness, safety, equity and a focus on population health, integration
is presented as contributing to healthier communities by improving access to care. The three proxy
indicators of system-level integration in that matrix are “percentage ALC (Alternative Level of Care)
days”, “ALC Days”, and “Hospitalization Rate for Ambulatory Care Sensitive Conditions (ACSC) ”
measured using aggregated hospital sector data.
There is anecdotal evidence of a lack of integration co-existing with high ALC rates in the
Ontario healthcare system (The Change Foundation, 2010b). Unlike many system indicators an
Ontario benchmark for the percentage ALC bed days indicator is listed at the time of writing, as 8.5%
rehabilitation hospitals and 0.3% for specialty hospitals, with a “theoretical best” of zero (Ministry of
Health and Long Term Care, 2012) . It is not without irony that this “system” integration measure is
currently used to hold Ontario’s acute care hospitals to account for what is deemed to be inefficient
institutional service delivery. Yet nominally and by its very nature, a high ALC rate is more reflective
of downstream inefficiency and lack of capacity in regional health systems in Ontario. Nevertheless,
this indicator is perhaps the only widely reported system integration measure in the Ontario system,
and thus may be useful for validation of this study’s final integration indicator.
The selection of these system integration measures is generally affected by the need for
parsimony in performance monitoring measures, attribution to a hypothesized chain of events in a
causal pathway, and availability of data. However, there are few tools available to calibrate the
correlation of the indicator with the phenomenon. Furthermore, those measures which are used, such
as ALC and ACSC (above), have an almost exclusively retrospective performance lens providing
little opportunity for governors and managers to adjust resources and priorities in real-time. It would
appear that, in the context of Ontario’s healthcare system at least, there is a clear need for further
research into measures of healthcare system integration which not only drive accountability but are
sufficiently timely to influence future priority-setting.
As previously noted, the literature has alluded to the need for robust information systems to
facilitate information exchange in integrated systems. Interoperability is the concept we use to
understand how systems communicate with each other (Gibbons et al., 2007); it also facilitates the
electronic exchange of information of interest to this study. Interestingly, integration and
interoperability in the business literature are often used interchangeably and refer to the “ability of
seamlessly” (Mouzakitis, Sourouni, & Askounis, 2009, p. 128). It is to this concept that we now turn
to gain a theoretical understanding in order for it to be incorporated into our explanatory model.