ARTE POPULAR

Software development is an important determinant of patient safety in all types of health IT. Opportunities exist for vendors, aided by users, to improve safety in the different phases or activ- ities of the software design and development life cycle. While in theory these activities are iden- tifiable, in practice the boundaries between them are not well defined and require varying levels of intensity to complete.

Health IT Vendor Design Activities

The key activities in the software life cycle are identifying requirements, software develop- ment and design, and testing (see Table 4-3).2

TABLE 4-3

Health IT Vendor Design Activities

Activities/Phases Features Opportunities to Improve Safety Requirements

Activity

– Developers articulate what the software must do and the circumstances under which such behavior is appropriate

– Developers articulate what the software must not do when safety is an issue – End users of the software

must be intimately involved in all aspects of the requirements activity

– Clinicians communicate their needs

– Clinicians identify data that must be captured or imported, with any requirements for conversion and validation such as full text entries

– Prototype testing – Safety analyses

Software Development

– Involves actual program- ming or coding that re- fl ects the software design – Software development

is often undertaken iteratively with testing – Results from testing

are used to inform another round of software development

– Iterative testing identifi es unintended consequences for early revisions and informs the next round of development

Design of User Interface Activity

– Designers defi ne the structure of the software – Software engineers decide

on the appropriate techni- cal approaches and solve problems conceptually

– Clinicians give feedback to designers about eff ec- tiveness or improvements needed for usability testing

2 _{Traditionally, these activities are called “phases” (e.g., requirements phase, design phase); this report adopts the “activities” terminology to} emphasize the point that, although these activities are conceptually distinct, some of them may be occurring simultaneously from time to time.

TABLE 4-3 Continued

Activities/Phases Features Opportunities to Improve Safety Testing Activity – Examines the code that

results from software development

– Examines code functional- ity and the extent that the code complies with the requirements

– Testing is often split into three separate subactivities:

• Unit testing, where individual software modules are tested • Integration testing,

where individual modules are assem- bled into an integrated whole and the whole assembly is tested • Acceptance testing,

where the entire assembly is tested to determine compliance with the requirements

– Involving clinician superusers can reveal code functionality – Involving clinicians in the

testing process provide an avenue for identifi cation and correction of code defects and workfl ow risks

– Dress rehearsals—real use by real users with actual data under realistic conditions— allow the clinicians opportu- nities to identify previously invisible fl aws

NOTE: The design of user interfaces illustrates the concurrent nature of some of the activities of software development—users have an important stake in the design of user interfaces, and their needs must be expressed in the requirements activity.

Software Requirements and Development Activities

Traditionally, the software development process begins with the explicit statement of what the software is intended to do and the circumstances under which such behavior is appropriate, otherwise known as the performance requirements. Clinicians communicate their needs in de- tailed statements based on evidence of safe practices whenever possible. Clinicians and software developers need to communicate safety needs and expectations for the clinical environment and the health IT product.

Traditional writing of requirements, although precise, can be a tedious task. When require- ments become very complex, it is difficult to be confident that they actually describe what the users want. At times, a sequence of prototypes can be an alternative that allows users to see what a proposed version of the software would actually do. User input can provide for a more effec- tive and accurate product. However, in a safety-critical area, even if the software task is defined by a sequence of prototypes, it will still be necessary to define the task sufficiently rigorously to permit adequate testing. The software functionality, whether developed from requirements or from experiments with prototypes, has to be understood as part of the process of delivering care and ought to reflect the desired changes to that process when it is revised and adapted in the light of operational experience.

Articulating requirements is often difficult. A critical path for identifying and validating re- quirements for software functionality includes assessment of current-state workflow, mapping the current state to the desired future state, and devising a plan to identify and address the gaps between the two. A process like this includes observation and documentation of the real-life

workflow as well as interviews with the clinician. Validation that the software meets redesigned workflow needs is accomplished in multiple phases of testing to obtain iterative feedback from users, as discussed later in this chapter.

Many organizations purchasing health IT products define their requirements only once— immediately prior to purchasing the product off the shelf. Compared to organizations that can specify their requirements to their vendors, organizations purchasing products off the shelf face a somewhat different challenge—that of assessing whether an existing product is suitable for their organization.

Best practices for developing software emphasize systematicity and quality control. Software developers should identify and record significant risks of failure in development or in perfor- mance and articulate a plan to reduce them. They also need to have an explicit definition of the quality criteria for the software, to develop and follow version control procedures, and to track reported bugs and index them. Performing a complete safety analysis requires a comprehensive inventory of the possible clinical harms that might befall a patient and an understanding of how the health IT software might contribute to those harms.

Design of User Interface

Although definitive evidence is limited, the committee believes poor user-interface design is a threat to patient safety (Thimbleby, 2008; Thimbleby and Cairns, 2010). The user interface is one of the most important factors influencing the willingness of clinicians to interact with EHRs and to follow the intended use that is assumed to promote safe habits. The more functional the user interface, the more it enhances usability of the product. Inadequate user interfaces can lead to error and failure. The interface is intended to facilitate a desired clinical task; when the clini- cian cannot readily locate data or perceives the amount of time required to perform a function is too long, the user interface needs to be evaluated. Poor interface design that detracts from clini- cian efficiency and affinity for the system will likely lead to underuse or misuse of the system (Franzke, 1995).

The goals of user-centered design are to create an efficient, effective, and satisfying interac- tion with the user. The interface design starts with an understanding of human behaviors and fa- miliar work patterns. Human nature is such that busy clinicians will trade thoroughness for effi- ciency, or they will modify their behavior to achieve efficiency. Shneiderman has identified eight heuristically and experientially derived “golden rules” for interface design that, if followed, sup- port the principles of EHR usability (Shneiderman et al., 2009), including consistency for similar tasks; universal applicability for a wide range of expertise; feedback for every user action; com- municating closure of an action sequence; design to prevent errors; allowing easy reversal of ac- tions; avoiding complex data entry and retrieval sequences; and reducing memory load (see Ta- ble 4-1). Although it is important to develop and follow principles for safe design interfaces, it is equally important that designers do not follow a formulaic checklist. Instead, designers need to continually interact with users to discover and address the safety issues unique to each clinical environment. Formal usability testing during development is essential.

In Vivo Testing: Uncovering Use Error Versus User Error

Often, miscommunications between developers and users leave critical ambiguities that can only be discovered through testing. Therefore, it is critical to test health IT during all stages of development to determine whether user requirements have been translated into software that ac-

tually does what the user wants. An important source of information for obtaining information about meeting clinician needs is operational prototype testing. The first versions of software rare- ly meet clinician needs fully, and observing how a clinician uses a software prototype will yield a great deal of information about what the clinician actually does and does not find useful and appropriate. Such observations are fed back to software developers, who then revise the software so that its performance better reflects user needs and safe practices as expressed in an operational context.

However, testing cannot be oriented solely to determining if software does what it is sup- posed to do when users follow all of the proper steps. Because users do make mistakes, a signifi- cant portion of testing must be devoted to seeing if the software responds properly when the user does something unexpected. For example, the user may enter data in an unexpected format. Test- ing is also a longer-term process where the experiences of users based on reported events and workarounds is considered as part of a postmarketing program designed to improve the design of software. To maximize the user–designer feedback loop, EHR products might include a “report here now” button on each screen wherein the user can indicate that a display was confusing, a workflow was cumbersome, or some other way that the design did not support optimal clinical care.

Software Implementation and Postdeployment Activities

Portions of the software life cycle led by vendors in partnership with users influence safer use of health IT such as training prior to implementation, addressing problems that appear during testing and implementation of software, and planning for the ongoing maintenance and upgrade activities that directly impact users. These activities occur not only as part of software develop- ment but also during implementation and subsequent phases of use in an organization.

TABLE 4-4

Health IT Vendor Implementation and Postdeployment Activities

Activities/Phases Features Opportunities to Improve Safety Software

Implementation Activity

– Installing the software in the users’ organization – Users are trained to use

the software

– Initial planning involves end users prior to deployment – Training is provider- or

user-specifi c to achieve desired learning Maintenance

Activity

– Fixing problems that appear after deployment, including errors that may have occurred during

• Software implementa- tion (i.e., programming errors) • Requirements (i.e., incorrect elicitation of performance requirements from users)

• Design (e.g., a dysfunc- tional architecture)

– Mechanism for rapid identifi cation of needed maintenance will avoid perpetuating fl aws – Performing episodic maintenance maintains trust by users – Planned maintenance is best practice

Upgrade Activity Modifying the software to meet new requirements that may emerge over time, such as enabling the software to work with new hardware

– Planning for upgrades with adequate backup systems is a requirement

– Planned safety testing of operational system on routine ongoing basis

NOTE: Traditionally, these activities are called “phases” (e.g., requirements phase, design phase); this report adopts the “activities” terminology to emphasize the point that, although these activities are conceptually distinct, some of them may be occurring simultaneously from time to time.

Software Implementation Activity

After the technology package is deemed ready for delivery to the user, it is transmitted to the user’s organization for initial use. Prior to using health IT products in the care of patients, exten- sive training must be done for the specific product and the specific organizational setting. It is customary for organizations to set expectations for training that require documentation of learn- ing modules and demonstrated competency. Resources to support initial and ongoing training are essential components of planned implementation activities. The period of initial use in an opera- tional environment is fraught with patient safety risks, because it is during this period that many problems are most likely to appear. Some of these problems will result from users who have not received adequate training—this will be true even if the technology is designed to require mi- nimal training. Other problems will result from operating the technology with the health of real patients at stake rather than in an artificial environment that is well controlled and does not re- flect an actual health care setting.

An organization typically selects one of two approaches to implementing the technology: ei- ther a big bang strategy (i.e., the technology is implemented for use throughout the entire organi- zation at the same time or nearly so) or an incremental approach (i.e., the technology is first dep- loyed for use on a small scale within the organization and then, as operating experience is acquired, it is deployed to other parts of the organization in a gradual, staged manner).

Experience with the implementation of large-scale technology applications suggests that, over time, success is possible with either approach. Nevertheless, a health care organization should plan on the simultaneous operation of both the new and old technologies (even if the old technology is paper based) for some transitional period, so that failures in the new technology do not cripple the organization. Backup and contingency plans are necessary to help anticipate and protect against a wide range of failures and problems with the newly implemented technology and are an essential part of implementation planning.

Maintenance and Upgrade Activities

Maintenance and upgrade refer to activities carried out after software is initially deployed to keep it operational, to support ongoing use, or to add new functionality to the software. Because upgrading a software package involves many of the same considerations as maintenance, the dis- cussion that follows will speak simply of “maintenance” with the understanding that the term also includes upgrades. During maintenance, health IT products are continuously subject to a va- riety of activities and interventions by vendors intended to correct defects, introduce new fea- tures, optimize performance, or adapt to changing user environment or technologies (Canfora et al., 2010).

Maintenance activities paradoxically can make products increasingly defect-laden and more difficult to understand and maintain in the future (Parnas, 1994). Maintenance activities inadver- tently tend to degrade software system structure, increase source code complexity, and produce a net negative effect on system design for a variety of reasons, such as production pressures, li- mited resource allocation, and the lack of a disciplined process. Maintenance also increases the complexity of health IT, increasing the likelihood for error.

End users and purchasers of health IT are not always aware of the specific risks associated with the maintenance phase. Installation of a patch (code upgrade) can disrupt existing functio- nality by introducing previously absent dependencies, and functionality can be lost by installing a patch. Maintenance work often requires taking software offline for a number of hours, and re- quires advance planning and notification of users to minimize disruptions in care. Maintenance also requires that actively engaged users be involved in the testing process prior to the imple- mentation of a patch or upgrade into the production environment.