UTILIDAD DEL COEFICIENTE INTELECTUAL

Both guideline and overall crediblity are significant for user satisfaction, and low credibility may create a situation where experienced clinicians refuse to take in new and updated clinicial recommendations. Individual recommendations are graded by work groups formed by the Norwegian Directorate of Health, and these work groups comprise of several clinicians from different hospitals and medical offices from different parts of Norway. Sources for the stroke guidelines are well doc- umented and methodical strong, and picked out by said working groups. The grading system is therefore considered as a credible way of evaluating recommen-

dations. Credible sources and trustworthy recommendations leads to agreement from clinicians, and no fallbacks to previous practice.

While the basis for this CDSS is well documented and tested, participants had to lean on their own experience and knowledge in order to be certain. Both AGR and GAS received feedback which indicates that none of them accentuate the research which recommendation grading is based on. Recommendations in both modules are presented as-is, with grading indicators. There is however room for extending this grading system so that the different levels become clearer. A simple solution to the credbility issue is to link grade letters and level indicators to additional information as found at Helsebiblioteket.no (Helsebiblioteket, 2013). In GAS, this information could reside on the guideline page, below the actual recommendations. Links from the grading levels could then lead the users down to the relevant back- ground information. Participants relied on additional experience and knowledge to make decisions, and used the DIPS CDSS as a supplement. Few participants did however describe the system as untrustworthy, and their prior experience of- ten included the same guidelines as used in this experiment. These guidelines, amongst others, are curriculum in medical studies at NTNU.

It is generally important to utilize critical thinking when using CDSSs and other support tools for decision making. This results in clinicians making their own deci- sions, and these decisions may differ from suggestions by a CDSS. A well-designed CDSS gives clinicians the option to choose, reflect and evaluate decisions suggested by the CDSS. Participants also mentioned that they need to use the AGR module over a period of time in order to fully trust the ranking. Only then are participants able to see a pattern in the ranking system, and therefore get some understanding of underlying ranking algorithms.

Chapter 11

Limitations

As with any software development project, this one also hit some obstacles during the development process. This chapter first look into possible improvements of the development process. It then dives into the experiment design, and point out different elements which could have been done differently.

11.1

Prototypes

This chapter lists those limitations related to implementation choices in both mod- ules, as well as limitations of the DIPS system. During the implementation stage, several key choices were made. These choices greatly impacted on the module usability in both modules, and selected solutions as well as discarded solutions are presented here.

11.1.1

API Access vs Scraping

Guidelines from the Norwegian Electronic Health Library are essential for AGR and GAS, and obtainment of them were therefore the first and most important task to complete. Representatives of Helsebiblioteket.no were contacted in order to receive access to some API for fetching guidelines. While waiting for access, a script were made (see chapter 7). The purpose of this script is to scrape and parse guidelines, as well as to add metadata for easier access. Helsebiblioteket.no came through and sent source files containing guidelines, however they did not provide any API. The received files also contains less metadata than the generated files, and the official source files were scrapped.

If this had been a larger project than just a basis for an evaluation of a theroretical system, then the first objective would be to create a common API. This enables other projects to access and utilize the guidelines without the hassle of scraping

and modifying them while also adding identifying and meaningful metadata.

11.1.2

Internet Explorer 6 vs Chrome

Both AGR and GAS were originally developed to work with the Chrome web browser and the Chrome browser plugin. The new DIPS system uses the Chrome plugin, but lacks other features critical to the usability experiment. Therefore, it was necessary to use the current (and old) DIPS system for the tests. One advantage of using the current system, is that test patients and other data are pre- stored here due to earlier tests done in the EviCare project. A great disadvantage is that this DIPS system uses a fairly outdated and old browser plugin; Internet Explorer 6 (IE6). IE6 is subject to much criticism, and even Microsoft drives to stop users from using this version. IE6’s market share has been estimated to 6,7% per april 2013 (Microsoft, 2013), and keeps falling.

Figure 11.1: IE6 market share from jan. 2008 to jan. 2013 (source: NetMarket- Share (2013))

IE6 have little to no support for several web technologies like JavaScript or HTML5. This disabled most dynamic features in AGR, and some features provided by Google Custom Search in GAS. The original AGR module had features like expanding and collapsing guidelines, and functions to redefine guideline order for the test and styling. The most prominent drawback of poor styling (CSS) support were shown when subchapter headings failed to indent. That, amongst many other more subtle styling flaws may have made the module less appealing and usable. Of course, there were multiple solutions to this compatibility problem. Three of those were considered realizable in this project;

• Install a plugin called Chrome Frame simulating Chromium in Internet Ex- plorer (IE)

• Upgrade the IE plugin to a modern version

• Rewrite the code to comply with older versions of IE 11.1.2.1 Chrome Frame

Chrome Frame were made after years of adapting certain web sites to outdated versions of IE, primarily found in larger organizations in governments or private sectors. It is basically a plugin for old IE browsers, which emulates a Chrome browser inside IE. This means that developers only have to test their web appli- cations on a limited number of semi-cross-compatible browsers, in theory. Or as Google describes it themselves:

Google Chrome Frame is an open source plug-in that seamlessly brings Google Chrome’s open web technologies and speedy JavaScript engine to Internet Explorer. (Google, 2013)

The HTML code were altered to support Chrome Frame, and to give users the ability to install Chrome Frame. This did however not work, even after some debugging.

11.1.2.2 Upgrade Internet Explorer plugin

DIPS were contacted in order to see if an upgrade to the IE plugin in DIPS were possible. After some consideration, this solution were dropped. An upgrade would probably consume time from an already tight schedule. Bugs could emerge by trying to integrate a modern version of IE in an outdated version of DIPS. This must however happen sooner or later if this DIPS system is to be used in future projects.

11.1.2.3 Code Refactoring - The Chosen Solution

This was deemed the most realizable solution, since no external effort were neces- sary. At first, the plan was to rewrite the JavaScript code to fit IE6’s specifications. On the other hand, the list of incompatible JavaScript and JQuery elements were long, and embraced most features. AGR did not work at all, while GAS worked to some degree. The easiest solution were to exclude all JavaScript and JQuery code in AGR, and write a temporary static version with simple HTML.

several features that could have increased overall usability. As mentioned in chap- ter 9, participants wished for features that could have been implemented more effortlessly using a newer browser. That is, features like expanding and collapsing guidelines, as well as a dynamic menu or mini map of the site.

Not only AGR suffered from IE6 compatibility problems, GAS had some related bugs as well. The most prominent one were that users are unable to use the enter key after entering search queries into the search box. Results were also erased when the user alternated between guideline info pages and the result page. Both AGR and GAS suffered from “black outs” as well, where the whole win- dow for external patient information went blank and had to be restarted. That significally affected the participants’ workflow.

11.1.2.4 Testing

Test routines outside the actual experiment are subject to criticism. One of two things could have been done from the project start. Either test the modules in IE6, or upgrade the IE6 plugin to IE10. The latter is as mentioned probably more time consuming than the former, and DIPS developers are understandably focused on creating the new refurbished version. Proper test routines could have cut down the development time significantly, while improved and increased communication with DIPS could have cut it down even further.

11.2

Experiment

The actual experiment also had some limitations and potential for improvement. The design featured some trade-offs between facilitating new users and facilitating more experienced users.

Participants were as mentioned in chapter 8.7 introduced to the think aloud method and made aware of the experiment goals. This prepping should have been done more thoroughly, as some students forgot to think aloud, and a minor- ity also seemed to miss what the goal was. One participant had to cancel task 1 in the usability test prematurely, due to struggles related to low experience with the domain and clinical practice in general. That participant had to constantly be reassured of the tests purpose; to test the system, and not the user. This is a key principle in usability testing, to calm the participants down when they experience difficulties.

11.2.1

Tasks

Task information were also conveyed textually, and this information should have been more thorough and detailed. More explicit information about think aloud and experiment goals could also have been added to the set of tasks, in addition to the oral and textual information given before task execution. This would act as a reminder, making the information easier to process.

One task also made participants search and find the window for external patient information, which proved to be a difficult task. This task could have been ex- cluded, as it had no part in the final evaluation. On another note, it introduced the DIPS system to participants, making it possible for them to familiarize with the user interface. Terje Røsand (2012, chapter 4.2) goes more in-depth in the description and analysis of these DIPS usability limitations.

11.2.2

GAS and AGR Order

All participants used GAS first (in task 1), before switching to AGR in task 2. This order could have made the AGR system seem more usable, as the participants get more acquainted with the surrounding user interface. Nevertheless, participants scored AGR much lower in the system usability scale forms. In order to avoid such uncertainties as this one, the experiment could have contained a much larger scaled usability test. This new version of the usability test should have contained a default system for guideline obtainment, without using either module. This de- fault module would act as a basis for evaluating both AGR and GAS.

Another possibility is to add more participants and collect more data, while also having the opportunity to randomize tasks among them. Randomization is often used in clinical trials, and in this experiment, it would reduce aforementioned is- sues created by participants recalling steps from previous tasks. (Sauro, 2004) In practice, this would mean that participants use AGR, GAS or default DIPS in random order. More participants is however more time demanding, since more time have to be used to perform the experiment, in addition to recruitment and analysis.

AGR may also incorporate randomization, by randomizing between a set of cor- rectly ranked guidelines and a set of ranked guidelines where the topmost guideline is less relevant than the following guidelines.

11.2.3

Patient Case

The patient case were rewritten and based on earlier work in the EviCare project. Some words were changed in order to accommodate inexperienced participants, the participant group consisted mostly of students with little experience. Cholesterol

lowering treatment were mentioned in the patient case text, which may have guided some participants more than necessary.

11.2.4

System Usability Scale

When answering the SUS forms, the participants were told to consider the whole DIPS system, they should instead have considered the actual modules, and tried to separate them from other DIPS elements. This would have gained more useful data, and could have been easy to integrate with a solution with more participants and randomization. SUS issues extend into interviews, since participants have their SUS answers fresh in mind when performing the interviews. When answering interview questions regarding system usability, some participants had a hard time focusing on just the modules, when they had to focus on the whole DIPS system in the previous SUS forms.

11.2.5

Participants

Timing around participant recruitment could have been planned better, meaning that students should have been provided a form of schedule where they could have marked their availability. Doodle (2013) and similar services provides exactly this feature, with little effort. As it was done now, the students enrolled first via e-mail, without specifying their available slots in their schedule, and then specified that in a later e-mail. This back and forth made it necessary to postpone the usability test with one working day. However, this had only minor additional effects on the experiment as a whole, only potentially losing some of the more busy students (mostly higher grade students).

Chapter 12

Conclusion

This chapter completes the discussion chapter by presenting the research questions, providing answers where possible, and evaluates the process of obtaining these answers.

12.1

Research Questions

The research questions presented in chapter 4.3 provides a basis for summarization of results discussed in chapter 10: Discussion and chapter 11: Limitations.

RQ1 How can a CDSS best utilize patient information as a

context for ranking relevant recommendations?

This project shows two different ways of navigating CDSSs; search-based naviga- tion and content-based automatic access to guidelines. The first, dubbed Guide- line Access using Search gives users a simple interface for accessing guide- lines without any additional means of navigation. The other, dubbed Automatic Guideline Ranking took results from a ranking algorithm and displayed the re- sults with little user interaction. These two modules each represent an extreme point in navigational spectrum shown in figure 12.1.

Several other solutions may emerge from these two, whereas one possible out- come is a combination of both modules. This combination could show guidelines automatically based on patient information from an EHR system, while giving the user the opportunity to search for other relevant and non-relevant guidelines. A new hypthesis is that this combination will gain better user satisfaction results. Increased user satisfaction increases adoption of CDSS by clinicians, and comput- erized decision making offers guidelines at the point of care, which in turn may increase quality of patient care.

RQ1.1 Which of search-based or content-based recommendation ranking gives best user satisfaction?

It seems like participants in this test struggled to some degree to find the guide- lines they were looking for in the search-based solution, and prefer solutions which automate guideline access. They seem to prefer the automated solutions when the module is considered as a stand-alone system. The automated solution greatly improves availability of guidelines from the search-based solution, which in turn creates a higher level of awareness for clinician users.

Clinicians prefer a solution with actions requiring little back and forth between different states or windows. External CDSS solutions outside the EHR system does require a lot of alteration between different systems with different designs, which decreases consistency and in turn user satisfaction. Integrating CDSS with EHR systems enables a smoother transition between patient information review and decision making.

These result may of course be applicable only to inexperienced medical students, and further research must be performed in order to explore how experienced clin- icians utilize search and content based access to clinical guidelines compared to inexperienced clinicians.

RQ2 How does CDSS integrated in the EHR system affect

clinical workflow?

This question is best divided and answered in two sub-question, where one sheds light on user repsonses to this new tool, and another which compares this com- puterized integration to manual solutions. Manual solutions and solutions with external guideline collections dominates clinical workflows today, as mentioned in chapter 5.

RQ2.1 How does clinical users respond to integrated decision support in the EHR system?

This project focused on integrating the CDSS prototypes with an existing EHR system. Responses from interviews show that clinical users prefer integration be- tween CDSS and EHR systems, over solutions which require the use of several conflicting and inconsistent external sources.

The EHR system were not tested without any CDSS integration, nor were any external CDSS tested. Scores from the SUS forms show that GAS integrated with DIPS EHR system gains a higher score than DIPS integrated with AGR (average SUS score of 42 with GAS and 24 with AGR). These SUS results does not how- ever conclude that either one of these combination gives better user satisfaction, as several dependent factors may affect the results. Factors include DIPS usability and the fact that participants learn how to use this EHR system as they use it. The most reliable data here is data from the usability testing and reflection in the interviews after each prototype test. Data here clearly suggest that participants prefer automatic access to guidelines in a context of patient information, but this depends on factors such as credibility and awareness. Credibility meaning that they trust the recommendations given to them, and trust that these recommenda- tions are indeed relevant to the selected electronic health record. Awareness means that an integrated CDSS displays clinical recommendations at the place and time of decision support. An integrated solution clearly displays recommendations at the place of decision support, and timing is decided by when the recommendations become available for clinicians in the integrated system. Both prototypes in this scenario gave recommendations at the time of decision making, but results show that search-based solutions may be more cumbersome and therefore more time consuming.

These results are not generalizable, due to a small and somewhat homogenous