Refuerzos/materiales estructurales

Thereafter, I continued to test the second null and second alternative hypotheses. I began by using the Shapiro-Wilk test to test the samples of computed SUS scores from the study for normality. Table6.3shows the details of the Shapiro-Wilk test results for all two samples which comprised of experienced and novice CIG modellers. The Shapiro- Wilk test result for the sample of SUS scores from experienced CIG modellers had a p- value that was small (p < 0.05). Hence, the results showed that the SUS scores from the sample of experienced CIG modellers were not from a normally distributed population. Due to the fact that the SUS scores did not come from a normally distributed population, I used a Wilcoxon Rank Sum test to test for statistical significance of the differences measured from the two conditions of the experiment.

Recall the second null hypothesis, H0: There is no difference in SUS scores between the ratings from experienced CIG modellers and those from novice CIG modellers. The medians of SUS scores from experienced CIG modellers and novice CIG modellers were 89.17 and 79.23, respectively. A Wilcoxon Rank Sum test was carried out to evaluate the differences in CIG modellers. The test showed that there was no significant effect of CIG modellers (The mean ranks of experienced CIG modellers and novice CIG modellers

were 13.67 and 8.31, respectively; U = 0.90, Z = 1.95, p = 0.05, r = 0.45). Due to the fact that the effect of CIG modellers on SUS scoring was not significant, the second null hypothesis can not be rejected. This means that there is no significant difference in SUS scores between the ratings from experienced CIG modellers and those from novice CIG modellers.

Table 6.4: Responses from the SUS survey with experienced CIG modellers

Question Score

1 2 3 4 5

1 - Would use the grammar frequently

0 – 0% 0 – 0% 2 – 33.3% 2 – 33.3% 2 – 33.3%

2 - Grammar was unnecessar- ily complex

6 – 100% 0 – 0% 0 – 0% 0 – 0% 0 – 0%

3 - Grammar was easy to use 0 – 0% 0 – 0% 1 – 16.7% 3 – 50% 2 – 33.3%

4 - Need support of a technical person

4 – 66.7% 2 – 33.3% 0 – 0% 0 – 0% 0 – 0%

5 - Functions were well inte- grated

0 – 0% 0 – 0% 0 – 16.7% 4 – 66.7% 1 – 16.7%

6 - Too much inconsistency 6 – 100% 0 – 0% 0 – 0% 0 – 0% 0 – 0%

7 - Most people would learn quickly

0 – 0% 0 – 0% 1 – 16.7% 0 – 0% 5 – 83.3%

8 - Grammar very cumber- some to use

6 – 100% 0 – 0% 0 – 0% 0 – 0% 0 – 0%

9 - Confident using the gram- mar

0 – 0% 0 – 0% 0 – 0% 3 – 50% 3 – 50%

10 - Needed to learn a lot of things

5 – 83.3% 0 – 0% 1 – 16.7% 0 – 0% 0 – 0%

6.4

Discussion

Although SUS is a subjective assessment of usability, it can be considered as a valid and reliable indicator of usability [249]. It is argued that the SUS has undergone some considerable amount of psychometric testing to evaluate its validity, reliability and sen- sitivity [231,250]. Furthermore, other studies have shown that results from the SUS can reliably converge at samples that are as low as eight [231,294]. Some researchers have argued that the word ‘cumbersome’ in statement number eight of the SUS question- naire can cause confusion when posed to non-native English speaking participants in a multinational setting [250,295]. Since this study was carried out in sub-Saharan Africa which is a predominantly non-native English speaking region, the validity of the results in the SUS survey was not negatively affected as the participants in the study were all proficient in the English language. Five experienced CIG modellers were university graduates who had completed their formal education from institutions whose medium of

Table 6.5: Responses from the SUS survey with novice CIG modellers

Question Score

1 2 3 4 5

1 - Would use the grammar frequently

0 – 0% 0 – 0% 4 – 30.8% 6 – 46.2% 3 – 23.1%

2 - Grammar was unnecessar- ily complex

8 – 61.5% 4 – 30.8% 1 – 7.7% 0 – 0% 0 – 0%

3 - Grammar was easy to use 0 – 0% 0 – 0% 3 – 23.1% 3 – 23.1% 7 – 53.8%

4 - Need support of a technical person

6 – 46.2% 3 – 23.1% 1 – 7.7% 2 – 15.4% 1 – 7.7% 5 - Functions were well inte-

grated

0 – 0% 0 – 0% 1 – 7.7% 8 – 61.5% 4 – 30.8%

6 - Too much inconsistency 9 – 69.2% 4 – 30.8% 0 – 0% 0 – 0% 0 – 0%

7 - Most people would learn quickly

0 – 0% 3 – 23.1% 2 – 15.4% 4 – 30.8% 4 – 30.8%

8 - Grammar very cumber- some to use

9 – 69.2% 3 – 23.1% 1 – 7.7% 0 – 0% 0 – 0%

9 - Confident using the gram- mar

1 – 7.7% 1 – 7.7% 4 – 30.8% 3 – 23.1% 4 – 30.8%

10 - Needed to learn a lot of things

4 – 30.8% 8 – 61.5% 1 – 7.7% 0 – 0% 0 – 0%

instruction was English. The sixth CIG modeller was a native English speaker. Further- more, all novice CIG modellers were university students who had received their prior education in English.

The results in Section6.3.1show that both novice and experienced CIG modellers rated the grammar of FCIG modelling language highly on the system usability scale. Hence it can be said that the participants in this study perceived the grammar of FCIG to be usable. Such a grammar evaluation can provide invaluable insights that can assist a soft- ware language designer to improve a software language of interest. Hence contributing to the likelihood of the software language to be adopted in practice.

From the ten dimensions of usability assessment on the SUS scale, the majority of both the novice and expert CIG modellers indicated that FCIG’s grammar was easy to use, had well integrated structural elements, did not have too much inconsistency in addition to not being very cumbersome to use. The reason for FCIG being perceived as a language with an easy-to-use and consistent grammar might stem from the fact that the language has a small and clear vocabulary consisting of just three main language constructs. Simplicity, clarity and consistency are widely accepted to be essential qualities of good modelling languages [180, 296, 297]. This can be further evidenced by the comments that the participants gave in their qualitative feedback after the survey. The following comments are selected from the feedback that was given by the novice CIG modellers:

1. Participant p5 commented, “The three constructs explain the model well. Specifi- cally Condition as it can load data that’s vital to making an informed recommen- dation”.

2. Participant p10 commented, “It was quite idiomatic and the syntax flows”.

Most experienced CIG modellers had similar positive perceptions across the same four usability assessment dimensions indicated positively by novice CIG modellers. In ad- dition to the four dimensions, most experienced CIG modellers also indicated that the grammar of the experimental language was not unnecessarily complex. Most of the experienced CIG modellers may have that perception on the grammar because of their prior familiarity with the process of modelling clinical knowledge which is typically car- ried out using a general purpose programming language. Furthermore, the experienced modellers can have such a positive perception because the semantics and vocabulary of the new CIG modelling language are based on existing concepts that are widely used in the clinical domain. Domain-appropriateness of a language, that entails that a software language be powerful enough to capture major domain concepts and be able to match the mental model of the domain, is very important for software language adoption [227]. This can also be evidenced from the comments that experienced CIG modellers gave in their feedback after the experiment as follows:

1. Participant p1 commented, “Condition, action, recommendation. These really cover the basics one would need to use in this kind of setup”.

2. Participant p5 commented, “The syntax, CAR for Conditions, Actions and Rec- ommendations were easy to follow”.

Although the mean SUS scores for both novice and expert CIG modellers were differ- ent, the statistical test results in Section 6.3.1show that the perception of experienced CIG modellers on the usability of the grammar of FCIG was not significantly different to that of novice CIG modellers. This is an expected result because the grammar of FCIG uses a small set of language concepts that have a direct mapping to the clinical guideline formalization concepts. By employing a small set of language concepts with adequate expressive power, both novice and experienced modellers are likely to find the grammar as usable. Incompatible domain abstractions in a DSL grammar can introduce limitations that can negatively impact a DSl’s usability [158,213,298].

6.5

Chapter summary

In this chapter, I evaluated the usability of FCIG’s grammar which is a novel, simple and compact syntax for modelling evolving CIGs. I achieved this by assessing the perceptions of CIG modellers on the usability of FCIG’s grammar. Novice CIG modellers were recruited from the University of Cape Town in South Africa where as experienced CIG modellers were recruited from EMR system implementing organisations in Malawi and South Africa. FCIG was found to have a pragmatic grammar for modelling CIGs. Both novice and experienced CIG modellers perceived FCIG’s grammar as a usable and practical grammar for modelling evolving computer-interpretable guidelines. In the next chapter, I describe how I evaluated FCIG for its efficacy in modelling evolving CIGs by comparing it with the HL7 certified standard Arden Syntax.

Experimental evaluation of FCIG

7.1

Introduction

As CPGs evolve over time, their corresponding computer-interpretable guidelines (CIGs) in clinical decision support systems are required to be kept up-to-date so that clinical advice is based on correct guideline recommendations. A CIG modelling architecture that explicitly models the CIG elements that are affected by clinical practice guide- line (CPG) changes has the potential to improve the maintenance of CIGs by enabling effective and efficient computational support, not available in current CIG modelling en- vironments, for encoding and maintaining CIGs. In order to enable such computational support, a CIG modelling language, FCIG, has been developed for use in a four-layer model-driven architecture. An experimental CIG modelling environment for FCIG has been implemented in Eclipse.

Recall the discussion on Arden Syntax in Chapter 2. Arden Syntax was established in 1989 and subsequently developed as a Health Level Seven (HL7) certified standard for modelling computer-interpretable guidelines [73,79]. HL7 is a not-for-profit, Amer- ican National Standards Institute (ANSI)-accredited standards developing organisation dedicated to providing a comprehensive framework and related standards for the ex- change, integration, sharing, and retrieval of electronic health information that supports clinical practice and the management, delivery and evaluation of health services [73]. Practical and technical limitations have forced researchers developing guideline mod- elling formalisms and execution software to confine the use of their technology to their home institutions [80]. Arden Syntax represents procedural clinical knowledge in MLMs. Each MLM contains sufficient knowledge to make a single decision that invokes a specific

action [78]. Arden syntax has been used for clinical decision support by generating clin- ical alerts, diagnostic interpretations, management messages and screening for research studies [81].

Arden4Eclipse is an editor for writing Arden Syntax Medical Logic Modules (MLMs) in the Eclipse IDE. It integrates Arden2ByteCode, an open source Arden Syntax compiler, so that Arden Syntax code can be easily written as well as executed. Arden2ByteCode runs on Java Virtual Machines (JVM) and translates Arden Syntax directly to Java bytecode (JBC) executable on JVMs [299]. It also serves as runtime environment for execution of the compiled bytecode. For straightforward use there is an Arden Syntax Editor plugin for the Eclipse IDE which integrates Arden2ByteCode so Arden Syntax code can be written and executed. Unlike FCIG, Arden Syntax has a formal syntax but no formal semantics [300].

Significant effort is required to maintain guideline-based clinical decision support sys- tems so that their recommendations are based on up-to-date CPGs. Adoption of a CIG modelling language that uses CPG structural elements that are affected by changes as representation primitives has the potential to provide computational-support for mod- elling evolving CIGs. This work introduced FCIG, in Chapter5, as such a language that has an explicit specification of elements that are affected by CPG changes in its formal language model.

This chapter presents the methods and related results of an experimental evaluation study that I carried out to compare the novel CIG modelling language FCIG with the HL7 standard Arden Syntax.