In consolidating the knowledge bases, there were few instances where the experts clearly disagreed with each other; most of the differences could be accounted for by small inconsistencies between the knowledge bases. For example, one knowledge base might use greater or equal to as a border between classifications, whereas the other uses greater than. Differences such as this were also quite common within each knowledge base, for example the rule for one gradation of classification might use less than 40 while the rule for the next level uses greater than 40, excluding any case which falls exactly on 40. The resolution to these border definition problems
were typically fairly arbitrary, as the experts tended to consider it unimportant which resolution method was used, as long as it was consistent. Each occurrence was of little consequence individually, but accounted for 1256 cases (42.4% of the dataset) reaching different classifications overall. Once resolved by the administrator these differences had little further effect.
Another source of disparity resulted from the manner in which the knowledge bases were built: one expert might add a rule as an exception, where another expert adds that same rule at the root level. If the first expert does not encounter any rules which should match the exception, but do not match the rule it is an exception to, then the mistake will not be noticed.
There were five identified instances of distinctly different definitions for classifications that did not have obvious solutions. To resolve these, the administrator initiated an email conversation including each of the two experts involved (the primary experts). In one instance both considered the other’s opinion too extreme, so a compromise was found that both experts were satisfied with. On another occasion one expert declared he had no objection to removing the differing condition from his rule. For one other conflict, the primary expert explained his rule difference to the other expert, who happily accepted the change once he understood. The final two differences were not able to be resolved by discussion of rule conditions alone. For one of these, one expert had included in the collaborative knowledge base an alternative rule for reaching a given classification, which had no counterpart in the independent knowledge base. Although confident that the rule was not an accurate definition, the other expert could see some logic behind it and was open to the possibility that it may be useable. In response, the expert who had added the rule suggested looking at how often the rule misclassified cases compared with the other, more widely accepted rule. Using the statistical tools implemented for the knowledge discovery section of this study (discussed in Chapter 4), it was found to give 158 false positives and 9 false negatives, with 7 cases matching both rules, out of the 1390 cases in the TAHS dataset. This was deemed far outside the expected parameters for intended coverage of the rule; hence, although the alternative rule may have correctly classified some cases, the inaccuracies were deemed too great and the rule removed.
The second conflict arose from a condition added to a rule by the expert in the independently developed knowledge base. The rule existed in both knowledge bases, in the same form except for that condition. The condition was added to cover a small contingency of cases that the expert considered a possibility. On learning that no other expert had included such a condition however, the expert expressed some uncertainty and a feeling that the other expert may in fact be better educated in this instance. To resolve the problem, he requested data on how many cases were affected. Given that only 15 of 947, or 1.5% of cases with the classification also matched on that condition, the expert decided that any potential benefit did not outweigh his uncertainty and decided to remove the condition.
Resolving these conflicts led to the resolution of 613 cases (20.7% of the dataset) that had previously reached different classifications between the knowledge bases. The equated classifications provided a simple way of comparing the results of the knowledge bases, to identify problematic differences. In consolidating the equivalent classifications into a common structure, one set of classifications (the
Obstruction group) were problematic: each knowledge base used different versions of the classifications, both in terms of gradation and in compound classifications with another classification (Reversibility, or, Positive response to bronchodilator). The experts were consulted as to which of the gradations of severity should be used, and which definitions to keep. It was considered relatively unimportant, the end result being much the same in terms of providing a sufficient interpretation; and the version included in the initial documentation was kept, as that document had been circulated and confirmed by other experts. The Reversibility elements were separated into separate classifications, at the assurance of the experts that this was not problematic or any less correct.
Each of the other groups were relatively simple to consolidate, once common differences were resolved. In each case the more detailed versions were included for completeness.
The numbers of conflicts presented here highlight that a standardised knowledge base and an expert system can be very beneficial to the domain: even between 3 experts, the two major contributors of which are high-ranked specialists in the field,
1869 out of 2963 cases (63.1%) received different classifications. Of these, 613 (32.8% of the conflicts) were major and needed intervention to resolve.
The very minimal input from the secondary expert in the collaborative knowledge base unfortunately precluded any comparison between collaborative approaches and post-acquisition compilation of knowledge bases.