PREGUNTAS CRUZADAS

3.3.7.1

Identified Errors

Listing the creator of a rule seemed effective in reducing the number of conflicts: it was observed that experts would be more cautious and thorough about changing rules when they were aware that it was another expert’s input, and they were not just correcting their own previous mistake. It was also observed that the experts were more comfortable changing the rules entered by the administrator than rule entered by another expert, especially when there was a feeling that the other expert would probably know better than themselves. However, it is assumed that this was successful partially because each participant was at least acquainted with each other expert, and knew of their qualifications and experience. For a larger scale study including many experts it may be important to, with participant consent, list each expert’s credentials so that other experts can check who is disagreeing with them, and to perhaps include a simple facility for communication between the experts. This would keep the system open to many experts of many backgrounds. It would also be an interesting study to ascertain what differences occur between displaying expert credentials and not: for a knowledge discovery system it may well be beneficial to attempt to convince experts to not dismiss any rule without examination of the data, by not letting them see who created the rule. This would, however, need to be balanced with the range of experience and knowledge of the

participating experts, to avoid having experts take too much time needlessly questioning domain principles, and would require an effective conflict resolution strategy.

3.3.7.2

Identified Conflicts

While there were multiple strategies in place to resolve conflicts in expert opinion, in practice it was found to require more direct intervention. The case highlighting approach, showing cases that had been disagreed with in red, proved unsuccessful due to the difficulties in organising the timely participation of multiple experts. Due to the typical restrictions on expert availability, experts preferred to work with the system in discrete time periods, trying to consolidate their interactions into as few sessions as possible. The busy and conflicting schedules of each expert further meant that these sessions were often quite separated, resulting in there being very little overlap in time between the inputs of one expert and another. Hence, it became very unlikely that an expert would ever, without direct prompting, access the system and see cases which had been disagreed with. With the possibility of experts noticing past conflicts effectively removed, it became reliant on the expert who made the new rule to contact the other expert and correct them or open discussions, which never occurred: given that the experts were busy, they unsurprisingly were not interested in taking the time to start a dialogue over every potential difference; each of which may only represent an input error. There was generally an attitude that the other expert had either only made a minor oversight, or were simply mistaken and that they would most likely realise their mistake if they ever looked at it again; further, that now that the system was correct (in their view), why should it need any further discussion? With these simple justifications and a busy schedule, it became clear that these methods of conflict identification would not work, as the likely outcome would be that only one party would be aware of any disagreement, and would probably pay little attention to it, considering it fixed.

Given the lack of success of expert-initiated methods, the conflict resolution fell to the other alternative: which was for the administrator of the system to resolve the conflicts by contacting the experts involved and initiating a discussion. This was a far from ideal situation as the administrator lacked the expertise to be able to interpret the classifications accurately. This combined with the different phrasings

and terminology used, and different levels of detail that experts used in their classifications, made identifying true conflicts and communicating them to the experts a more difficult and inefficient task than it may have been. Nevertheless, of the 5 conflicts identified, all were easily resolved by email discussions within a few days of being identified. This approach is considered successful however, as it kept required expert time to a minimum, which is still the bottleneck and major problem faced in such work.

There were minor miscommunications due to misunderstandings of the domain from the non-expert administrator. It is suggested that in a larger scale project this may need to be resolved by having someone with sufficient expertise in the area act as mediator; or, by ensuring sufficient commitment from the experts involved that they would be able to regularly access the system and review cases, in an overlapping time period. It is expected that in a commercial setting with definite outcomes for the experts involved, especially if they are paid to participate, that either of these would be a viable possibility.

A third option of conflict resolution was considered for this study, whereby whenever a conflict arose from an expert changing a previously accepted case, the system would generate an email to the expert who made the initial classifications thereby automatically initiating discussions. It became apparent that this would be impractical for the experts involved as their schedules for interacting with the system were widely deviant, and they would not welcome unsolicited emails, particularly if many were minor errors that required no further action. Due to the lack of verification in this process and the potential for a large number of emails to be generated, this is not recommended as the best solution; particularly with the possibility of frustrating the experts who are, in any knowledge acquisition venture, the most valuable and important resource.