POLITICAS Y OBJETIVOS DE MANEJO DEL PNS

To reiterate, the objective of the approach used was to acquire a set of basic ontol- ogy structures from the participants which, through a process of automated and semi- automated mediation, converge into a single consensual ontology that would provide a representation of the target domain. This acquired ontology would also contain concepts for which there was some, but not unanimous support or which have concept models that conflict; these concepts would be noted so that a knowledge engineer could make the final adjudication. At this stage the participants are only required to define a con- cept hierarchy and the object properties needed to define the relationships between these concepts.

Using the competency questions they were then asked to determine the concepts and object properties that would be needed to model the domain, and in particular to answer their particular set of competency questions. Then each group was tasked with engineering the ontology that would be needed to answer their set of competency questions, using MoKi1 _{[25]. The complete instructions given to the participants are}

provided in Appendix B.

MoKi was chosen because it is available through a web-browser, meaning that it was readily available, and because for the tasks required it felt more intuitive than alternatives such asWebProtege.

After collecting the competency questions and provisional ontology structures a rep- resentative from each group was chosen to extend and refine their ontology. To achieve this they were provided with additional competency questions obtained from the filtered aggregate of other groups. The idea behind this was that, given a broader set of com- petency questions, the participants would have to produce more complete and therefore more expressive ontologies. While this extension might appear contrived as the filtering was based on a deliberate attempt to restrict the scope of the model, this helped to simulate a broader and more in-depth participation that would be achieved from having a greater participation. While this was a satisfactory solution for Experiment 1, it is acknowledged that this has the potential to introduce bias, therefore this expansion step was not repeated in Experiment 2 where a greater level of participation is achieved.

The informal competency questions obtained from the participants are listed in Table 4.1. It is fairly evident that the quality of these questions varied significantly, which is to be expected as the participants are, in terms of their knowledge of the domain in question, representative of crowd input. The task that presents itself was how to remove

those questions which do not assist in the defining of an adequate ontology structure. The final competency question-set was manually chosen by filtering questions according to the following criteria:

• Does the question intuitively require a constraint or particular concept structure to be specified in order to answer?

• Would the competency question be better answered in an alternative domain?

• Is there considerable ambiguity in the phrasing of the question such that the competency which is being tested is unclear?

Additionally, some competency questions could have been better expressed as two or more separate queries, typically where the participant had used the ‘or’ clause. In these cases the competency question is replaced by two or more queries that represented all the possible outcomes of the original. Additionally, where instances were referenced directly the competency question is altered to provide a concept as variable (denoted by square brackets). Given these criteria, the refined competency questions were then re-circulated to the groups in order to refine their models, as listed in Table 4.2.

Question Group(s)

What is the most popular song this week? 1,2 What was the album published in [year] by [artist]? 1 Which female artist won an award this year? 1 Which classic English album cost is less that 10? 1 What is the most popular single? 2 What is the most popular group? 2 Does the store have a search function? 2 Does the store have a download function? 2 Which male artist sangone love? 3 What albums is artist best known for? 3 List the new rock songs? 3 How many members are in thewailers? 3 What was the No1 album of 2012? 3 This group/person belongs to which genre? 4 This song belongs to which record? 4 This person belongs to which group? 4 This record belongs to which artist? 4 Which album has [artist] released? 5 Which genre is [artist] best known for? 5 Which artist won an award last year? 5 What song charted last week? 5 What song is free this week? 5

Table 4.1: Unprocessed competency questions obtained from participants.

The revised list of competency questions was then re-circulated amongst the par- ticipants and they were asked to augment their models according to the extended list. At this point five ontology structures had been obtained from our participants, each designed to answer an aggregate set of competency questions, which itself, had been obtained from the crowd of participants. The number of entities (concepts and object properties) that had been obtained at this point is summarised in Table 4.3.

Group No. Concept Count Property Count

Group 3 18 18

Group 4 8 6

Group 5 9 7

Group 6 18 10

Group 8 10 9

Table 4.3: Summary of concepts and properties obtained from combined competency question list.

Question Action What is the most popular song this week? Ambiguity What was the album published in [year] by [artist]? - Which female artist won an award this year? -

Which classic English album cost is less that 10? Domain

What is the most popular single? Ambiguity

What is the most popular group? Ambiguity Does the store have a search function? - Does the store have a download function? - Which male artist sang [song]? - What albums is artist best known for? -

List the new rock songs? Ambiguity How many members are in [group]? - What was the No1 album of [year]? - This group belongs to which genre? Split This person belongs to which genre? Split This song belongs to which record? - This person belongs to which group? - This record belongs to which artist? -

Which album has [artist] released? Ambiguity Which genre is [artist] best known for? - Which artist won an award last year? - What song charted last week? - What song is free this week? -

Table 4.2: Competency questions filtered according to stated criteria. Entries emphasised were removed from consideration from second stage refinement

4.3.2 Mediation Results Formatting

Table 4.4 shows the number of edits that were made across the ontology set. The most common modifications were made using the de-capitalisation rule. No edit distance changes were required. The formatting rules were applied using a small program written in Java which automatically updated the ontology files.

Table 4.4: Formatting Edits

Group De-Capitalisation Composite Edit Distance

Group 3 30 12 0

Group 4 12 0 0

Group 5 15 0 0

Group 6 25 5 0

4.3.3 Automated Mediation