Los profesionales médicos

We want to specify the tasks to the participants so that they would use the system to

answer questions similar to those that typical users might have. We reviewed the four

proposed use cases in Section 3.2, including choosing the right terms for SPARQL, finding

interesting facts, and detecting potential errors in instance co-reference or ontological

alignment; and from them we tried to see which use cases can be cast as a proper task for

The first use case is to use the system to choose the right terms for SPARQL. We

propose this use case as the most promising usage of the TC system. However, during

the study or in the real world, we shall not expect that the participants or average users

already understand the concepts of SPARQL before they start to use the system. A

practical way is to give a tutorial and tell the participants that we usually want to choose

the terms that are used in the most instances when no other clue is available for us to

decide which candidate term is better for queries. We illustrate the set of instances with

Venn diagram and use a made up example about the “movie” classes and the “directedBy”

properties. Then the task is stated as

Task 1. Given a pair of keywords for the class and the property as the query

purpose, can you find a matched class and a matched property from some

source (specified by their namespaces), so that using this pair of terms (the

best combination) would retrieve most of the instances in the dataset?

We also need to decide the pairs of keywords. There are a few considerations when

picking the keywords. Firstly the topic should not require any specific background knowl-

edge, i.e. it should not be in life science domains, nor about academic publications, given

that the participants could be a freshman without such knowledge. Also because we will

have multiple pairs, we want the topic of each pair to be different from those of all other

pairs. The most important reason is that we do not want a participant to acquire any

information in the task of one pair and have this information bias the subsequent task of

topic each request is related to. We list the final pairs and the best combinations we

have for the study as follows. Note that in this experiment we only consider the syntactic

matches, so that the tasks are more straightforward to the participants and the less open

questions make it easier to analyze the results. However in the future we should also

consider involving synonyms as answers.

• Task 1.1 Keyword for Class: “Company”, for Property: “location”

Best Combination: {dbpediaowl:Company, dbprop:location} (19208 instances)

• Task 1.2 Keyword for Class: “Scientist”, for Property: “award”

Best Combination: {dbpediaowl:Scientist, dbpediaowl:award} (3072 instances)

• Task 1.3 Keyword for Class: “Town”, for Property: “population”

Best Combination: {dbpediaowl:Town, dbpediaowl:populationTotal} (23104 in- stances)

• Task 1.4 Keyword for Class: “Actor”, for Property: “starring”

Best Combination: {yago:Actor109765278, dbpediaowl:starring-} (15225 in- stances)

After considering the second use case of “learning interesting facts”, we decided that it

is not specific enough to investigate with a user study. Although we believe the TC system

provides users an easy but flexible way to explore the dataset and learn something, it is

hard to quantify whether their findings are interesting and valuable, in fact, interesting and

the opportunity of interesting findings in other tasks.

The other two use cases are “detecting errors”, one involves co-reference errors and the

other ontological errors. We believe that non-expert users will find it difficult to analyze

errors in order to determine what the reason of the error is. However it might not be

hard for the users to find something that sounds improbable as long as they are presented

with a interface that they clearly understand what the data indicates. We give a made

up example of overlapping between instances of movies and book, and suggest that the

reason is because some instances of movies are considered the same as their original novel

books. This task is stated as

Task 2. Given a class as the context, browse all the overlapping class tags in

the system, can you find something that is incompatible, i.e. it is impossible

that something is both an instance of the context class and an instance of the

tag?

Again we follow the criteria we use for Task 1 to pick the context, and we also want

the total number of overlapping class tags to be shown in 10 - 15 pages. We finally decide

two contexts: Task 2.1 lgv:Stadium; Task 2.2 dbpediaowl:College. We shall discuss

more with these two sub tasks in Section 5.3.

In order to compare the TC and HL systems, we have each participant to equally use

both systems, which also means that each system will be equally used across participants.

Also we want each question to be equally answered using either system, so that we can

order of the questions shown to the participants but randomized the order of systems to

be used. To ensure the order does not change after refreshing the page, we used a pseudo

random order determined by each participant’s login user name.