MEDICIONES Y PRESUPUESTO

A normal execution of IBSE comprises the three consecutive steps shown in figure 6.4, even though they can also be executed independently. These steps are the following:

Figure 6.4: Automatic experiment process in IBSE.

1. To generate machine-readable benchmark descriptions from a group of ontologies. In this step, and from a group of ontologies located in a URI, one RDF file with one benchmark for each ontology is generated, using the vocabulary of the benchmarkOntology ontology. This description generation can be skipped if benchmark descriptions are already available.

2. To execute the benchmarks. In this step, taking into account all the different combinations of ontology interchanges between the tools, each benchmark described in the RDF file is executed and its results are stored in another RDF file, employing the vocabulary of the resultOntology on-tology.

To execute a benchmark between an origin tool and a destination one, as described in section 6.1, first the file with the ontology is imported into the origin tool and then exported into an intermediate file and, second, this intermediate file is imported into the destination tool and then exported into the final file.

Once we have the original, intermediate and final files with their corre-sponding ontologies, we extract the execution results by comparing each of these ontologies, as shown in section 6.1. This comparison and its output depend on an external ontology comparer. The current implementation

uses the OWL comparer of the KAON2 OWL Tools⁹, although other com-parers can be inserted by implementing a Java interface.

3. To generate HTML files with different visualizations of the re-sults. In this step, different HTML files are generated with different visualizations, summaries and statistics of the results.

Representation of benchmarks and results

This section describes the two OWL ontologies employed in the IBSE tool:

the benchmarkOntology¹⁰ontology, which defines the vocabulary that represents the benchmarks to be executed, and the resultOntology¹¹ontology, which defines the vocabulary that represents the results of a benchmark execution.

These ontologies are lightweight since their main goal is to be user-friendly.

They are described in appendix E using the RDF/XML syntax.

Figures 6.5 and 6.6 show the graphical representation of the benchmarkOntol-ogy and of the resultOntolbenchmarkOntol-ogy ontologies, respectively. Next, the section presents the classes and properties that these ontologies contain. All the datatype prop-erties have as range xsd:string, with the exception of timestamp whose range is xsd:dateTime.

Figure 6.5: Graphical representation of the benchmarkOntology ontology.

9version 0.27 http://owltools.ontoware.org/

10http://knowledgeweb.semanticweb.org/benchmarking_interoperability/owl/

benchmarkOntology.owl

11http://knowledgeweb.semanticweb.org/benchmarking_interoperability/owl/

resultOntology.owl

Figure 6.6: Graphical representation of the resultOntology ontology.

benchmarkOntology. The Document class represents a document containing one ontology. A document can be further described by properties that have Document as domain. Such properties are the following: documentURL (the URL of the document), ontologyName (the ontology name), ontolog-yNamespace (the ontology namespace), and representationLanguage (the language used to implemented the ontology).

The Benchmark class represents a benchmark to be executed. A bench-mark can be further described with properties that have Benchbench-mark as domain. Such properties are the following: id (the benchmark identifier);

usesDocument (the document that contains one ontology used as input);

interchangeLanguage (the interchange language used); author (the bench-mark author); and version (the benchbench-mark version number).

resultOntology. The Tool class represents a tool that has participated as ori-gin or destination of an interchange in a benchmark. A tool can be further described with properties that have Tool as domain. Such properties are the following: toolName (the tool name), and toolVersion (the tool version number).

The Result class represents a result of one step or of the whole bench-mark execution. A result can be further described with properties that have Result as domain. Such properties are the following: execution (if the whole interchange, the first step or the second step are carried out without any execution problem); informationAdded (the triples added in the whole interchange, in the first step, or in the second step); informa-tionRemoved (the triples removed in the whole interchange, in the first step, or in the second step); and interchange (if the ontology has been

interchanged correctly from the origin tool to the destination tool, in the first step or in the second step with no addition or loss of information).

The BenchmarkExecution class represents a result of a benchmark exe-cution. A benchmark execution can be further described with properties that have BenchmarkExecution as domain. Such properties are the fol-lowing: ofBenchmark (the benchmark to which the result corresponds);

originTool (the tool origin of the interchange); destinationTool (the tool destination of the interchange); and finally, timestamp (the date and time when the benchmark is executed).

Inserting a new tool

As the experiment requires no human intervention, we can only insert new tools by accessing them through application programming interfaces (APIs) or through batch executions. There are other ways of executing an application automatically (e.g., Web Service executions) but they are not present in the current tools. Nevertheless, to adapt the IBSE tool in order to include other types of executions should be quite straightforward.

Inserting a new tool in the evaluation infrastructure is quite easy, it can be performed either by implementing a Java interface in IBSE or by building a program that imports an ontology from a file and exports the imported ontology into another file.

To insert a new tool in the evaluation infrastructure, only one method from the ToolManager interface has to be implemented: void ImportExport(String importFile, String exportFile, String ontologyName, String namespace, String language). This method receives as input parameters the following: the location of the file with the ontology to be imported; the location of the file where the exported ontology has to be written; the name of the ontology; the namespace of the ontology; and the representation language of the ontologies, respectively.

This method has already been implemented for the tools participating in the benchmarking, i.e., GATE, Jena, KAON2, the NeOn Toolkit, Prot´eg´e-Frames, Prot´eg´e-OWL, SemTalk, SWI-Prolog, and WebODE.

Most of these tools have implemented the Java interface, because they pro-vide Java methods for performing the import and export operations. In the case of non-Java tools (SemTalk and SWI-Prolog), these operations were performed by executing precompiled binaries.

As an example, figure 6.7 shows the implementation of the ImportExport method for Jena.

Inserting and evaluating ontology comparers

We mentioned before that the IBSE tool uses external software for com-paring the ontologies resulting from the experiment. IBSE currently uses the diff methods of an RDF(S) comparer (rdf-utils¹² version 0.3b) and of an OWL

12http://wymiwyg.org/rdf-utils/

public void ImportExport(String importFile, String exportFile, String ontologyName, String namespace, String language)

throws BadURIException {

// Create the Jena models

Model model = ModelFactory.createDefaultModel();

Model model_out = ModelFactory.createDefaultModel();

try {

// Import the ontology in the file into a Jena model FileInputStream inFile = new FileInputStream(importFile);

model = model.read(importFile,null,null);

inFile.close();

// Export the contents of the model into a file

FileOutputStream outFile = new FileOutputStream(exportFile);

String queryString = "DESCRIBE ?x WHERE {?x ?y ?z}";

Query query = QueryFactory.create(queryString);

QueryExecution qexec = QueryExecutionFactory.create(query, model);

model_out = qexec.execDescribe();

model_out.write(outFile);

// Close the models model.close();

model_out.close();

} catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); }

}

Figure 6.7: Implementation of the ImportExport method for Jena.

comparer (KAON2 OWL Tools¹³ version 0.27).

Nevertheless, other ontology comparers can also be inserted into the IBSE tool by implementing a method from the Comparer interface: int Compare-Files(String origin file, String compared file, String added file, String deleted file, String language). This method receives the following input parameters: the lo-cation of the two files to be compared; the lolo-cation of the two files in which the inserted and removed triples will be stored; and the language in which the ontologies are written respectively.

Since the software used for ontology comparison could have execution prob-lems, we needed a previous evaluation of this software to ensure the validity of the benchmarking results.

The evaluation of the comparers, which consists in detecting errors in them, was performed in two steps:

1. The interoperability experiment was carried out with the tools that have OWL as knowledge model, because these tools should interchange all the ontologies correctly since no ontology translation is required for doing so.

In this step, we analysed the cases where the interchanged ontology was different than the original one.

2. The interoperability experiment was carried out with all the tools. In this

13http://owltools.ontoware.org/

step, we analysed the cases where the comparison of two ontologies caused an execution error in the comparer.

After carrying out the previous steps we found several problems in the KAON2 OWL Tools ontology comparer. These problems were the following:

When one of the ontologies was empty, the comparer returned that the ontologies were the same.

The comparer returned complete definitions of the differences between the ontologies and not only the differing triples. For example, if two ontologies only differ in one triple

Ontology 1: Ontology 2:

ns1:Person rdfs:type owl:Class; ns1:Person rdfs:type owl:Class;

ns1:Person rdfs:label "Person";

the comparer returned not just the triple but also the whole definition of the classes or properties involved:

Diff:

ns1:Person rdfs:type owl:Class ns1:Person rdfs:label "Person";

When the comparer compared two ontologies with blank nodes, it gener-ated different node identifiers and, therefore, it returned that the ontolo-gies were different.

When one of the ontologies was not a valid OWL ontology in the RDF/XML syntax, the comparer throwed an exception.

The comparer is not robust and threw an exception when it compared ontologies with unexpected inputs, as for example, the incorrect class naming produced by some tools¹⁴, or the incorrect use of the OWL lan-guage constructors, i.e., use of rdf:Property instead of owl:ObjectProperty or owl:DatatypeProperty; use of a resource both as an object and as a datatype property; use of rdfs:subclassOf statements with no object (“<rdfs:subClassOf/>”); or use of untyped object properties.

The first two problems were solved by adapting the output of the comparer inside IBSE. The behaviour of the ontology comparer in the other cases was documented and taken into account when analysing the interoperability results.

This is not an exhaustive evaluation of the comparer but, after analysing all the cases of the whole benchmarking results in which the interchanged ontologies were not the same, we found no more comparer errors.

14i.e., “#http 3A 2F 2Fwww.w3.org 2F2002 2F07 2Fowl 23Thing”