Entrades de cinema en català per tipus de versió 2012

The notion of ontology quality and the process of evaluating it is one of the most significant and also complicated challenges in the ontology domain. Recent evolution in the semantic web has led to an incredible increase in the number of available ontologies and therefore, the main challenge today is not finding ontologies, but it is to find the right ones, or as Kolbe et al. (2019) called it, the “well-fitting and requirements-meeting” ones. Despite the importance of this matter and the widespread research on this topic, there are still many unanswered questions about ontology evaluation and selection for reuse.

Diversity of tools, languages, and methodologies, lack of a generic framework for building ontologies and also identifying the relevant quality elements that should be used for ontology assessment and evaluation are among some of the main challenges in the ontology domain (Burton-Jones et al., 2005; Hlomani and Stacey, 2014). A considerable number of tools and systems have been proposed in the literature to support the process of ontology selection and reuse; however, many of them are not available anymore. Moreover, most of the researches in

the ontology domain are concerned with constructing a new approach, instead of coming up with a generic one or something like a common benchmark for ontology development.

Identification of the set of metrics that should be used in the evaluation process has known to be a central issue of this domain (Burton-Jones et al., 2005). As was discussed in section 2.5, a wide range of metrics have been proposed in the literature to help ontologists and knowledge engineers assess the quality of ontologies before selecting them for reuse. However, as seen in the literature, some of those proposed metrics are very similar and different names have been used to refer to the same criterion. Moreover, previous studies have failed to identify the importance of each of those metrics in the ontology evaluation process. Metrics proposed for ontology evaluation usually tend to be useful in a specific context or application and it is hardly possible to find a general quality metric that will work for different use cases and scenarios (Burton-Jones et al., 2005).

Knowledge captured by ontologies might change regularly and not everyone in a domain might agree on what is presented as facts in an ontology or their meaning and relationships (Bard and Rhee, 2004). Hence, subjectivity has been considered as one of the issues that the research in ontology evaluation domain is concerned with. According to the literature, the main focus of many of the current approaches is on identifying and measuring different objective characteristics of ontologies, such as consistency, semantic validation, and hierarchy. In the real world, however, ontologies are usually evaluated subjectively, meaning that ontologists and knowledge engineers are usually looking for a “well-defined” ontology that “best” fits their application requirements.

Hlomani and Stacey (2014) have identified three different types of subjectivity in ontology evaluation: (1) subjectivity in the selection of the criteria for ontology evaluation, (2) subjectivity in the thresholds for the measurements of each criterion, and (3) influences of subjectivity on the overall quality evaluation. As a part of the ontology evaluation process, it is important not only to come up with a set of right criteria that can be used to evaluate ontologies, but it is also very important to establish who the right ontology evaluators are (Hlomani and Stacey, 2014).

Supekar (2005) blamed the evaluation approaches of their time for neglecting the importance of subjective qualities of ontologies and for not providing helpful subjective information, such as peer reviews and ratings for ontologies. By contrast, Gangemi et al. (2006) argued that

automatic or semi-automatic techniques should be applied to ontology evaluation and make the evaluation process less subjective. Despite all the endeavour in this field, there is still no automatic method or approach that can be used to assess the quality of an ontology (Amith et

al., 2018) and ontology evaluation and selection has always been based on some kind of human

experts’ judgment (Lewen and d’Aquin, 2010). Hence, subjectivity and bias are some of the inevitable parts of these types of evaluation processes; this is against the idea of the good science, that is to exclude subjectivity from scientific experiments (Hlomani and Stacey, 2014). Community is one of the most significant notions in the ontology domain. One of the main aims of developing ontologies was and still is to use them as a shared conceptualization between different groups of people working in the same domain (Gruber, 1993). However, most studies in the field of ontology evaluation have only focused on syntactic and structural aspects of ontologies and have failed to address how social aspects and community recognition affect the quality of an ontology (Mcdaniel, Storey and Sugumaran, 2016).

As was mentioned before, some of the evaluation approaches like OntoMetric (Lozano-Tello and Gómez-Pérez, 2004), Semiotic Metric Suite (Burton-Jones et al., 2005) and NCBO BioPortal Recommender (Jonquet, Musen and Shah, 2010) have identified social based quality metrics; however, Mcdaniel, Storey and Sugumaran (2016) argued that they are not able to fully evaluate the level of ontology acceptance in the community. It can be argued that social quality of an ontology might depend on a wide variety of metrics, rather than being only based on acceptance or popularity.

Besides different dimensions of ontologies that have been studied in the literature and different metrics that have been identified to assess those dimensions, how evaluation metrics are measured is the other very important aspect of ontology evaluation. Mcdaniel, Storey and Sugumaran (2016) blamed some of the measurement approaches for reducing the assessment accuracy. Acceptance in BioPortal, for example, is based on the existence of or the number of visits to an ontology in a specific website; therefore, this metric is not applicable to the ontologies that are in other libraries (Mcdaniel, Storey and Sugumaran, 2016).

To address some of the above-mentioned shortcomings, this study investigates the ontologists and knowledge engineers’ opinions about the notion of quality and the factors it depends on. Moreover, this research aims to examine the role of community in ontology evaluation and determine how social interactions can help in the evaluation process. The importance of

different quality metrics will also be assessed. Finally, it will be investigated if there exists a generic set of quality metrics that people tend to consider for ontology evaluation, no matter in what domain they work in.