Microdureza y caracterización tribologica

We first automatically compiled a list of verb-noun expressions, to be anno- tated by human experts. This is based on previous attempts at extracting a lexicon of MWEs, as in Villavicencio (2005). Annotators were not provided with any context making the task more feasible in terms of time. Human annotators were asked to label the expressions as MWEs only if they have sufficient degrees of idiomaticity. In other words, a verb-noun MWE does not convey literal meaning in that the verb is delexicalised.

In this phase, an expression type (rather than token) was evaluated based on its potential literal/idiomatic interpretations. For example in the expres- sion take a break, the light verb take does not carry literal meaning and is an MWE, while have coffee will be marked as literal as the verb have bears the

literal meaning of drinking. Some expressions potentially have both kinds of interpretation e.g. have a baby which other than its transparent meaning, can have the idiomatic interpretation of giving birth in she had a baby in ABC hospital. We considered these three possibilities (literal, idiomatic, or both) for an expression type in this experiment.

The experiment initially covered Spanish and was performed after having tested the guidelines for two rounds of pilot annotations in English. More specifically, we first extracted verb-noun expressions in English from BNC and in Spanish from SpanishWaC. Two annotators marked up the expres- sions. By analysing the disagreement between annotators, and based on their feedback, we improved and finalised the guidelines. The finalised annotation task involves three tags: tag 1 (MWE) if the expression is idiomatic; tag 0 (non-MWE) if the expression is literal. We also introduced tag 2 for the expressions which in some contexts behave as MWEs and in others not, e.g. have children, which in some contexts means to give birth and hence can be an MWE. The finalised guidelines were applied to Italian annotations.

We focused on four of the most frequent verbs: fare, dare, prendere and trovare in Italian and tener, hacer, formar and tomar in Spanish.1 Using SketchEngine (Kilgarriff et al., 2004), we extracted all the occurrences of these verbs when followed by any noun, from the itWaC corpus for Italian 1_{The verbs were selected from the list of most frequent light verbs, by native-speaker}

annotators who are knowledgable of the MWE phenomena. The idea was to cover as much productive and ambiguous verb-noun expressions as possible. The translations for the light verbs are not provided, since they are polysemous verbs with several different potential translations.

and SpanishWaC for Spanish. All extracted verb-noun expressions had the verb lemmatised.

After removing all occurrences with a frequency lower than 20, the ex- traction of verb-noun candidates featuring the above four verbs in Italian resulted in a dataset of 3, 375 expressions. Two native speakers annotated every candidate expression with 1 for an MWE if the expression is idiomatic, with 0 for a non-MWE if the expression is literal, and 2 for the expressions which in some contexts behave as MWEs and in others do not. An example of this is dare frutti, which has a literal usage that means to produce fruits but in some contexts means to produce results and is an MWE.

The observed agreement between the annotators was 0.73. The observed agreement is simply the percentage of expressions annotated identically by the two annotators, without considering the chance agreement. Different coefficients have been defined in order to calculate chance-corrected agree- ments (Artstein and Poesio, 2008). In the case of this study with only two annotators, we did not register much variation between different coefficients. Therefore we chose to report the most common measure which is Cohen’s Kappa coefficient (Cohen, 1960). The Kappa measure was only 0.40 for the annotation of Italian out-of-context expressions.

For Spanish, since the corpus is smaller, we set a lower threshold and removed the expressions with frequencies lower than 10. In several rounds of pilot annotations, we observed insufficient inter-annotator agreement. In the end, the first annotator marked up all 1, 924 expressions and the sec-

ond annotated almost 33% (623 expressions), according to which the Kappa agreement was measured to be 0.36 and the observed agreement was 0.66.2

Even though this out-of-context ‘fast track’ annotation procedure saves time and yields a long list of marked-up expressions which could be useful in upstream NLP tasks, the results of this kind of annotation are not promising enough. Annotators often feel uncomfortable due to the lack of context. The low rate of agreement between annotators is indicative of the challenge. Also, we believe that idiomaticity is not a binary property; rather it is known to fall on a continuum from being completely semantically transparent, or literal, to entirely opaque, or idiomatic (Fazly et al., 2009). This makes the task of out-of-context marking-up of the expressions more challenging for annotators, since they have to choose a value according to all possible contexts of a target expression. This difficulty and the fact that there are many expressions that in some contexts are MWEs and in some contexts not, prompted us to initiate a subsequent annotation task and data preparation where MWEs are tagged in their contexts.