Propuesta Arquitectónica:

In this chapter, we focus on the interpretation of AA ‘marking criteria’. Visualisation techniques can help us shed light on AA ‘black boxes’, and inspect the features they yield as the most predictive of a learner’s level of attainment. As data-driven approaches are quantitatively very powerful, visualisation can help us gain a deeper understanding on their workings. The latter is particularly important for learning models designed to imitate the value judgements examiners make when they mark a text. We build a visual user interface (hereafter UI) which allows investigation and interpretation of a set of linguistic features discriminating between passing and failing FCE ESOL exam scripts. The UI displays directed graphs to model interactions between features and supports exploratory search over FCE learner scripts. Our experiments demonstrate that proper analysis and visualisation of AA features can support SLA research, and, in particular, can shed light on understanding the linguistic abilities that characterise different levels of attainment and, more generally, developmental aspects of learner grammars. Additionally, we illustrate how hypothesis formation through visualisation of discriminative features can aid the identification of new discriminative features, and thus further contribute to informing the development of AA systems.

The UI is developed to analyse features described in Briscoe et al. (2010). Briscoe et al. have also treated FCE AA as a classification problem, and used a binary discrimi- native classifier to learn a linear threshold function that best discriminates passing from failing FCE scripts, and predict the class to which a script belongs. To facilitate learning of the classification function, the data should be represented appropriately with the most relevant set of features. As mentioned in the previous chapters, they found a discrim- inative feature set which includes, among other feature types, word and POS ngrams. We extract the discriminative instances of these two feature types and focus on their linguistic analysis. Table 5.1 presents a small subset ordered by discriminative weight. A major advantage in using (supervised) discriminative classifiers to support hypothesis formation over, for example, clustering techniques, is that they assign weights to features

Feature Example VM RR (POS bigram: +) could clearly , because (word bigram: −) , because of necessary (word unigram: +) it is necessary that the people (word bigram: −) *the people are clever

VV∅ VV∅ (POS bigram: −) *we go see film

NN2 VVG (POS bigram: +) children smiling

Table 5.1: Subset of features ordered by discriminative weight; + and − show their association with either passing or failing scripts.

representing their relative importance.

We believe the investigation of discriminative features can offer insights into assessment and into the linguistic properties characterising the relevant CEFR level (see Chapter 1, Section 1.4.2), which can, in turn, be exploited to identify new discriminative patterns that further improve performance of AA systems. However, the amount and variety of data potentially made available by the classifier is considerable, as it typically finds hundreds of thousands of discriminative feature instances. Even if investigation is restricted to the most discriminative ones, calculations of relationships between features can rapidly grow and become overwhelming. Discriminative features typically capture relatively low-level, specific and local properties of texts, so features need to be linked to the scripts they appear in to allow investigation of the contexts in which they occur. The scripts, in turn, need to be searched for further linguistic properties in order to formulate and evaluate higher-level, more general and comprehensible hypotheses which can inform reference level descriptions and understanding of learner grammars.

The appeal of information visualisation is to gain a deeper understanding of important phenomena that are represented in a database (Card et al., 1999) by making it possible to navigate large amounts of data for formulating and testing hypotheses faster, intuitively, and with relative ease. An important challenge is to identify and assess the usefulness of the enormous number of projections that can potentially be visualised. Exploration of (large) databases can quickly lead to numerous possible research directions; lack of good tools often slows down the process of identifying the most productive paths to pursue.

In our context, we require a tool that visualises features flexibly, supports interactive investigation of scripts instantiating them, and allows statistics about scripts, such as the co-occurrence of features or presence of other linguistic properties, to be derived quickly. One of the advantages of using visualisation techniques over command-line database search tools is that SLA researchers and related users, such as assessors and teachers, can access scripts, associated features and annotation intuitively without the need to learn query language syntax.

We modify previously-developed visualisation techniques (Battista et al., 1998) and build a visual UI supporting hypothesis formation about learner grammars through vi- sualisation of discriminative features. Features are grouped in terms of their relative co-occurrence in the corpus and directed graphs are used in order to illustrate their re- lationships. Selecting different feature combinations automatically generates queries over FCE data and returns the relevant scripts as well as associations with meta-data and dif- ferent types of errors committed by the learners. In the next sections we describe in detail the visualiser, illustrate how it can support the investigation of individual features, and

discuss how such investigations can shed light on the relationships between features and developmental aspects of learner grammars. Furthermore, we illustrate how hypothesis formation through discriminative features can aid the identification of new discriminative features. In the last section of this chapter, we evaluate the visualiser through usability testing and user feedback; ensuring its quality is essential to successful use by target users. To the best of our knowledge, this is the first attempt to visually analyse as well as perform a linguistic interpretation of discriminative features that characterise learner English, whose analysis can also inform the development of AA systems. We would also like to point out that we also apply the visualiser to the publically-available FCE ESOL texts (see Chapter 2, Section 2.1.1.1) and make it available as a web service to other researchers.2