Contextualització i fonamentació

The transformation scheme implementing the transformCEV function in Algo-

rithm 1 (page 133) depends on accurate classification of signs of syntactic com- plexity. Information about the classes of signs is required to detect the different elements of the handcrafted rule activation patterns in input sentences (presented in Section 5.2.2). The method for sentence transformation exploiting machine- learned activation patterns also exploits information about the tags applied to signs of syntactic complexity. Accurate sign tagging serves as an important first step in applying the associated transformation schemes.

Matching of the handcrafted rule activation pattern used in the transformation scheme for transformCEV depends on accurate detection of two classes of signs:

2. left boundaries of subordinate clauses ({SSEV, SSCM, SSMA, SSMAdv, SSMI, SSMN, SSMP, and SSMV})

Thus, when simplifying Type 1 sentences, confusions between tags of the set specified in 2 are irrelevant (e.g. confusion between SSEV and SSCM). By contrast, confusion between tags of different sets specified in 1 and 2 is relevant (e.g. confusion between CEV and SSMA). Table 5.4 displays the accuracy with which the sign tagger assigns the two sets of class labels relevant to simplifying Type 1 sentences. There, row SSX pertains to signs tagged with any of the class labels in the set listed in 2. Considered over the full set of signs, the tagger assigns these class labels with a micro-averaged F1-score of 0.9318.

Table 5.4: Evaluation of the sign tagger over tags exploited in the simplification of Type 1 sentences

True- False- False-

Tag P R F1 Support Pos Pos Neg

CEV 0.7991 0.7991 0.7991 876 700 176 176

SSX 0.9794 0.9251 0.9515 6076 5621 118 455 Micro average:

All 0.9556 0.9092 0.9318 6952 6321 294 631

The transformation scheme implementing the transformSSEV function in Al-

gorithm1depends on accurate detection of four classes of signs. This information is required to detect, in input sentences, different elements of the handcrafted rule activation patterns. Once these elements have been identified, the rules imple- menting the associated transformations are easy to apply. The four classes of signs are:

1. noun phrase coordinators (CMN1),

2. right boundaries of finite relative clauses (ESEV),

3. right boundaries of direct quotes (ESCM), and

4. left boundaries of subordinate clauses ({SSEV, SSCM, SSMA, SSMAdv, SSMI, SSMN, SSMP, or SSMV}).

Thus, for sentence simplification, confusions between tags in the set speci- fied in 4 are irrelevant (e.g. confusion between SSEV and SSCM). By contrast, confusion between tags in the sets specified in 1-4 are relevant (e.g. confusion between SSMA and ESEV or between CMN1 and SSMP). Table5.5 displays the accuracy with which the sign tagger assigns these four sets of class labels. There, row SSX pertains to signs tagged with any of the class labels in the set specified in 4. The sign tagger assigns class labels belonging to these four sets to signs with a micro-averaged F1-score of 0.8862.

Table 5.5: Evaluation of the sign tagger over tags exploited in the simplification of Type 2 sentences

True- False- False-

Tag

P

R

F

Support

Pos

Pos

Neg

CMN1 0.7286 0.6628 0.6942

1041

690

257

351

ESEV 0.5261 0.4789 0.5014

1041

272

245

296

ESCM 0.9207 0.9379 0.9292

322

302

26

20

SSX

0.9794 0.9251 0.9515

6076

5621

118

455

Micro average:

All

0.9142 0.8598 0.8862

8480

6885

646

1122

Over all the tags exploited in the two types of sentence simplification, the tagger assigns class labels with a micro-averaged F1-score of 0.9075.

For the approach to sentence simplification based on machine-learned rule ac- tivation patterns (Section5.2.3and Chapter4), implementation of these patterns depends on accurate identification of clause coordinators (signs of class CEV) and the left boundaries of finite subordinate clauses (signs of class SSEV). As already observed, the sign tagger is able to identify these signs accurately (F1 = 0.7791

for signs of class CEV while F1 = 0.9467 for signs of class SSEV). Information

on the tags of other signs occurring in the same sentences as these items was also found to be a useful feature in the tagging models for compound clauses and complexRF NPs.

Considered over all signs of syntactic complexity, with a micro-averaged F1 =

0.7991, I am optimistic that the sign tagger will be useful for matching both the machine-learned and handcrafted rule activation patterns in English text and implementing the sentence transformation schemes specified in Section5.2.1.

5.3 Contribution to Research Questions

RQ-3,

RQ-

4, and

RQ-5

The work described in this chapter makes partial contributions to three of the research questions set out in Chapter 1.

In response to research question RQ-3:

To what extent can an iterative rule-based approach exploiting auto- matic sign classification and handcrafted patterns convert sentences

into a form containing fewer compound clauses and fewer complexRF

NPs?

the chapter presents a generic sentence simplification algorithm which exploits an automatic sign classifier (described in Chapter 3), two sets of sentence trans- formation schemes (Sections 5.2.1.1 and 5.2.1.2), and a set of rules exploiting handcrafted activation patterns which implement those schemes (Section 5.2.2). The intrinsic evaluation of this approach presented in Chapter 6 will make a direct contribution to RQ-3.

In response to research question RQ-4:

How does the accuracy of automatic sentence simplification compare when using a machine learning approach to detect the spans of com- pound clauses and complexRF NPs and when using a method based on

handcrafted patterns?

this chapter includes a description of the development of a generic sentence simplification algorithm (Section 5.2.1) and a set of sentence transformation schemes (Sections 5.2.1.1 and 5.2.1.2) which can be implemented using machine- learned rule activation patterns.11 _{This comprises the second part of my three-}

part response to RQ-4.

In response to research question RQ-5:

Does the automatic sentence simplification method facilitate subse- quent text processing?

this chapter presents a generic sentence simplification algorithm, the sentence transformation schemes that it applies, and rule sets exploiting handcrafted ac- tivation patterns which implement those schemes. This comprises the first part of my response to RQ-5. The extrinsic evaluation of this approach presented in Chapter 7 will complete my response to RQ-5.

Intrinsic Evaluation

The evaluations described in this chapter address research questions RQ-3 and RQ-4. RQ-3 is concerned with evaluating the accuracy of a sentence simpli- fication system exploiting automatic sign classification and handcrafted rule- activation patterns, while RQ-4 is concerned with comparing the accuracy of a sentence simplification system of this type with that of a system exploiting machine-learned rule activation patterns.

In this chapter, I present my evaluation of the sentence analysis and sentence transformation methods developed in my research. This includes evaluation by comparison of the output of the methods with human simplified text (Section6.1), by reference to automatic estimations of the readability of their output (Section 6.2), and by reference to readers’ opinions on the grammaticality, comprehensi- bility, and meaning of their output (Section6.3). Section 6.1 includes evaluation of a sentence simplification method that uses handcrafted patterns (presented in Section5.2.2) and a method using machine learning to identify the spans of com- pound clauses and complexRF NPs (Section 5.2.3). Analysis of the two addresses RQ-4.

6.1 Comparison with Human-Produced Simplifi-