Holistic quality ratings or reader-focused methods (Schriver, 1990) are a broad category of text assessment where the rater’s evaluation is based on given criteria requiring understanding and interpretation of text. A primary trait scoring is a kind of holistic rating, consisting of a set of criteria for successful writing on a selected genre. The focus of scoring is on a narrow range of aspects or traits, the most salient criteria associated with the task (De la Paz & Graham, 2002; Lloyd-Jones, 1977). In order to evaluate the impact of the Dialectic Method on writing performance a primary trait rubric is adopted (Trigwell, 1992) and applied as a marking method of the essay content. An independent evaluator reads the text and applies the rating scheme, following guidelines.
3.5.2 Analytic procedure
In order to evaluate the impact of the Dialectic Method on writing performance, a primary trait rubric was used (Trigwell, 1992). It is based on an essay grading scheme which was originally proposed by the Educational Services and Teaching Resources Unit at Murdoch University (Table 3.3). The application of guidelines was also adopted (Table 3.4). The scoring was done by an independent evaluator, a teacher from the EISU School who was not involved in the study.
The primary trait rubric consists of four parts (Table 3.3). In the first part, the GENERAL STRUCTURE items Relevance of topic, coverage of topic and adequate analysis refer to rhetorical, content and textual aspects of the essay. The four items grouped under ARGUMENT, refer to argumentation strategies or argument moves. The wording in the 3rd and 4th item of ARGUMENT anticipates the possibility that the participant has not voiced a personal position clearly. The CONCLUSION section refers to how the developed argumentation is reflected in the conclusion, where usually the writer’s position is presented to confirm if already presented in the introduction. One of the problems in argumentative writing is to take a position and support it throughout the essay. The reader anticipates in the conclusion information that is relevant to the writer’s position. This presupposes that the writer has explored her arguments in relation to a position (semantic structure) and in the conclusion she presents a reminder a summary or a discussion of the position. In some cases participants realise that the support of their position is weak and may decide to change their mind, even if they are half way in their writing, and to conclude with another position (Oostdam et al., 1994). This is not a good rhetorical strategy. The ENGLISH section includes one item only; language in general is possible to affect the scoring of all items.
Itemised rating scale
1.GENERAL STRUCTURE
a. Essay relevant to topic 4 3 2 1 Essay has little relevance b. Topic covered in depth 4 3 2 1 Superficial treatment of topic
c. Adequate analysis of subject 4 3 2 1 Descriptive account of subject
d. Logically developed argument
4 3 2 1 Essay rambles and lacks continuity
2. ARGUMENT
a. The student has clearly taken a position
4 3 2 1 The student has not taken a clear position
b. The student has supported her position (and other presented positions) with arguments
4 3 2 1 The student is not providing support for hers or any other presented position
c. The student is referring to both arguments and
counterarguments (or
advantages and disadvantages) in relation to her position and other presented positions.
4 3 2 1 The student is NOT referring to counterarguments (or
disadvantages) in relation to her position OR any presented position.
d. The student has mentioned arguments in favour of the opposing position and has refuted them
4 3 2 1 The student has not considered at all the opposing position nor its advantages
3. CONCLUSION
a. The student’s essay concludes nicely
4 3 2 1 The student’s essay has no conclusion
b. In her conclusion, the student refers back to one or two main points of her argumentation and draws a conclusion.
4 3 2 1 The student’s conclusion is not relevant to her argumentation
c. The student’s conclusion is consistent with her position
4 3 2 1 The student’s conclusion contradicts her position 4. ENGLISH
The written English was very good
4 3 2 1 The written English was very bad
ARGUMENT : Guidelines on the itemised rating scale 4 3 2 1 C le ar P os it ion The student is clearly taking a position; she has defined more than one position in relation to the topic in question and has clearly taken one position. The student’s intention is to take a clear position despite language difficulties or lack of conceptual clarity with regard to the topic
The student avoids commitment to one position; she is taking the ‘middle’ way. The essay lacks in a position that stands out
The student changes her mind during the essay development. She takes a position in the beginning, which later she seems disregard or ignores. S uppor t t o pos it
ion The student is supporting
(developing) her position and possible opposing position(s) with arguments The student is supporting her position but little is said in favour of the stated opposing position.
The essay develops around one
position. Other positions maybe stated but not supported. Very little or no information is provided in support of a position. C ount er ar gum ent s All positions presented,
including the one the author is taking and possible opposing ones are developed with arguments and counterarguments. The positions presented, are developed with lesser counterarguments than when a 4 is normally awarded Lesser and unsuccessful attempts to develop counterarguments are noted. Unsuccessful counter argumentation may be related to lack of conceptual clarity with regard to the topic or poor knowledge of linguistic indicators for expressing counter argumentation. no counter argumentation is noted R ef ut at ion Most counterarguments are refuted. Some successful and some unsuccessful attempts to refute counterarguments are noted.. Counterarguments are not refuted or are unsuccessfully refuted.
The student does not consider arguments supporting the opposing position or does not consider at all the opposing position. Counter argumentation and refutation are not applicable.
3.5.3 Evidence of reliability
The primary trait rubric consists of 12 items which are grouped in 4 sections: General structure (4 items), Argument (4 items), Conclusion (3 items) and English (1item). The 12 items of the primary trait rubric were tested for scale reliability in order to confirm that they measure the same construct as a scale. Cronbach's alpha was .92 for the pretest scores and .91 for the posttest.
When the reliability of the hypothesized subscales were tested separately alpha for General structure was .92 for pretest scores and .91 for the post test; alpha for Argument was .55 for pretest scores and .72 for posttest; alpha for Conclusion was .85 for pretest scores and .95 for posttest. However, those subscales were not used further as they were not backed by adequate empirical evidence.
A factor analysis would have help to validate such subscale structure. However, due to the limited sample size it was not possible to conduct a factor analysis. Instead, a new construct, the ‘overall quality’ of argumentative essay is computed on the basis of the mean value of the 12 items (‘OVERALL QUALITY pre’ and ‘OVERALL QUALITY post’).
3.6 Argument structure analysis
3.6.1 Theoretical rationale
The analytic approach starts from properties of the text that are easily identifiable. For example, analysis or representation of text structure based on propositional analysis “offers a formal system for representing the micro- and macrostructure of a document” (Kellogg, 1994, p.59). In analytical approaches, such as propositional analysis (Kintsch, 1974), coherence
analysis (Halliday & Hasan, 1976), and feature analysis (Sharples, 1985), the categories of analysis, are applied at the sentence, connective conjunctives, and word level.
The role of structure analysis in the evaluation of essays should be seen in the context of objective analysis that starts from properties of the text and aims to evaluate complex structures of argumentation. In second language (L2) assessment, language difficulties may obscure the writer’s actual attempt to formulate argumentation. The adopted method of discourse structure analysis examines the structures and underlying procedures from a different perspective than that of the essay evaluator. The emphasis is more on the argument structures than on the accuracy of language used to express these structures.
Crammond’s model (1997, 1998) of representing argumentation structures is considered suitable for analysing the complexity of argumentation structures. Argumentation structure is understood as a complex network of semantically interlinked arguments. Argument is defined here in terms of a modified and elaborated Toulmin model (Crammond, 1997, 1998), consisting of claim, data, warrant and other components that will be discussed later.
Crammond’s model is adopted for four main reasons. First, Crammond’s model represents the structure and interdependency of arguments in extended persuasive discourse, i.e. argumentative discourse that develops around a main claim or position. Second, it defines the role of text segments in a complex argumentation structure and the relations between arguments including the functions of counter argumentation and refutation.
Third, it can serve as point of reference for estimating the participant’s level in terms of the range and complexity of argument structures they use. Crammond’s model has already presented empirical results in identifying developmental differences between experts and students in using complex argumentation (Crammond, 1997, 1998) and has been considered
for its analytical potential (Coirier et al., 1999; Lunsford, 2002; Yeh, 1998a, 1998b). Although further validation of the model with L2 writers may be needed (given the current study participants are non-English native speakers), it can serve as a point of reference for establishing the level of argumentation skills of the participants. Other methods of discourse structure have been reviewed, such as PISA (Sanders & van Wijk, 1996; van Wijk & Sanders, 1999) and Rhetorical Structure Theory (Mann & Thompson, 1988, 1992). However the scope of their conception is generic to text analysis and not specific to argumentation. They have been used in the analysis of argumentation structure but there are limited empirical findings on the evaluation of argumentative text as result of an intervention or as definition of developmental levels of argumentative writing expertise.
Finally, with a small adaptation the model is possible to represent argumentation structures at both a Macro- and Microstructure level. Table 3.5 describes 4 levels of representation and illustrates how Crammond’s model is used in this thesis in order to conceptualise 4 levels of argumentation structure analysis. This analysis starts at a Micro-structure level, level 1, where the text is segmented in T-Units. The text segments are encoded according to the Argument Structure Grammar (Level 2), where the use of codes and relation proposed by the Grammar (Crammond, 1997, 1998) are considered at a Microstructural level. Moving onto the following level, level 3, the scope of analysis refers to how the components encoded in the previous level contribute to the coherence of the text. In other words, the text is now seen from a Macro-structure point of view. Complexity of argumentations structures refers to chains of arguments, how arguments (as defined above) are embedded in other argument structures, thus, applying the proposed model (Crammond, 1997, 1998) in a recursive way. The Argumentation Structure Components encoded in Level 2, and interlinked (co-ordinately, subordinately combined or embedded) at Level 3, are assigned to 4 categories according to
the main or implied position of the text. This is considered to be level 4, where Crammond’s model is extended to define the balance and orientation of argument structure components.
Level Title level Scope Measures
Level 4 Orientation and Balance of Argument Structure components Orientation of Argumentation Structure components with regard to the position Support Counterargumentation Refutation (Neutral) Non-argumentative Level 3 Complexity of argumentation structures (Crammond 1997; 1998) Coherence of essay. Embedded arguments Argument chain depth Number of chains
Density of arguments per text Subclaims Level 2 Argument structure components (Crammond 1997; 1998) Use of argument structure components: Application of Argument Structure Grammar
Argumentation Grammar structures: Support and Justification
Counterargumentation and Refutation
Level 1 Segmentation Objective unit of
analysis Text Segments, T- Units
Table 3.5: Levels of representation of argumentation structures in Crammond’s model
Crammond’s analytical model derives from a model of semantic representation in discourse. The complex text analysis procedures proposed by the model refer to the framework of Frederiksen (Frederiksen, 1975, 1987), which allows a precise description of written argumentative texts. Crammond’s model (1998) was conceived in order to identify the developmental features and characteristic weaknesses of students’ persuasive writing by referring to argument structure. Her model is based on Toulmin’s (1958) model but also modifies Toulmin’s schematic representation by allowing two aspects of complex argumentation to be represented, which are very important in the current analysis and representation of Argumentation Structure.
First, argument in Toulminian terms is used as a unit of analysis of persuasive discourse. The basic claim-data and warrant model is validated elsewhere (Knudson, 1992; Scardamalia & Paris, 1985) as the most significant predictor of holistic writing scores, assigned to students’ texts. But what is more important is that Crammond’s model allows the analysis of extended persuasive discourse. Her elaborative modifications to Toulmin’s model allow the representation of chains of arguments. Chains of arguments can be created by subordinately compound arguments. Chains of arguments, related in coordinated way, form a tree like graph, namely, the entire argument model of extended persuasive discourse (Crammond, 1998, p.237). Seeing argumentation structure from this perspective allows us to discuss complexity of argumentation structure.
Second, Crammond’s (1998) model gives emphasis to counter argumentation and refutation by including some new components to Toulmin’s basic model. The countered rebuttal consists of a potential rebuttal, in other words a challenging (or counter arguing) statement, and a response to rebuttal, that is the refutation to the challenging statement (see example in Figure 3.5, p.85). The component of potential rebuttal, as well as the reservation component (equivalent to the exception component in Toulmin’s terms, which limits the applicability of a claim) and alternative solution component are forms of counter argumentation.
Figure 3.5: Example of tree diagram yielded by the application of the model COUNTERED REBUTTAL.2: RESPONSE TO REBUTTAL2: ARGUMENT .1
DATA .1: It helps to keep control over animals.
CLAIM .1: I believe that it is acceptable for animals to be trained by humans
CLAIM .2:I believe that it is good to train animals
to entertain…
DATA .2: so that they can display their talents and learning capabilities
POTENTIAL REBUTTAL.2 :
I am not sure if the animals know what applause is…
SUBCLAIM .1.1
ARGUMENT .2
ARGUMENT .3
CLAIM .3: …but they at least sense the audience is
praising them for their efforts
DATA .3: If you have ever see an owner praise a
dog you know that the dog shows appreciation
3.6.2 Analytic procedure
Table 3.5 (p. 83) illustrates 4 levels that reflect an incremental application of argument structure semantics starting from the text segmentation level (level 1). Except level 1, which refers to the segmentation of text in units, the other 3 levels represent different aspects of semantic and rhetorical argumentation structure analysis.
The analysis procedures start with the segmentation of the text. At segmentation level, the analysis of text in segments is done by identifying a) major (finite) clause and b) adjuncts bound clauses. The latter are secondary clauses introduced with binders (until, when, because, if, since, etc.). As in Crammond’s linguistic analysis (Crammond, 1997, p.46), the clause analysis is based on Winograd’s (Winograd, 1983) description of clauses types. Segments are then numbered and the total number is noted.
At level 2 and 3, the essay protocols were analysed on the basis of an Argument Grammar (Appendix II) formalised in a set of production rules (Crammond, 1997, 1998). The links proposed by Crammond were simplified to facilitate the coding process, while the proposed components were kept the same (Appendix III). The components of Argument Grammar (claim, subclaims, data, data backing, warrant, warrant backing, constraint, potential rebuttal, countered rebuttal, reservation and alternative solution) were identified in the essays by one analyst, in this case the author of the thesis.
Figure 3.6: Tree structure of essay representing hierarchy and complexity of argumentation structure components
The components Claim, Data, Data Backing, Warrant, and Warrant Backing, are defined as in Toulmin’s model. Definitions for the remaining Subclaim, Constraint, Reservation, Potential Rebuttal, Countered Rebuttal (including a potential rebuttal and a response to rebuttal), and Alternative Solution are given in Appendix I. For a detailed description of the components and relation of Argument Grammar the reader may refer to (Crammond, 1997, p. 51-60) and (Crammond, 1998, p.257-264).
The analysis of argument structure of the segmented texts was facilitated by atlas.ti, a software tool that supports the management of qualitative analysis and representation of results. Figure 3.7 shows one of the essays annotated with the Argumentation Structure
Components in atlas.ti. Note that the last two sentences are coded as Response to Rebuttal. Figure 3.6 shows how the annotated components are interlinked forming a hierarchy of structures.
In order for a piece of discourse to be classified as an Argument, it has to consist minimally of a Claim-Data complex. Such Argument units are identified and, along with Claims, indexed (numbered) for reference purposes subsequent to the coding of argument components. This step involves the double coding of some text segments. The double coding is possible for the Subclaim, Data, Warrants, Reservations, Countered Rebuttals and Alternative Solution. Each of this can be coded as an argument, consisting at least of a Claim-Data complex, the basic argument. The argument is then considered an embedded one. Each of the above mentioned structures could also be coded as a claim only, to which may be related a subclaim. In this case although we do not have embedded arguments a chain is formed. Thus chains are generated when embedded Arguments and Subclaim relations are coded.
The analysis of each essay according to the Argument Grammar (Crammond, 1997) yields a tree structure diagram for each essay. Figure 3.6 is the tree structure diagram for the text annotated with the Argument Grammar components, shown in Figure 3.7.
Figure 3.7: Essay annotated with Argument Grammar Components
Levels of argument structure analysis
Level 4: Balance and Orientation of Argumentation Structure components
Forty-two essays, 9 from the paper group and 12 from the computer group, written before (protest) and after (posttest) the intervention, were analysed according to the Argument Grammar and procedures presented in the previous sections. The components of Argument Grammar (structure, claim, subclaims, data, data backing, warrant, warrant backing, constraint, potential rebuttal, countered rebuttal, reservation and alternative solution) were identified in the essays. The components were then grouped semantically in 4 greater categories, according to their relation to the main position of the essay:
2. COUNTER: components challenging the position or expressing counter argumentation,
3. REFUTE: components refuting the challenging statements
4. NEUTRAL: components referring to scene setting, or background information.
The components’ number from each category is divided with the number of the segments in each essay. Thus, the length of the essay, and specifically the main and secondary clauses, are considered when the balance and orientation of argumentation structure components is computed. The four measures on level 4 are therefore:
1. SUPPORT: supporting components / T-Units per essay 2. COUNTER: counter arguing components / T-Units per essay 3. REFUTE: refuting components / T-Units per essay
4. NEUTRAL: neutral components / T-Units per essay
Level 3: Complexity of Argumentation Structure
The length of argument chains and number of embedded arguments are counted as a measure of argument complexity. As already mentioned chains are generated when embedded Arguments and Subclaim relations are coded. The example (Figure 3.5), given by Crammond, illustrates this measure. In the example the SUBCLAIM 1.1 and RESPONSE TO REBUTTAL .2 are double-coded as embedded Arguments, ARGUMENT .2 and ARGUMENT.3 respectively. The depth of the argument chain is 3 because, starting from the top ARGUMENT.1, two more levels of argument follow, represented by embedded arguments, ARGUMENT .2 and ARGUMENT.3
1. Embedded arguments, as defined above
2. Argument chain depth- maximum length: The depth measure represents the longest argument chain. Embedded arguments and subclaim relations form