PREPARACIÓN DE LA MUESTRA

This section will go into detail about the development procedure of the rating scheme we created to describe and analyse (prospective undergraduate) students’ collaborative conjecturing and proving processes. We followed the guidelines proposed by Seidel and

A rating scheme for assessing process characteristics of collaborative conjecturing and proving colleagues (2005): In a first step, a theoretical foundation for the high inference rating scales was generated. We conceptualized collaborative conjecturing and proving processes as a two- dimensional construct comprising the components: (i) individual-mathematical process characteristics and (ii) social-discursive process characteristics. Since we were interested in both components of collaborative conjecturing and proving processes, our literature search included research from mathematics education as well as from psychology and the Learning Sciences. We aimed to identify a set of theory-based process characteristics of collaborative conjecturing and proving, predicting the success of collaborative conjecturing and proving processes. Process characteristics should be defined with regard to our main assumption, based on the hypotheses put forward in the literature, that students engaged with a high level of quality in one or more of these process characteristics are likely to formulate an interesting conjecture and a valid proof (valid in the sense of being accepted by the corresponding mathematical community).

Our literature search consisted of two parts: a structured search using specific key words in databases (such as the keyword “successful” AND “proving”) and a slightly more unstructured literature search performed by cross-referencing the articles that were found to be interesting within the structured search. The structured search was a part of the literature search for our research review as described in Chapter 5 (Study I). All potentially relevant literature was selected and we deduced seven process characteristics of collaborative conjecturing and proving processes, three related to the individual and four related to the social-discursive component of mathematical argumentation. Table 11 describes the process characteristics we inferred from the literature. Subsequently, the unity of analysis was chosen. Initially, we decided to make qualitative judgements over the entire collaboration process for each of the seven process characteristics. The formulation of the rating scales resulted from our identified set of process characteristics and our theoretical framework of mathematical argumentation skills (e.g., including sub-processes of exploration and systematization; cf. Chapter 2). The scaling of the individual rating scales (five quality levels) was determined in accordance with the methods used in other studies (cf. Gartmeier et al., 2015; Seidel, 2005) and in coordination with the video recordings at hand. In order to achieve a high interrater agreement, we created detail coding rules as well as descriptions and anchor examples for each rating scale (Langer & Schulz von Thun, 2007; Seidel, 2005). The levels of the rating scales roughly correspond to German school grades (1 = excellent to 5 = fail). Video-recordings that were used for developing and testing were not included in the final data analyses (Seidel, 2005). The rating scales were tested and validated with video-recordings from previous studies by comparing the judgments of two to four researchers (that discussed the rating scales in a group several times). Based on the observations during the test phase, it was decided to change the unit of analysis by dividing the entire collaboration process into two parts to reduce the complexity of

A rating scheme for assessing process characteristics of collaborative conjecturing and proving the coding process. After the rating scales were developed, the rater training started and inter- rater agreement was checked using a randomly selected sample of 10 video-recordings. A more detailed description of the rater-training is presented in the next section. The training was finished after both raters agreed that their codes are based on a shared theoretical understanding and the inter-rater agreement appeared to be good (ICC ≥ 0.6). As a last step, all 49 video-recordings of prospective undergraduate students’ collaborative conjecturing and proving processes were analysed. Figure 8 shows the entire development procedure for the rating scheme.

Figure 8: Development process for rating scales (adapted from Seidel, 2005)

In the following, we provide an overview of the fundamental coding rules and the initially constructed rating scales, and present one rating scale in more detail. It has to be noted that the descriptions of the rating scales and of the coding rules are not to be seen as a finalized product, since they only acted as support for decision making for the coding and they are to a certain degree specific to our concrete context (undergraduate student population, task, collaborative setting). However, the descriptions of the rating scales and coding rules served as basis for the rater training sessions and were adapted during the training (cf. Seidel, 2005). The final version of the coding scheme can be found in the Appendix.

A rating scheme for assessing process characteristics of collaborative conjecturing and proving 6.2.1 A brief description of the rating scheme

Instruction and basic rules:

The coding scheme is a guideline for the rating of undergraduate students’ collaborative conjecturing and proving processes. It is structured according to our conceptualization of collaborative conjecturing and proving processes, consisting of two facets, one individual and one social-discursive component. The coding scheme comprises seven rating scales, four related to individual and three related to social-discursive process characteristics. We established the following rules:

 The rating-scales serve to assess students’ collaborative conjecturing and proving processes along selected theory-based process characteristics.

 The individual rating scales, the concepts involved, and their quality levels are explained scale-wise.

 For each rating scale: The pass requirement for a quality level is that the student fulfils all its criteria. If this “minimum requirement” is not satisfied, a code for one of the lower levels has to be given.

 The raters watch the video-recording up to one half of the collaboration process and evaluate this part of the recording for the seven process characteristics, after that, the second part is watched and rated. The raters can stop and replay parts of the video- recordings as required.

 Every learner is individually assessed along the seven rating scales.

 If one half of the collaboration process contains few explicit content for a single learner and process characteristic, so that a specific rating is not possible, then there is no evaluation for those process characteristics. Code “missing data” (9) will be assigned to the coded segment for this characteristic.

6.2.2 Overview of the rating scales

The rating scales were developed in order to assess (i) to what extend ideas and arguments are formulated correctly and precisely, and to what extend arguments are reworked in the case of a presumed error or wrong direction (accuracy and precision); (ii) to what extent different ideas are developed, combined, and linked to the definitions and underlying concepts involved in the proving problem, and to what extent the learner brings in new perspectives on the collaborative conjecturing and proving process (cognitive complexity); (iii) to what extent the learner explores both what is available to use (without having any initial idea of how to proceed) and what could be proved, and to what extent the learner critically investigates different conjectures by generating examples, counter-examples, and testing the constraints (critical exploration); (iv) to what extent the learner formulates structurally sound arguments by basing claims on data and using warrants to justify the link between the data and the claim, and to what extent the learner defines the scope of the argument by using qualifiers (argument

A rating scheme for assessing process characteristics of collaborative conjecturing and proving structure); (v) to what extent the student questions the learning partner’s ideas, arguments and proving strategies to comprehend his/ her approach better, and the level of elaboration of these questions (critical question); (vi) to what extent the learner actively contributes to longer phases of a coherent joint discourse by exchanging ideas and taking the learning partner’s contributions into account, and to what extent the learner refines arguments until a joint argument is built (turn-taking); (vii) to what extent the learner identifies errors and impasses within the learning partners’ arguments, and to what extent these errors are explained and alternative solution steps are proposed (reaction to the learning partners’ errors). A more detailed description about the process characteristic argument structure is presented below as an excerpt from the coding scheme (and translated to English).

Quality levels of the process characteristics “argument structure”

Question: To what extent does the learner formulate structurally sound arguments during the proving discourse? Does the learner base his or her claims on warrants that justify the link between the data and the claim? To what extent does the learner define the scope of an argument by using qualifiers?

Criteria: Arguments that are central within the collaboration process are taken into account. These arguments are evaluated with regard to their structural elements. A central criterion is that claims are based on data and warrants that give support for the link between the data and the claim. Depending on the context of the discourse, the argument has to be qualified in order to demonstrate the degree of certainty with which a conclusion is drawn.

In each case, it is not relevant whether the claims, warrants and data are correct, but whether they are structurally complete in themselves.

Moreover, it is not central that every argument is complete, but that the parts that are essential in the discourse and that are not already obvious elsewhere are explicated.

1

The learner's arguments are detailed in their structure throughout the collaboration process, in particular at all phases of the discourse where it is possible and helpful. Claims are explicitly connected to warrants, and qualifiers are adequately used with regard to the type of warrant. Warrants usually go beyond empirical or intuitive support, and are related to the reference theory.

2

The learner's arguments go beyond the formulation of a “blank” claim. Only in a few cases, arguments lack cues as to the data on which the claim is based and with what degree of certainty the conclusion is drawn. In addition to empirical or intuitive support, warrants are mainly related to the reference theory.

A rating scheme for assessing process characteristics of collaborative conjecturing and proving

3

The learner's arguments mainly go beyond the formulation of a “blank” claim. However, at a few key points of the discourse, where it would be necessary, there is a lack of cues as to the data on which the claim is based and with what degree of certainty the conclusion is drawn. Warrants are mostly limited to empirical support.

4

The learner’s arguments that go beyond the formulation of a “blank” claim are rather rare. At many key points, there is a lack of cues as to the data on which the claim is based and with what degree of certainty the conclusion is drawn. Warrants are mostly limited to empirical support.

5

Even in the case of central arguments and phases of the discourse in which it would be appropriate and necessary, the learner puts forward claims that lack warrants and qualifiers. The arguments produced by the learner are mostly limited to the formulation of claim, possibly with few exceptions.