I developed a conversational-move coding scheme (see Appendix E) to code the syntactic contributions evident in chat-logs of group discourse during the four
aforementioned situated activities in Nephrotex. The coding scheme consists of seven functional move categories: APT-Facilitative (APT-F), APT- Conversational (APT-C), Declarations (DECLARE), Proposals (PROPOSE), Eliciting (ELICIT), Administrative (ADMIN), and Other. The first category encompasses the eight conversational moves outlined in the APT framework (Michaels et al., 2008; Adamson et al., 2012). Because the APT schema is designed as a facilitative framework for intervention purposes it was expanded for use in this analysis. Specifically, the second category (APT-
Conversational) was developed to distinguish between utterances that were facilitative in nature and those that appeared endogenously in student discourse in Nephrotex. This category is comprised of a variation of six of the eight facilitative moves designed to account for the unprompted or natural occurrence of APT moves.
While the primary concern of this study has to do with the effects of APT, early exploratory data analysis suggested that merely coding utterances as APT or non-APT could disregard other naturally occurring, and potentially salient, characteristics of discourse found in Nephrotex. To account for this, the other five functional categories were included in the coding scheme to identity other types of conversational moves. The bases of these categories were drawn from extant schema, and reflect other aspects of discourse (common in CSCL environments and/or simulation-specific) such as utterances that are about administrative or procedural business (i.e., talk about what students should be doing or have done), presentations of information/research to the group, questions
posed to teammates relevant to the group’s task, proposals about solutions or processes, and “behaviors” commonly found in online, synchronous conversations (e.g., repairs, presence/departure comments), as well as off-task comments and small talk.
This coding scheme was applied to the utterance data for each player, in each group, in each conversation (i.e., room), where an utterance is defined as every word included in a single message sent in the chat program11. Although each utterance is an
independent data point, utterances were coded in the context of the conversation to ensure meaningful interpretation. I recruited and trained an external rater to co-code
conversational moves in the data and to test the reliability of the application of the coding scheme to the chat-log data. Using a small sample of the data12, I trained the rater on the use of the scheme in each conversation to clarify the characteristics of each code, address discrepancies in interpretation, and discuss disagreements. Based on this, the rater and I calibrated and refined our application of the coding scheme, modified code definitions, and when necessary, revisited prior coding to account for new understandings or modifications of the code criteria applied later in the process.
To test for inter-rater reliability the co-rater independently coded a 50% sample of the data not used for calibration/training (n=7844 utterances, from 96
conversations). Using the results of the rater’s and my own coding of the data, I calculated and interpreted two inter-rater reliability indices. The first was the percent agreement and the second was Cohen’s kappa, which accounts for agreement by chance, to evaluate the reliability of applying the coding scheme to identify a unique
11 The coding scheme also included a code for “combine with above” used to identify contiguous utterances
by individual students, though entered as separate contributions, which were later collapsed into one contribution. This coding was included in the inter-rater reliability testing.
conversational move for each utterance. I conducted these test at two levels of the data. The first considered the reliability of coding each conversational move in the coding scheme. The second considered the reliability of coding the conversational move at the category level (i.e., APT, Declare, etc.). Summaries of these inter-rater reliability tests are presented in Table 3.4, below.
Table 3.4
Summary of inter-rater reliability statistics (n=7844)
Level of
Test Agreement
Expected
Agreement Kappa Std. Err. Z Prob>Z
Move 85.91% 5.02% 0.85 0.0026 333.96 0
Category 89.56% 15.88% 0.88 0.0047 184.81 0
The percent agreements between the co-rater and myself at the conversational move and conversational move category levels were 85.91% and 89.56%, respectively. The associated Cohen’s kappa statistics (k) were 0.85 and 0.88, respectively, and indicate a “near perfect” level of agreement (>0.81, see Landis & Koch, 1977)13 at both levels of reliability testing. A summary of frequency counts for each rater, for each category, is provided in Table F1 (see Appendix F).
Given that the primary focus of this study is related to APT, additional inter-rater reliability tests were conducted to test the reliability of coding each conversational move vs. not (i.e., 1=move, 0=not the move). As summarized in Table F2 (see Appendix F), kappa statistics for each APT move ranged from 0.62 to 1.00, indicating a “substantial” to “near perfect” level of agreement for all APT moves. After establishing reliability for the application of the coding scheme to the data, I proceeded to code the remaining chat-
13 An additional interpretation of kappa statistics can be taken from Fleiss (1981) who suggests that all
log data.
Using this coded data, I then calculated continuous variables representing the proportional use of each group’s discourse at the design cycle and conversation levels attributable to each conversational move category (see Appendix G for summary statistics)14. Next, to characterize groups’ use of each type of talk in subsequent analysis, I generated categorical variables to rank each group's discourse. Groups were divided into four equally sized groupings (quartiles)15, based on their proportional use of each type of talk (see Table 3.5, below for descriptive statistics). Groups that used a proportion of talk in the upper quartile were categorized as “high” (3); those that used a proportion in the lower quartile were categorized as “low” (0); groups that were within the interquartile range above or equal to the median were categorized as “moderately high” (2); and groups within the interquartile range below the median were categorized as “moderately low” (1).
14 Because the focus of this study is on the relationship between the substantive aspects of collaborative
discourse and the development of an engineering epistemic frame, utterances coded as conversational moves in the “Other” category (Enter, Express, Repair, State) were dropped from the dataset prior to variable generation (18% of utterances). Corollary analysis indicated that there was a near perfect correlation (r=0.99, n=110) between the distribution of each type of talk for groups with and without this category, and their exclusion altered the final distribution of each conversational move in the dataset by, on average, 3.6% (range: 1% - 4%).
15 Use of quartiles for ranking was justified in two ways. First, I note that the mean and median values are
similar, suggesting that the data is, overall, evenly distributed around the means. Secondly, because quartiles are less affected by outliers in the data, they are an effective approach to characterize the overall distribution of the data in my sample for the purposes of this analysis.
Table 3.5
Descriptive statistics of proportional use of conversational move categories (types of talk) at each level of analysis (i.e., segment)
--- Quantiles ---
Variable/Segment Mean S.D. Min q25 Median q75 Max
Academically Productive Talk (APT) APT Combined Conversation 1 0.47 0.12 0 0.39 0.48 0.55 0.68 Conversation 2 0.44 0.1 0.22 0.38 0.44 0.52 0.63 Conversation 3 0.22 0.11 0 0.15 0.22 0.29 0.48 Conversation 4 0.46 0.14 0.11 0.37 0.47 0.55 0.75 Design Cycle 1 0.46 0.08 0.3 0.4 0.46 0.52 0.58 Design Cycle 2 0.31 0.08 0.09 0.24 0.31 0.37 0.49 Entire Sample 0.38 0.11 0.09 0.31 0.38 0.46 0.58 APT-Conversational (APT-C) Conversation 1 0.39 0.11 0 0.33 0.38 0.47 0.6 Conversation 2 0.39 0.1 0.19 0.32 0.38 0.45 0.59 Conversation 3 0.2 0.1 0 0.14 0.19 0.26 0.44 Conversation 4 0.39 0.12 0.11 0.3 0.39 0.46 0.69 Design Cycle 1 0.39 0.07 0.24 0.35 0.39 0.44 0.55 Design Cycle 2 0.26 0.07 0.09 0.21 0.26 0.32 0.44 Entire Sample 0.33 0.1 0.09 0.26 0.33 0.39 0.55 APT-Facilitative (APT-F) Conversation 1 0.08 0.04 0 0.05 0.09 0.11 0.19 Conversation 2 0.05 0.03 0 0.03 0.05 0.08 0.12 Conversation 3 0.02 0.02 0 0 0.02 0.04 0.08 Conversation 4 0.07 0.05 0 0.04 0.06 0.11 0.2 Design Cycle 1 0.07 0.03 0.01 0.05 0.06 0.09 0.12 Design Cycle 2 0.04 0.02 0 0.03 0.04 0.06 0.1 Entire Sample 0.05 0.03 0 0.03 0.05 0.07 0.12
--- Quantiles ---
Variable/Segment Mean S.D. Min q25 Median q75 Max
Other Types of Talk Administrative Conversation 1 0.1 0.06 0 0.06 0.09 0.14 0.29 Conversation 2 0.11 0.08 0.02 0.06 0.09 0.13 0.33 Conversation 3 0.1 0.08 0 0.05 0.07 0.14 0.31 Conversation 4 0.12 0.12 0 0.04 0.08 0.19 0.67 Design Cycle 1 0.11 0.05 0.02 0.07 0.1 0.13 0.28 Design Cycle 2 0.11 0.07 0.01 0.05 0.09 0.14 0.36 Entire Sample 0.11 0.06 0.01 0.06 0.1 0.13 0.36 Declaration Conversation 1 0.22 0.08 0.09 0.15 0.21 0.27 0.5 Conversation 2 0.17 0.06 0.04 0.13 0.17 0.2 0.35 Conversation 3 0.36 0.12 0.1 0.3 0.35 0.42 0.7 Conversation 4 0.24 0.1 0 0.18 0.24 0.31 0.5 Design Cycle 1 0.19 0.05 0.07 0.15 0.19 0.23 0.32 Design Cycle 2 0.32 0.07 0.18 0.29 0.31 0.37 0.51 Entire Sample 0.25 0.09 0.07 0.18 0.24 0.31 0.51 Eliciting Conversation 1 0.1 0.04 0 0.08 0.1 0.12 0.19 Conversation 2 0.13 0.04 0.05 0.1 0.13 0.16 0.23 Conversation 3 0.14 0.06 0 0.11 0.14 0.17 0.31 Conversation 4 0.12 0.06 0 0.08 0.11 0.16 0.27 Design Cycle 1 0.12 0.03 0.05 0.1 0.12 0.13 0.18 Design Cycle 2 0.13 0.04 0.06 0.1 0.13 0.16 0.29 Entire Sample 0.12 0.04 0.05 0.1 0.12 0.14 0.29 Proposal Conversation 1 0.11 0.08 0.04 0.07 0.1 0.14 0.5 Conversation 2 0.15 0.05 0.08 0.12 0.15 0.18 0.31 Conversation 3 0.18 0.13 0.02 0.12 0.14 0.2 0.8 Conversation 4 0.06 0.05 0 0.03 0.05 0.1 0.16 Design Cycle 1 0.13 0.04 0.06 0.1 0.12 0.16 0.24 Design Cycle 2 0.13 0.06 0.04 0.1 0.12 0.15 0.44 Entire Sample 0.13 0.05 0.04 0.1 0.12 0.15 0.44