Talk-aloud protocols were recorded using the built-in webcam on an iMac and EvoCam [computer software]. Audio was recorded through an externally connected Blue Snowflake™ microphone. These videos, which continuously recorded both the learning and test phase, were transcribed by the author and several research assistants. The protocols were broken down into utterances, defined as one classifiable thought, usually at the level of a sentence. An utterance lasted anywhere between one and ten seconds when participants were working on problems. When participants were reading text or making long calculations, longer streams of text were coded into one utterance. Periods of silence longer than a few seconds typically indicated the beginning of a new utterance. The length of these utterances was recorded, as well as whether or not the student was referring to a prior example or the current problem being worked on.
To construct the rubric for coding utterances, a specific set of cognitive strategies and processes were identified, based on analysis of materials used in the study and past research that coded talk-aloud protocols of learning (Chi et al., 1989; Gadgil, Nokes-Malach & Chi, 2012; Renkl, 1997). Self-explanation was included, defined as an inference beyond the provided text or problem to provide underlying justifications or reasons (see Table 4). Other beneficial cognitive processes that were expected to be used were analogical comparison between problems, qualitative reasoning about aspects of the problem and the procedures used to solve it, as well as error correction (both at a conceptual level, and that the level of correcting errors in calculations). Additionally, as prior research had observed a stark difference in the amount of comprehension monitoring between those who learned well and those who did not, these were coded as well. These coding categories formed the basis for the original rubric that was used.
A process of iterated revision was used to refine this rubric to more closely match the data. Specifically, the original rubric was used by the author to classify a subset of data. During that coding, a number of other types of elaborative statements were noted, and the rubric was subsequently revised to more accurately reflect all of the types of utterances made by participants. In particular, additional categories were added (e.g., within-problem comparisons, descriptions of problem-solving actions, etc.) to more accurately capture the range of constructive statements participants made. This revised rubric was used by the author and a research assistant on a subset of the collected protocols. Disagreements between the two coders were discussed. Many instances could be resolved, with one coder agreeing that a different category was the more appropriate one. In cases where the discussion revealed that there were differences in how the coding categories were interpreted, or where they were unclear, or where they overlapped too closely with another category, the definition was revised to address these issues. The final coding rubric (see Table 4) was then used by the author to code each of the utterances recorded in the protocols.
Table 4
Rubric Used to Code Elaborative Statements
Code Type Definition Example statement
Explanation Inferring beyond the provided text or problem to provide underlying justifications or reasons.
“The sample size is the same so you don’t have to account for that”
Between Problem Comparison
Drawing a comparison to another problem
“So what we can do was, uhm, the same thing for the previous pages.”
Comparing Test Material to
Learning
Drawing a comparison on the transfer problem to the worked example, or earlier problems
“So we’ll just um, do that procedure we were doing before. Like the standardized number.”
Qualitative Reasoning
An estimation (about the outcome of a problem, or a particular step) made without doing the work out.
“The fireball… looks the most, looks the most accurate.” Evaluation
A question or some statement that indicates the intent to evaluate whether one is on the right track, or to check their work.
“Let’s just go back and check.”
Conceptual Error Correction
A statement that reflects an understanding that something is incorrect with their strategy or reflect noticing a misconception about the problem.
“Oh yeah then divide by 5 because there’s 5 values.” Calculation
Error Correction
Noticing of a small error that is not explicitly conceptual. Small calculator errors would fall into this category
“.364, 1.346, no. 1.364, minus 2…”
Question
An open-ended question to oneself or a statement that some particular problem step or feature is unclear.
“But how do they get the deviation?
Within-Problem
Comparison Contrasting and/or comparing within a problem.
“Yeah, Susan got a higher test score with the test with a lower average.”
Describing a Feature
Describing some feature of the problem, that has been noticed or observed.
“So Susan scored above the dev… one deviation above the mean.”
Describing Action
The description of a problem-solving step being taken or about to be undertaken.
“Starting at 258, 296 is about 40, we’ll go in increments of 10.”
Negative Monitoring
A statement which reflects a lack of understanding, in general.
“I’m not sure some of my thinking was that I don’t know how well I did on this.”
Positive Monitoring
A statement which reflects acknowledgement of understanding, in general.
“Well, now I think I’m fine.”