research (e.g., see review in Fuchs, Fuchs, & Maxwell, 1988) as well as in research on relevance instructions (e.g., see review by McCrudden et al., 2007). For this study, students were asked: “Please write what you learned from this article. Include as much detail that you remember from the text as possible.” Although recalls are commonly used to measure students’ memory for the text (i.e., factual learning), its open-ended format allows the structure of the students’ conceptual
43
representation to be captured, and thus provides a means for assessing conceptual learning as well.
Factual learning. I analyzed the experimental text, Sense of Sweet, using a set of guidelines that were based on principles of expository text analysis by Meyer (1985) and Mayer (1985). I identified a total of 76 idea units, which were included in a rubric that was used to score students’ free recalls. An idea unit typically consisted of a predicate clause (a verb and the surrounding adjunct words), but could also be a descriptive phrase, or a specific term. (See Appendices E and F for the Free Recall Rubric and a detailed description of the guidelines used to develop it.) When scoring recalls, I examined student protocols for idea units that matched— or which provided the substantive gist of—the idea units in the rubric. The total number of idea units in student recalls served as one measure of factual learning.
The learning of main ideas and less-essential details. Since the experimental pre- reading instructions were designed to orient students’ attention to important ideas in the text, I also counted the number of important ideas found in student recalls using a separate Main Ideas Rubric (see Appendix G). I created the Main Ideas Rubric using Meyer’s system of text analysis (Meyer, 1985) and Chambliss’ framework for explanations (Chambliss, 2002). Using their principles, I arranged the idea units of the text into a hierarchy based on their structural importance (i.e., how necessary they were to understanding the main ideas of the text). There were four levels: main ideas, important details, less important details, and least important details. When scoring for the presence of main ideas, I counted any idea unit from level 1 or 2 as an “Important Idea.” I counted any idea unit from level 3 or 4 as a “Less-essential Detail”. (See Appendices G and H for the Main Ideas Rubric and a full description of how it was constructed.)
Using this method, 32 idea units were categorized as “Important Ideas” and 44 as “Less- essential Details.” One more category of “Important Ideas Related to Sweet Receptors” was created during the analysis of results. This category consisted of 19 ideas units, drawn from Paragraphs 1, 2, 4, and 5. The reason I omitted the main ideas from Paragraph 3 was that those related more to the general concept of tastes, rather than sweet receptors. I hypothesized that the relevance instructions would reveal more of an effect on concepts that directly related to sweet receptors and their mechanism, and wanted to be able to detect this effect, if it were present.
As a check of validity, the Main Ideas Rubric was reviewed by two reviewers. One was a graduate student at Teachers College working in the field of literacy and text structure, and the other a high school science teacher with ten years of teaching experience. The graduate student was asked to review the propositional analysis of the text structure and the resulting hierarchy of ideas. The high school teacher was asked to review the rubric through a science disciplinary lens—to review whether the hierarchy of idea units reflected the instructional importance of those ideas relative to the topic of tasting.
Conceptual learning and free recall rubric. The free recall can also reflect the structure and coherence of a student’s knowledge, and thus serve as a measure of conceptual learning. Thus, I developed a holistic rubric that could be used to evaluate the quality of the student’s overall mental model of the content as reflected in the recall. The rubric evaluated student protocols for the following characteristics:
1. inclusion of structurally important (or main) ideas;
2. explanatory coherence of the answer—reflected by (a) the inclusion of the important steps in the mechanism of tasting sweetness and (b) connections among these steps
45
reflected by the organization of the ideas (e.g., steps in a sequence) or by explicit causal terms (e.g., because of, leads to);
3. integration and elaborateness—the amount of detail that is provided to support the main ideas; and
4. accuracy of ideas.
I created the categories of the rubric based on a taxonomy developed by Tabaoda (2003) and Guthrie and Scafiddi (2004) that categorized the quality of student responses based on the organization of information. In their taxonomy, the lowest levels of knowledge are characterized by isolated facts or lists of facts. In contrast, higher levels of understanding are reflected by increasing connections between facts to form concepts or interrelationships between concepts. The highest levels of knowledge reflect “coherently organized relationships among concepts that are supported by factual details” or a “well-supported explanation of the essential relationships in the topic” (Tabaoda, 2003, p. 161).
I modified the categories of knowledge in the rubric developed by Tabaoda (2003) to reflect the goal of the reading task in this study—for students to form an accurate and coherent explanation—and corresponding mental model—of tasting sweetness. I assigned the categories to a five-point scale. From the lowest to highest level, they are: (0) Incorrect facts; (1)
Unassociated facts or concepts; (2) Rudimentary (or basic) explanation; (3) Partial explanation; and (4) Complete explanation. The degree of elaboration was indicated by the addition of 0.5 points. For example, a simple partial explanation scored 3.0 points, whereas an elaborated partial explanation scored 3.5 points.
The general characteristics of each level in the holistic recall rubric are provided in Table 3.3 on the following page.
Table 3.3
Levels in the Holistic Rubric and Their General Characteristicsa
Level General characteristics of the explanation—degree of coherence
(includes completeness and causality), accuracy, and elaborateness
4.5 Complete explanation, elaborated
Presents a rich, coherent, and comprehensive explanation that is accurate. Demonstrates understanding of the causal mechanism underlying the phenomenon and includes elaborated information about the supporting concepts.
4
Complete explanation, simple
Demonstrates a complete understanding of the process of tasting sweet. Presents information in a coherent manner that demonstrates
understanding of the causal mechanism, including key supporting details (i.e., the recognition based on shape).
3.5 Partial explanation, elaborated
Presents components of an explanation that are highly elaborated, yet is missing a critical supporting detail in describing the causal mechanism (i.e., the recognition based on shape) or does not clearly demonstrate knowledge of causal mechanism.
3 Partial explanation, simple
Demonstrates a partial understanding of the process involved in sensing sweet. Presents components of an explanation, including the
involvement of the receptors, but does not clearly describe the mechanism involved (i.e., the recognition based on shape). 2.5
Basic explanation, elaborated
Demonstrates rudimentary understanding of the process involved in sensing sweet, but includes more elaborated information. Presents simple components of an explanation, yet is missing critical supporting details, such as the receptor’s role.
2 Basic explanation, simple
Demonstrates rudimentary understanding of the process involved in sensing sweet. Presents simple components of an explanation, yet is missing critical supporting details, such as the receptor’s role. 1.5
Unassociated concepts
Presents a concept or multiple concepts that are elaborated, but the concepts are not directly related to each other. Does not organize ideas in a way that suggests an explanation.
1
Unassociated facts
Lists facts or pieces of information, but does not show underlying organization or relationship between the facts. Demonstrates no understanding of causal process or mechanism.
0
Inaccurate facts
Presents inaccurate facts or concepts based on a misreading of the text or prior misconceptions.
aThe General Characteristics are paraphrased from the rubric in Appendix I.
The key distinction between Level 1 (Unassociated facts or concepts), and the remaining Levels 2 through 4 is that answers in the higher levels include signs of an explanation, i.e, the
47
presence of connective or causal conjunctions (e.g., because of, due to, then) or a description that implied causality (e.g, a description of a sequence or process). The complete rubric can be found in Appendix I.