research conclusion can be drawn when the data are collected using valid and reliable instruments (Creswell, 2012; Fraenkel et al., 2012). In this section, therefore, an attempt that had been made to increase the validity and reliability of the three PADIs is described.
Relying on the content validity, the researcher-developed diagnostic instruments were expected to cover the basic underlying concepts of plant anatomy that had been taught by the instructors (Cohen et al., 2011). As outlined in Section 3.3.2 (Step 5), the three instruments had been content validated by three in-service instructors of plant anatomy and one biology expert from Curtin University. Despite some minor revisions, overall, the panel of the four experts indicated that the three PADIs were valid in terms of the content coverage.
89
To ensure that the three instruments were reliable, two methods of reliability―internal consistency and interrater reliability were employed. These two different approaches were applied because of the distinctive formats of the tests in the diagnostic instruments. The first parts of the PADI-I and III were designed using multiple-choice questions, whereas the close-answered and drawing tests were apparent in the first part of the PADI-II and the second part of the three PADIs, respectively. The Cronbach’s alpha coefficient (Cronbach, 1951) was calculated to examine the internal consistency of students’ responses to the objective test―the multiple-choice questions. Table 4.7 shows the results of this calculation. In contrast, the interrater reliability procedure (see Section 3.3.5.1) was applied to evaluate students’ subjective answers to the close-answered questions and drawing tests. The percentages of agreement for these two formats of the tests are presented in Table 4.7.
Table 4.7 Reliability of the Three Diagnostic Instruments
Instrument Format Reliability
Determinant Value
PADI-I Multiple-choice questions Alpha coefficient 0.52
Stem Drawing Percentage of agreement 100%
Stem Labelling Percentage of agreement 100% Root Drawing Percentage of agreement 87.5% Root Labelling Percentage of agreement 100%
Leaf Drawing Percentage of agreement 100%
Leaf Labelling Percentage of agreement 100% PADI-II Close-answered questions Percentage of agreement 89.5%
Drawing Percentage of agreement 87.5%
Labelling Percentage of agreement 87.5%
PADI-III Multiple-choice questions Alpha coefficient 0.59
Drawing Percentage of agreement 100%
Labelling Percentage of agreement 100%
The data indicate that the three PADIs were reliable as research instruments. As shown in Table 4.7, the percentages of agreement for both close-answered and drawing tests ranged from 87.5-100%. This means that the disagreement between the two reviewers who were involved in the assessment occurred no more than 12.5% of the students’ responses to the tests. These values well exceed the minimum acceptable percentage of interrater reliability, that is, 80% (McHugh, 2012). In
90
contrast, the alpha coefficients for the multiple choice questions are relatively low, that is about 0.5 (see Table 4.7). This value implies the weak correlation among items within a test (Nunnally & Bernstein, 1994). Two possible factors were identified as sources of the tests’ heterogeneity including the diverse content of the test and the small number of items (Nunnally & Bernstein, 1994). As apparent in Appendix A, each of the PADIs covers five different topics of plant anatomy, namely, plant cells, tissues, roots, stems, and leaves instead of one particular concept or topic. These heterogeneous items influence the alpha coefficient which reflects the degree of item correlation. Moreover, the number of questions in Section 1 of both the PADI-I and III are fewer than 30 items. This small number of items will decrease the test reliability because the alpha coefficient is a function of the number of items (Nunnally & Bernstein, 1994; Streiner, 2003). Despite these facts, the alpha coefficient at the point of 0.5 is an absolute threshold (Nunnally & Bernstein, 1994) and classified as moderate (Salvucci, Walter, Conley, Fink, & Saba, 1997). Furthermore, Roszkowski and Spreat (2011) contend that for research such as the current investigation which aims to characterise the level of the participants’ conceptual understanding of plant anatomy, the lower reliability values are acceptable.
Based on the above information, it is suggested that the three PADIs were sufficiently valid and reliable as research instruments. This fact implies that any conclusion that is drawn from the data collected using these instruments is warranted, thereby, ensuring the trustworthiness of this investigation.
4.8 Summary of the Chapter
In this chapter, the participants’ performance on the three diagnostic instruments has been presented and analysed in response to the first research question. Supporting one another, as briefly schematised in Figure 4.13, the results from the three PADIs suggested that the majority of the students who enrolled in the three different semesters did not achieve a satisfactory conceptual understanding of plant anatomy. This insufficient comprehension of the basic concepts had led to difficulties in knowledge application leading to the construction of students’ misconceptions. Five mechanisms of the undergraduates’ incorrect conceptions that were identified in this study include surface-level reasoning, inappropriate transfer, narrow understanding,
91
incorrect application of concepts and superimposing concepts. The possible sources of this learning problem were the students’ insufficient understanding of the basic components of plant structures, such as tissues, and limited learning scaffolding. These inferences can be justified because the three PADIs were shown to be sufficiently valid and reliable as research instruments.
Figure 4.12 Students’ performances on the three diagnostic instruments. Students’ ability to interpret visual representations is prominently affected by their existing knowledge (Schnotz & Bannert, 2003; Schönborn & Anderson, 2009). Thus, based on the fact of students’ insufficient conceptual understanding and misconception as described in this chapter, it can be predicted that students would have difficulties when interpreting pictures in the tests. To clarify this assumption, several student volunteers were interviewed. Their responses to the interviews will give insights of the undergraduates’ actual ability to interpret visual representations of plant anatomy being presented. This information is presented in Chapter 5.
92