Writers on mixed-methods advocate the use of validity procedures for both the quantitative and qualitative phases of the study (Creswell, 2003:221). To verify the quantitative phase of the study the researcher described the validity and reliability of the scores for the four components of student learning from past uses of the ILS questionnaire as well as the internal consistency scores for the four learning patterns attained from this study. The final version of the ILS questionnaire was constructed, using factor, reliability, item, and test-retest analyses (Vermunt & Vermetten, 2004:364). Cronbach’s alpha, also called the alpha coefficient was used to measure internal consistency reliability meaning the degree of relatedness of the individual items (Rosnow & Rosenthal, 1996:404). The items of the test are scored dichotomously, that is, scored ‘1’ if marked correctly and ‘0’ if marked otherwise. The more highly correlated the scores, and the more items there are the higher is the reliability (ibid.:126). In several studies with a total of 795 regular university students, the internal consistencies of these scales (Cronbach’s alpha) varied between .63 and .85 for processing strategies, from .48 to .79 for regulation strategies, between .70 and .89 for conceptions of learning, and from .57 to .84 for learning orientation, for regular university students (Vermunt & Vermetten, 2004:364,367). In this particular study the internal consistencies of the four learning patterns (Cronbach’s alpha) for the pre-test and post-test results varied between .83 and .86 for the reproductive learning styles, from .89 to .88 for meaning directed learning styles, between .77 to .87 for the undirected learning styles and from .41 to .70 for the application directed learning styles for the SciMathUS students.
4.5 CONCLUSION
This chapter provided the theoretical foundation for the methodology of the evaluation study. In this chapter the researcher presented an evaluation diagram that served as a framework for this study. The three primary means of gathering data namely semi-structured focus group interviews, observations and the application of the ILS questionnaire was discussed. Quantitative data analysis consisted of a comparison of mean scores between O1 (prior to exposure to the PBL approach) and O2 (after exposure to the PBL approach), a t-test and analysis of variance whereas the qualitative data were analyzed for patterns and general themes.
This chapter further indicated how the research design was structured and provided important guidelines to verify reliability and maximize the credibility and the trustworthiness of the research findings. Although these guidelines helped the researcher gain the confidence to initiate the study, the researcher soon realized that there was no substitute for experience. Many mistakes were thus made; some things happened that were not anticipated, other things worked out serendipitously in spite of the researcher. The research process involved long periods of uncertainty. It was a long, hard journey with many challenges along the way (Correll, 2002:261; Merriam, 2002:421-422).
In the light of the concerns voiced about the Hybrid PBL approach, the question arose whether introducing a Hybrid PBL approach within a shorter one-year foundation programme could create and support conditions for learners to develop self-directed learning skills and whether such an innovation is worth the effort. Chapter 5 will attempt to address these questions.
CHAPTER 5: THE EVALUATION OF THE HYBRID PBL APPROACH
FOR SCIMATHUS
5.1 INTRODUCTION
In order to determine whether introducing a Hybrid PBL approach into a conventional curriculum does promote more favourable learning patterns in students in a particular context and whether these patterns seem sustainable, the effects of this intervention on the learning processes of students in the SciMathUS programme were carefully studied (referred to as learner-oriented evaluation). For these purposes both qualitative and quantitative data were generated (as discussed in Chapter 4). Multiple data construction strategies were used.
Quantitative data generated from 35 students prior to and after exposure to PBL consisted of:
• The completion of the Inventory of Learning Styles (ILS) for pre-PBL evaluation purposes before being introduced to PBL during the orientation week (25 January 2007).
• The completion of the Inventory of Learning styles (ILS) for post-PBL evaluation purposes after experiencing the PBL approach via three integrated problems during the third term (19 July 2007).
Qualitative data was generated from students and tutors. Qualitative data generated from students prior to and after exposure to PBL consisted of:
• The completion of a pre-PBL evaluation questionnaire designed by the researcher before being introduced to PBL during the orientation week (25 January 2007).
• The completion of a post-PBL evaluation questionnaire designed by the researcher after experiencing the PBL approach during the third term (19 July 2007).
Qualitative data generated from students during PBL exposure consisted of:
• Data generated from semi-structured focus group meetings with eight student groups and classroom observations during and after working on problem 1 (The Palmiet Problem) and smaller PBL problems during class (1 February 2007).
• Data generated from semi-structured focus group meetings with eight student groups and classroom observations during and after working on problem 2 (The Amazing Race problem) and smaller PBL problems during class (29 March and 25 April 2007). • Data generated from semi-structured focus group meetings with eight student groups
and classroom observations during and after working on problem 3 (The Two Oceans Marathon Problem) (17 – 19 July 2007).
Qualitative data generated from staff during the implementation of PBL comprised of:
• Feedback provided to students after completion of problem 1 and problem 2 (1 February and 25 April 2007).
• Three semi-structured focus group meetings with staff directly after completion of problem 1, problem 2 and problem 3 (29 March, 25 April and 19 July 2007) to explore student learning patterns and the teaching strategies employed.
• A staff meeting on 25 April 2007.
• Informal comments generated from staff members during and after completion of problem 1, problem 2 and problem 3.
Prior to and after PBL exposure the students’ comments and experiences generated from the qualitative data were compared with the quantitative results generated from the completion of the ILS Inventory for pre- and post-evaluation purposes to ascertain whether exposure to the Hybrid PBL approach had changed students’ learning patterns in a favourable direction. Student leaning patterns were further explored through classroom observations and semi- structured focus group sessions with students and tutors at the research site during exposure to PBL (whilst working on problem 1, problem 2 and problem 3). The qualitative and quantitative data were analysed concurrently. This involved creating codes and themes qualitatively (guided by the learning components on the ILS Inventory), then consulting the number of times they were indicated by students and groups in the text data. This was then compared with the quantitative data. Although 47 students participated in the study only 35 students completed the pre-evaluation ILS Inventory. The qualitative results of the same 35 students were considered in analysing and interpreting qualitative and quantitative data. The findings and interpretations will now be discussed in more depth.