According to Merriam (2002:13), observation is the best technique to use when one wants to observe an activity, event, or situation firsthand or when a fresh perspective is desired. However, observation is not a matter simply of opening one’s eyes and ears to people in real- life situations but rather to train one’s eyes and ears. The researcher therefore needed an attitude of curiosity and a heightened attention as well as observational skills in order to attend to those very details that most filter out automatically in day-to-day life (Ely, 1991:42).
During the course of the two years the researcher took copious field notes (written accounts of the things she heard, saw, experienced and thought) in the course of collecting or reflecting on the data observed during the study (Greeff, 2002:317-318). Keeping extensive field notes were essential in order to capture the context of observations (Mouton, 2005:108). Since the researcher was concerned to get as rich descriptions as possible, she always sat down immediately after an observation session and jotted down her impressions. The field notes included both empirical observations as well as interpretations. Empirical observations included what participants were saying and doing in relation to working on actual PBL tasks and the skills and attitudes they displayed. The researcher further wrote down how the class atmosphere was experienced, how the problems were approached and completed and described the interactions between the learners and learners and tutors. Interpretations also included the emotions, preconceptions, expectations and prejudices of the researcher (Merriam, 2002:13; Ely, 1991:54). The researcher’s reflective journal was therefore an immense help where plans, questions, enthusiasms, doubts and ruminations were jotted down. This was an attempt to separate what was observed from thoughts. Every effort was made to keep observations and interpretations separate.
4.4.3 Data analysis
Data analysis is the process of bringing order, structure and meaning to the mass of collected data. It does not proceed in a linear fashion and is not tidy (De Vos, 2002:339-340; Merriam, 1988:127). At whatever point the evaluation data were collected, it was important to have set procedures for analyzing the data because according to Mertens (1998:239) one of the most frequent flaws in the evaluation process is the inadequate planning of data analysis procedures. This is specifically significant when different kinds of data and multiple data sources are used (Capeling-Alakija et al., 1997, Cresswell, 2003:14). In choosing data analysis procedures the researcher considered the evaluation questions, approaches, data collection techniques, and kinds of data collected (Caffarella, 1994:138).
Data collection and data analysis occurred simultaneously as an interactive, continuous and on-going process (Cocklin, 1996:94; De Vos, 2002:341) and was only separated for convenience. The data analysis within this mixed-method research occurred both within the quantitative approach and the qualitative approach, and also between the two approaches (Creswell, 2003:220). Although the quantitative and qualitative data were sometimes presented in separate sections, the analysis and interpretation combined the two forms of data to seek convergence among the results, which coincided with the very nature of a concurrent study (ibid.:222). The quantitative and qualitative data generated from the evaluation research required vastly different competencies of the researcher in the analysis process (Caffarella, 1994:136).
Analysis of the quantitative data
It is important to identify the variables that the ILS questionnaire measured before explaining how the quantitative data were analysed. Variables are an event or condition that the researcher observes and measures or plans to investigate which is liable to variation or change. The dependent variable (Y) was the ‘effect’ (or outcome) in which the researcher is interested, namely whether the learning patterns, (in other words the self-regulation patterns) of students have changed. Students’ scores on each of the four subscales constituted the dependent variables. The independent variable (X) was the presumed ‘cause’ which led to changes in the dependent variable, namely exposing students to the Hybrid PBL approach (Rosnow & Rosenthal, 1996:44). The ILS questionnaire was therefore used to measure the relationship between the dependent variable (Y), namely student learning patterns (in other
words self-regulated learning) and the independent variable (X), namely introducing the students to the Hybrid PBL approach. At the same time the student learning patterns were explored using qualitative semi-structured focus group interviews and classroom observations with the participants at the research site (Creswell, 1998:101).
The pre-post evaluation design involved observations and measurements before commencement or implementation of the PBL intervention (O1) followed by the PBL intervention (X). After completion of the PBL programme, another set of post-evaluation observations and measurements was administered (O2). Standard quantitative analyses, such as a comparison of mean scores between O1 and O2 and a t-test (a test of significance used to judge the tenability of the null hypothesis of no relation between the two variables X and Y) and analysis of variance (or ANOVA – the subdivision of the total variance of a set of scores into its components) then indicated whether there was a statistically significant difference between the pre-evaluation and post-evaluation participants. A statistically significant difference means that any differences that are observed between a mean pre-evaluation score and a mean post-evaluation score is not due to chance factors but (most likely) indicate ‘true difference’ (Babbie & Mouton, 2001:349; Rosnow & Rosenthal, 1996:416). A p-value (probability value or level obtained in a test of significance) also called alpha and significance value was therefore used (Rosnow & Rosenthal, 1996:412). The key question, namely did the learning patterns of the programme participants change, was answered through this pre- evaluation-post-evaluation design (Babbie & Mouton, 2001:349). Follow-up analyses were used as needed to evaluate the mean differences between each dependent variable (Rosnow & Rosenthal, 1996:4).
Analysis of the qualitative data
The qualitative data consisted of observations made by the tutors, the researcher and students themselves on the extent of change in students’ learning patterns and self-regulated skill levels, values and beliefs. These observations combined with data obtained from the focus group interviews focusing on the participants’ experiences of the whole PBL process resulted in over 150 pages of transcripts and comments (Caffarella, 1994:136). The researcher could identify with Cocklin (1996:93) when confronted with the sheer depth and breadth of qualitative and quantitative material gained through the extensive period of the research. It was in this period that the researcher experienced considerable anxiety wondering if there was enough of the right data and how to bring together the qualitative and quantitative results. To
analyse the qualitative data and develop themes the researcher used the four components and patterns of student learning identified on the ILS questionnaire (Vermunt & Vermetten, 2004:359) since it provided a useful tool in both the insight being sought and in managing and organizing the extent of data being gathered (Ely, 1991:88). The components on the ILS questionnaire therefore guided data-construction and data-analysis, and provided an explanatory framework for the interpretation of findings (Babbie & Mouton, 2001:366).
To assist in the analysis process the researcher applied a combination of the processes proposed by Mertens (2005:423), Miles and Huberman (1994:9), Ely (1991:87), De Vos (2002:340) and Cocklin (1996:88-115) as guidelines since all qualitative data analysis is in a sense idiosyncratic. After data were collected and recorded in a systematic manner the data was organized into computer files. The researcher then read the transcripts in their entity several times in order to get a sense of the whole before breaking it into parts (De Vos, 2002:343). The parts consisted of data being divided into semantic units, such as paragraphs, lines and phrases (Berg, 1995:178). Each unit was subjected to a pre-determined set of questions. Did the data comply with the descriptive components on the ILS questionnaire? Did it comply with the original objective of the research study? This initial criterion provided useful categories to sift the data (ibid.:181). The researcher therefore studied and re-studied the raw data to develop detailed intimate knowledge of it (Ely, 1991:150; Mertens, 2005:423). Because data is just raw data and not information, raw data had to be coded, weighed, collated, processed, analyzed and synthesized to produce information that could be used to make decisions (Anderson, 1992:233). This meant that the data collected was not immediately accessible for analysis, but required some processing (Miles & Huberman, 1994:9). In the concurrent strategy the researcher qualified quantitative data. Here the researcher created factors and themes from the quantitative data that could then be compared with themes from the qualitative data. The researcher also quantified qualitative data. This involved creating codes and themes qualitatively, then consulting the number of times they occurred in the text data (Creswell, 2003:221).
The qualitative data from the various sources were then analysed using the thematic and content analysis procedure of open coding as described in Berg (1995:185) and Frank and Barzilai (2004:46). The researcher took the text apart and data was grouped under the components and patterns (themes) of student learning identified in the ILS questionnaire (Ely, 1991:150). These patterns were given names (codes) and were refined and adjusted as the
analysis proceeded (Mertens, 2005:423; Merriam, 2002:14; Miles & Huberman, 1994:9). The coding process was frequently interrupted by the researcher writing a theoretical note. The researcher therefore engaged in a continuous process of reflection and analysis which included writing initial impressions and reflections in the margins of the pages (Ely, 1991:87; Miles & Huberman, 1994:9).