Análisis de la primera etapa: Exploración de la problemática

The student responses to the linguistic biography questionnaire were aggregated in an excel spreadsheet and a simple count process was used to explore data patterns and associations between students’ parental education; home locality; language of teaching and learning; degree of multilingualism and self-reported degrees of ‘academic literacy’ proficiency with/against students’ actual ‘academic literacy’ performance. (See Appendix Seven and Eight for the Questionnaire and the Coding Sheet). This was used to describe who the students were in more detail and to offer insights into possible external barriers to the

110 efficacy of an intervention like RtL. The results of this analysis will also be used to highlight future research pathways. It must be noted that only students’ academic essay scores were used against the linguistic biography responses as I felt this would give a better representation of ‘academic literacy’ skills than the narrative. The purpose of the questionnaire was merely a ‘bank’ of information about students to turn to if the data patterns emerging from the assessment scores and case studies were proving difficult to interpret. For example, if a student functioned as an outlier, I could turn to the linguistic biography to assess if any external variables were to blame.

The individual student writing portfolio’s chosen for the case study were analysed on two different levels. Firstly, the overall assessment scores used for analytical purposes were disaggregated further into the subcategories of writing being assessed (See table 4.1 for a description of these subcategories). Instead of an overall grade being used as with the quantitative data analysis, in this case, the overall grade was broken down into the subcategories. For example, where a student may have been graded 38/42, all 14 categories (awarded a mark of 3 per category = 42) were taken into account to get a micro perspective of students change in ‘literacy’ skills. This was done to highlight which specific academic skills were enhanced with RtL, and which specific aspects of academic writing may need more attention should RtL be used further. An analysis of emerging patterns in students’ writing portfolio was then conducted with specific reference given to the subcategories mentioned in Table 4.1. This essentially allowed for triangulation. The case study served to investigate a smaller number of students’ writing portfolio’s in more detail with respect to their change in academic writing skills as per the categories used in the marking rubric.

4.9.2.1 Case Study Method

The case study is a common approach to qualitative inquiry but is not necessarily always qualitative in nature. In some cases, it may necessitate the use of mixed methods (Stake, 2003). In this case, a mixed method approach involves the integration of both quantitative and qualitative data methods to validate the findings of one set of data (Bryman, 2006). This is a relevant construct of case studies for this particular study as selected student writing

111 portfolios were analysed with a case study approach but assessment scores (numerical) were given to the writing portfolio’s which added an extra element to the analysis.

The case study approach to inquiry is useful in that it allows an in-depth examination of data through multiple sources of data, lending itself well to a means of triangulation (Tellis, 1997). It is also useful in that it offers a study of real people, in real situations without having to present findings alongside abstract theories and principles (Cohen et al, 2011). Further, Cohen et al (2011) states that case studies are able to break through situations in a manner that may not be possible through numerical analysis. This is an important statement for this study as a quantitative analysis of students’ writing portfolios may not necessarily tell the full story. To get a richer description of the change in students’ writing performance, a case study allowed for a deeper analysis of what was really going on within students’ writing portfolios. Hitchcock and Hughes (1995) argue that case studies are particularly useful when the researcher has little control over extraneous variables (classroom disruption, student truancy, student behavior) and because of this, the use of a case study allows for a rich, in-depth discussion of all factors at play within the study; allows for a blending of a description of events that took place together with an analysis of the data, and the researcher is integrally involved in the analysis procedure.

However, case studies are not free of criticism. One major criticism of the use of case studies is the supposed absence of research rigour. A lack of systematic processes and procedures, or a lack of inclusion of certain parts of the data by the researcher can be a limitation (Yin, 2009). But then again, this could be a limiting factor of all types of research. It is not only case studies where the researcher has to make a judgment call about what data to include and what data to omit. It is hoped that a drive towards responsible, ethical research can remedy this situation as best as possible. Another major criticism of case studies is that it holds a certain type of preoccupation for unique, sometimes individual cases which does not lend itself easily to generalisability (Pring, 2006). In the case of this study, I feel this is not as much of an issue as the study is looking at a very specific micro analysis of RtL implementation at two very specific school sites and given that the sampling process was purposive, generalisability of the findings of this study to the greater South African population was not a consideration, or objective.

112 A possible criticism of this study relates to codifying (or converting) qualitative data (students’ pieces of writing) into quantitative data (numerical ‘academic literacy’ scores). However, the criticism is not unique to this study, but could be levelled at any other study, for that matter, which seeks to ‘measure’ something or other. Trying to measure something inherently ‘qualitative’ in nature (like ‘academic literacy’ skills) is an age-old problem. For example, one might equate the problem with trying to measure one’s intelligence quotient (IQ). As a quantifiable measure of one’s verbal-linguistic and logical-mathematical abilities, for many years we were led to believe the traditional IQ ‘score’ represented the ‘truth’ about one’s intelligence. However, we now know that intelligence is multi-faceted [see Howard Gardner’s (1983) Theory of Multiple Intelligences] and also very often, culture specific. At the heart of any ‘measure’ is how accurately it captures the ‘truth’ – some definable reality (an ontological problem). Despite various ontological arguments and continued controversy surrounding quantifying inherently qualitative phenomenon, one really needs to ask the following question (concerning validity): does the measure (whether qualitative or quantitative) accurately capture what it seeks to measure? In the case of the present study, this is tantamount to asking whether or not the numerical assessment score obtained from grading or marking a student’s piece of writing, accurately measures the intended (theoretical) construct (i.e. ‘academic literacy’ skill or performance).

What determines or comprises a valid measure of ‘academic literacy’ skills in the context of the present study is a contentious issue, and could be debated to no end. However, of greater concern is the error with which such ‘truth’ is measured (measurement or observational error). Loosely defined, measurement error refers to deviations from some or other ‘truth’, irrespective of how the ‘truth’ is defined. Rowntree (2000: 36) provides us with a common- sense explanation of measuring some or other phenomenon, which might be useful to remember in the context of this study. “Whatever the subject-area, it is sensible to remember that the observations or recorded values are really just APPROXIMATIONS to some true values. If we have counted carefully and enjoyed good co-operation from our questionnaire respondents (or if we have measured carefully with a small enough unit of measurement), our data should contain minimal error. The sample-figures will be accurate enough for us to base sensible decisions on them.”

113 In defence of the more quantitative approach adopted by this study, bearing in mind the abovementioned proviso, a number of points must be made. First, the rubric assessment tool used (mutually) exclusive numerical rating scale categories 0, 1, 2 and 3, representing the smallest possible unit of measurement and as few rating scale categories as possible to minimise measurement error. In other words, one is more easily able to assign a student’s work to a particular performance descriptor as opposed to having to make a choice between assigning a student’s performance to one of a greater number of (mutually) exclusive rating scale categories (e.g. 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10); or, having to choose between mutually inclusive rating scale categories (e.g. 0-3, 4-5, 6-7 or 8-10). Second, the rubric used to convert the qualitative data into quantitative data was consistent with other studies of RtL, making comparisons across studies more feasible. Third, this study focused on certain ‘micro’ elements of writing, with fourteen separate sub-categories of assessment criteria being used in the rubric. This entails a more detailed (and hopefully more accurate or valid) description of performance with respect to the specific elements of writing (and associated ‘academic literacy’ skills) being demonstrated/assessed, vis-à-vis the more generic-type rubrics used ordinarily as part of the secondary school English FET curriculum. Fourth, regardless of concerns about whether or not the RtL assessment rubric being used is a valid instrument or tool with which to measure or test ‘academic literacy’ skills (otherwise known as construct validity), it must be noted that the rubric was applied as consistently as possible for all students and all pieces of assessed writing throughout the intervention, thereby hopefully minimising problems related to measurement error.