Both quantitative and qualitative data collection instruments were used. Data collection instruments were different from one another. Some instruments collected quantitative data while as the other collected qualitative data. The instruments were used in different phases. First to be used were quantitative data collection instruments followed by qualitative data collection instruments. After the intervention quantitative
data collection instruments were used. However, each instrument collected data which provided a particular perspective on the research question. Data collection instruments included classroom observation checklist and two different interview schedules which were used to collect qualitative data from teachers. Written, reading and auditory instruments were used to collect quantitative data on learners’ achievement levels in phonics, phonemic awareness, fluency, vocabulary and comprehension. Classroom observation checklist was used to collect qualitative data on observable teachers’ instructional practice. One interview schedule was used to collect qualitative data from the teaching and learning environment. A second interview schedule was used to collect qualitative data on the use of the balanced literacy approach from teachers in the experimental group.
a) Observation instruments
The researcher used a classroom observation checklist as one instrument for collecting data. The classroom observation instrument was structured. It was used to code classroom activities and to provide quantitative measures of classroom activities (Appendix 2). The observation checklist was rated on a 1 to 5 equal appearing interval scale with 0 representing not done, 1 representing very poor and 4 representing
outstanding. This provided a common set of measures for each teacher. The areas of
focus were planning and preparation of lessons, lesson delivery, and instructional approaches used. Ratings of specific characteristics were completed during observation. It was hoped that use of a structured observation instrument would reduce the possible bias of unstructured classroom observation. This instrument was used to collect qualitative data on the teachers’ classroom practice in teaching reading.
In order to increase the validity of observation each teacher was observed on a regular basis, once every month, by two people for thirty-five minutes’ observation periods, during the sixteen weeks of the investigation. It was not possible to have three people involved in observing teachers as originally planned because of pressure of work and time constraint at the work place. The researcher, together with a co- researcher who was conversant with the balanced literacy approach observed teachers teaching. Before each observer observed alone, the two of us had to achieve a
Besides classroom observation there was also an examination of documents such as lesson plans. An examination of the daily lesson plans provided a broader spectrum of teachers’ instructional approach, which included not only performance in the classroom but also what the teachers intended to have happened. The multidimensional assessment of the teachers’ approaches was likely to capture enough information and thus increase levels of reliability and validity. Training in observation skills for the researcher was not necessary. The researcher has been a secondary school English teacher for more than ten years. She has also used observation instruments before for more than five years during teaching practice moderation in Teacher Training Colleges. The other observer works for the Malawi Institute of Education as well. He too has been involved not only in teaching practice moderation but also in various research activities concerning literacy in Malawi. His background is presented in the last section of this chapter.
There were consistent results among researchers who rated the teachers in the classrooms. Observation data had an inter-observer reliability coefficient of .80.
b) Teacher interview schedules
Structured teacher interview schedules were used to collect some qualitative data in order to substantiate the findings of the quantitative data. One interview schedule collected data on the teaching and learning environment such as availability of books, desks and library at school. In addition to the above interview teachers in the experimental schools were interviewed in order to find out
(a) whether or not they understood exactly what the balanced literacy approach meant and what it involved,
(b) their experience in the use of the approach,
(c) what in their opinion were the benefits of using the approach,
(d) besides lack of books, desks any problems that teachers might have experienced as they implemented the approach,
(e) ways teachers thought would resolve the problems,
(f) what improvements did they think were required if the balanced literacy approach were to be repeated,
(g) what they enjoyed doing most as they implemented the balanced literacy approach.
Teachers in the control group were asked similar questions that touched on
a) Their understanding of the instructional approach they used in teaching reading and what it involved,
b) their experience in the use of the approach,
c) what in their opinion were the benefits of using this approach,
d) besides lack of books, desks any problems that teachers might have experienced as they used their instructional approach,
e) ways teachers thought would resolve the problems,
f) what improvements did they think their approach required for it to work well, g) what they enjoyed doing most as they taught reading using their instructional approach.
The interviews were carried out during the last two weeks of the intervention. The interview questions were formulated in such a manner that, teachers’ responses would bring out aspects of using the balanced literacy approach that could not be revealed by classroom observations alone. The interview data and classroom observations were used in order to provide methodological triangulation.
c) Pre-tests and post-tests
The pre-tests and post-tests form part of the process of evaluating the effectiveness of the balanced literacy approach in the teaching of reading and writing in English to standard four pupils in Malawi. The intervention’s aim was to provide an instructional approach, which would raise pupils’ learning achievement in reading and writing. The post-test was administered after the balanced literacy instruction was completed. A comparison of the pre-test and post–test results ruled out memory effect. There was a gap of sixteen weeks between when the pre-test and post-test were administered.
During the quantitative phase of the study pre-test and post-test instruments were used to collect data. Achievement tests were administered at the beginning and conclusion of the research (see Appendices 3 and 4). The objective of the tests was to find out if the intervention would result in the improvement of performance of grade four pupils in literacy in English in the selected experimental group. The pre-test was administered to check whether the entry points of participants were the same. It also
a post-test was administered to both groups of participants to measure any pupils’ gains in learning as a result of the intervention. Test items were adapted from Pupils’ Book for standard four (Chilora et al. 1994) Activities with English: A course for
primary schools. Before writing the test learners were asked to provide the following
information about them; name, sex, age, language spoken at home, mothers highest level of education, mothers’ occupation, fathers’ highest level of education, fathers occupation. Teachers were asked to verify the information pupils gave. Teachers were also asked to provide information on pupils’ attendance during the intervention.
Issues of validity and reliability in test instruments used in a study need to be considered. Validity refers to how well test items measure what they are supposed to measure. Without validating a test or tests one cannot generalize with confidence the end results of testing. Dimsdale and Kutner say that “a successful study should use an assessment instrument that directly measures the skills being taught by the intervention” (2004:11). For the test items to be valid in this study they should measure pupils’ knowledge and skills in phonics, phonemic awareness, fluency, vocabulary and comprehension. Secondly valid test items should have adequate sampling of the content (Cohen et al. 2000). The validity of the test items was determined in terms of what Norcini and Grosso (1998) recommended. Norcini and Grosso say that one way of ensuring relevance of test items is to collect ratings of item relevance from job incumbents. In following this recommendation ten subject matter experts were asked to match the test items against the content measured as well as to determine the item relevance. Each of the ten subject experts were asked to make a judgement regarding the relevance of each of the five content areas (phonics, phonemic awareness, fluency, vocabulary and comprehension) using a five-point scale where a rating of 1 indicated not at all relevant and a rating of 5 indicated
highly relevant. The subject experts were also asked to use the numbers 2 through 4
to indicate intermediate degrees of relevance. (Please see Appendix 6 for more details on how this was done).
The level of agreement among subject experts as regards test items’ relevance was that seven out of ten said that the test items on phonics and phonemic awareness were highly relevant while as the other three said the items were intermediately relevant. The level of agreement among the subject experts as regards relevance of test items
on vocabulary, fluency and comprehension was that all the ten said the test items were highly relevant. Generally the ratings collected from the ten subject experts regarded the test items as highly relevant. The ratings were more than 4 on average on a 5-point scale. The test items especially the written test and reading text were believed to have content validity because the items covered the curriculum that was being taught and pupils were likely to have been familiar with them.
In order to find the reliability coefficient of the test items the KR21 formulae was used. The reason for using this formula was that the formula is used for a single test. The study did not use test retest on pupils to avoid practice effect and familiarity with the content. The reliability coefficient of the written test was found to be .70, while those of the auditory and reading fluency tests were at .75.
i) Test instruments
Three tests instruments were used to collect data from pupils, written tests, reading texts and auditory. The written test instrument collected data on pupils’ abilities in phonemic awareness, vocabulary and reading comprehension. The auditory tests collected data on pupils’ abilities in phonics. The reading texts collected data on pupils’ reading fluency.
ii) Test items
The test items measured literacy achievement through different types of tasks and ways of scoring. The test items were designed to capture a picture of the learner’s achievement in phonemic awareness, vocabulary, fluency and text comprehension. It is claimed measures in phonemic awareness, phonics, vocabulary, fluency and text comprehension are linked to one another both theoretically and psychometrically and have been found to be predictive of reading proficiency (Big Ideas in Beginning Reading no date). The test items focused on those skills that are essential in early reading.
• Phonics and phonemic awareness
The phonemic awareness task tested pupils’ ability to hear and manipulate sounds in words. Phonemic awareness is auditory (Antunez 2002; Big Ideas in Beginning
one. The test administrator read an auditory prompt to a pupil, asking the pupil to respond either by identifying or manipulating a specific phonological unit in a word. Just to give few examples, Question A in the pre-test tested pupil’s ability to identify words that end with the same sound (rhyming words) such as cat and mat. Question B, in the same pre-test tested pupil’s ability to manipulate sounds in words for example substituting the /b/ sound in bat with /r/ to make rat. Question A in the post- test asked the pupil to identify the initial sounds in words for instance to say whether the initial sounds in fat and fun are the same or different. Question B in the post-test was on phoneme deletion. In this question a pupil was asked to say a word without the first sound for example, win without /w/ to make in.
Cassady et al. (2005) argue that this method, that is, having a test administrator read an auditory prompt to a pupil, presents a threat to validity and reliability in assessing phonological skills. This is more so where there are a number of test administrators involved in reading the auditory content to the learner. Cassady et al. (2005) argue that administrator’s dialects, speech rate, enunciation, diction or accents are bound to vary and this can provide a different test stimulus to the learners. In order to limit the threat to validity and reliability posed by the involvement of multiple administrators for questions A and B, this study included writing tasks (Please see Question C). In addition one test administrator was used for the auditory and reading test items.
Question C measured pupil’s ability to make analogies between the spelling patterns in words. For instance pupils could be given the word beak to create new words such as peak and speak. In the classroom pupils can use analogy to read new words such as already cited. Research findings have shown that children can use analogy to decode new words (White 2005). The research findings also suggest that analogy has a role to play in the initial stages of reading acquisition.
• Fluency
The test item on fluency (Task G in Appendices 3 and 4) is a task where an individual learner was asked to read English material from the prescribed standard four textbook (Chilora et al. 1994) Activities with English: A course for primary schools. The task measured pupil’s ability to read grade level text quickly and accurately. Pupils read aloud a passage from their grade level textbook for one minute. A stopwatch was
used to determine the number of words a pupil read in one minute. A score was given representing the number of words the pupil read correctly. (For further details on administration and scoring please see Appendix 7). Correctly read words (CRW) per minute and errors per minute were used to determine fluency. A word was scored as an error each time it was omitted, mispronounced, or substituted. According to the Partnership For Reading (b) this procedure is valid and reliable and strongly correlated with reading comprehension (The Partnership For Reading 12/14/2005). It was easy to monitor pupil’s progress overtime in reading prescribed grade level text book by seeing if there was any increase in words read correctly per minute from the pre-test to the post-test. The length of the pre-test passage Khumbo and the
grasshopper was 184 words, while that of the post-test Timba the beautiful bird was
185 words.
• Vocabulary
Questions D and E were writing tests on vocabulary. In one task pupils were given pictures and a list of names/words relating to the pictures. Pupils were expected to match the picture with the appropriate name/word. In the other task, pupils were given a definition in a sentence. Pupils had to select from a group of words, the correct word to which the definition referred. This later task came from the prescribed text book (Chilora et al. 1994) Activities with English: A course for primary schools.
• Comprehension
The reading comprehension task (Question F) was a writing test used to measure pupil’s level of understanding of a written passage or text. Pupils were asked to read a passage from their grade level prescribed textbook (Chilora et al. 1994) Activities with
English: A course for primary schools and respond to written questions after reading.
The letter that Teleza wrote to Jane was used for the pre-test while as the letter that Juliet wrote to Maria was used in the post-test. Marks were awarded for each comprehension question that was answered correctly. (For scoring of marks please see Appendix 7).
3.5.5 Pilot testing
was felt that the trial study would help to identify any problems that might have been there in the design and to try and address these before carrying out the main study; problems such as those that might have been in the procedure (practicality of procedures), materials (the reliability and validity of the test items) and methods. Pilot testing is “an important means of assessing the feasibility and usefulness of the data collection methods and making any necessary revisions before they are used with the research participants” (Mackey and Gass 2005: 43). Cohen et al., say in cases where observational research is being done, “a pilot must be conducted to ensure that the observational categories are appropriate ...unambiguous and effectively operationalise the purposes of the research” (2000:129). Pilot testing was also used to train the research assistant in as many elements of the research process as possible.
Since it was not practical for the study to randomly assign pupils to the experimental and control groups a quasi-experimental design was used during the quantitative phase of the study. Four schools in Zomba rural were pilot tested for a period of sixteen weeks (two terms with a one month holiday in between). The sixteen weeks excluded four weeks of administering pre-tests and post-tests. Two schools out of four were randomly placed in the experimental group while the other two were placed in the control group. A total of 294 pupils were initially included in the sample. Out of this 124 pupils were in the experimental group while as 170 pupils were in the control group. Table 4 below shows details of pupils who wrote the pre-test and post-test in the pilot study.
Table 4: Number of pupils who wrote the pre-test and post-test in the pilot study. Name of School Pre-test Girls Pre-test Boys Pre-test Total Post-test Girls Post-test Boys Post-test Total Both Pre- & Post- test Mwanje Exp. 24 30 54 32 36 68 38 17G+21B Khuluvi Exp. 28 42 70 17 18 35 24 14G+10B Kanjuli Control 23 32 55 23 35 58 38 17G+21B Namadidi Control 75 40 115 48 33 81 61 41G+20B TOTAL 150 144 294 120 122 242 161
Just as in the main study the number of pupils writing the pre-test and post-test in the pilot study was not constant. It either increased or dropped during the post-test. As was mentioned earlier on the dropping or increasing of pupils’ numbers was a result of pupils’ absenteeism, drop-out or transfer to other schools.
There was no gender balance in the number of girls versus that of boys who wrote the tests. In the pilot experimental schools there were more boys than girls. However, in one of the pilot control schools the number of girls was almost two times that of boys. There were seventy-five girls against forty boys.
The pupils’ age range in the pilot study was almost the same with that of the main study. The pupils’ ages ranged from eight to nineteen years. Pupils spoke Chichewa, Chiyawo, Chilomwe, Chinyanja and Chingoni. About 80 percent spoke Chichewa, followed by 16 percent who spoke Chiyawo. The remaining 4 percent was shared by pupils who spoke Chilomwe, Chinyanja and Chingoni. The majority of the pupils’ fathers and mothers were subsistent farmers. The highest educational level for the majority of the parents was primary school (about 60 percent).
intervention. The training was done mid way of the intervention in order to ensure that teachers understood the balanced approach if they did not during the first training