4.2.1. Results
The mean score in the reading test was M = 4.75 (total scores ranged from 0 to 7), SD = 1.84, and there was no difference in reading outcomes in relation to gender of the learners (p = .565) (Savić, 2014). For the six schools involved in the research study, the reading task results ranged from M = 4.40 (for School 2) to M = 5.54 (for School 6), with standard deviations between SD = 1.67 (for School 6) and SD = 1.97 (for School 2). Table 4.9 summarises the results for all six schools.
Table 4.9: Reading test results per school (means and standard deviations)
School Participants N
Mean
(min. 0, max. 7) Std. Deviation
S1 80 4.94 1.80 S2 85 4.40 1.97 S3 75 4.43 1.73 S4 71 4.51 1.92 S5 101 4.57 1.73 S6 90 5.54 1.67 Total 502 4.75 1.84
The Serbian young learners’ achievement rates are found to be comparable to the achievement of young learners in the ELLiE study, in which “the total mean result is 4.1 and mean values in different contexts range from 2.8 to 5.5” (Munoz & Lindgren, 2011, p. 113). In the 7 individual items of the reading test, the Serbian learners’ scores ranged from 35 per cent correct answers (for item 3) to 86 per cent (for item 2) (Savić, 2014), as compared to the
174 ELLiE study reading achievement ranging from 32 per cent (for item 3) to 84 per cent (for item 2) (Szpotowicz & Lindgren, 2011). In both studies the young learners found item 2 the easiest (the Serbian and ELLiE study scores being 86 and 84 per cent, respectively), and item 3 the most difficult of the seven items. As shown in Figure 4.1, the scores in the Serbian study are slightly higher for all seven items, but the discrepancies between the Serbian and the ELLiE study scores differ across the task. In percentages, the discrepancies are 8, 2, 3, 10, 15, 6, and 22 per cent for items from 1 to 7, respectively. The biggest discrepancy is observed in item 7, in which the Serbian young learners scored 82 per cent, while young learners in the ELLiE study scored 60 per cent.
60 52 52 55 32 84 75 82 58 67 65 35 86 83 0 10 20 30 40 50 60 70 80 90 100
Answer: AHA, caught you! Answer: Come on let’s go and watch TV Answer: Why do you need it? Answer: Wait let me think. Answer: Oh it’s you. Something very strange just
happened. I think we’ve got ghosts! Answer: MMMMM! This is going to be good! Hey
where is the orange juice? Where is the bread? Answer: There is some chocolate spread, Yum!
Serbian results ELLiE results
175 To determine the statistical significance of the above differences the independent- samples t- test was applied, with the null hypothesis stating that there would be no difference between the two independent samples. In this kind of t-test, “scores from two different groups on the same measure are compared” (Larson-Hall, 2010, p. 267), using the mean and standard deviation for each of the two groups. However, since the only descriptive statistics available for the ELLiE sample were the mean score and the scores in each of the individual items of the reading test (continuous dependant variables), these scores were used to calculate t-values in the t-test7. Statistical significance of t-values obtained was determined on the basis of degrees of freedom (df) and a t-test significance chart. The degrees of freedom “allow us to determine whether the t-value obtained reaches a level of statistical significance” (Levon, 2010, p. 86) when compared to the t-test significance chart. The obtained t-test values, degrees of freedom (df = 502 + 1400 – 2 = 1900), t-test significance chart with p-values, and status of the null hypothesis for each of the seven items, are shown in Table 4.10.
The data in Table 4.10 shows that the null hypothesis is rejected for items 1, 4, 5, 6 and 7, which means that there is statistical significance of differences in reading scores for these items, i.e. some factor(s) behind the differences can be assumed. On the other hand, for items 2 and 3, the null hypothesis is true, indicating that there is no statistical significance of differences in scores. The t-test significance table for 1900 degrees of freedom gives the values for 95 per cent significance (p<0.05) and for 99 per cent significance (p<0.01). Table 4.10 shows that the results are statistically significant and that the analysis is 99 per cent sure for items 1, 4, 5, and 7, as the calculated t-values are greater than the required 2.58, i.e. they are 3.92, 3.98, 6.03, and 10.19, respectively; in case of item 6 the statistical significance is at the p<0.05 level, as the t-value for this item is 2.33, which is greater than the required 1.96
7 For t-test of independent samples with equal sample size, the degrees of freedom is calculated by taking the
total number of subjects in both groups and subtracting 2, while for samples larger than 200 subjects, it is insignificant if the samples are of equal size.
176 for statistical significance of 95 per cent, meaning that we can be at least 95 per cent sure that the null hypothesis is false.
Table 4.10: Output from the Independent-Samples T-Test (comparison of achievement in seven items in percentages) for Serbian and ELLiE studies
Items Scores from independent samples (%) Samples (N) t-test values Degrees of freedom (df) Significance value
intervals (p)* Null hypothesis (H0) 0.05 0.01 1 Serbia 83 502 3.92 1900 1.96 2.58 H0 false (p<0.01) ELLi E 75 1400 2 Serbia 86 502 1.09 1900 1.96 2.58 H0 true (p>0.05) ELLi E 84 1400 3 Serbia 35 502 1.22 1900 1.96 2.58 H0 true (p>0.05) ELLi E 32 1400 4 Serbia 65 502 3.98 1900 1.96 2.58 H0 false (p<0.01) ELLi E 55 1400 5 Serbia 67 502 6.03 1900 1.96 2.58 H0 false (p<0.01) ELLi E 52 1400 6 Serbia 58 502 2.33 1900 1.96 2.58 H0 false (p<0.05) ELLi E 52 1400 7 Serbia 82 502 10.19 1900 1.96 2.58 H0 false (p<0.01) ELLi E 60 1400
* results are interpreted in t-distribution for large samples significance value intervals The reading comprehension items were sequenced in terms of lexical (syntactic and semantic) difficulty and differed in scaffolding provided by the pictures (became progressively more challenging). From syntactically simple items containing content words (the nouns snack, glass, orange juice, bread, chocolate spread), and assisted by pictures showing concrete objects denoted by these content words, the items became more complex linguistically as the content words (the nouns back, idea, ghosts, and verbs turn, happen, need) did not refer to concrete objects in the pictures or to the immediate context iteslf, but
177 required learners to respond to the written prompts and to activate their background knowledge of the world. Moreover, four verbs refered to the past actions (happened, turned, caught, escaped) not presented in the pictures, but expected to be imagined. The differences in individual items scores can be accounted for upon a detailed linguistic analysis of the prompts and multiple choice answers offered in the task, which is given in the following paragraphs (see Appendix 2 for pictures).
Items 1 and 2. Both the prompt and the options of these two items are syntactically simple (see Appendix 2): the statements have subject-verb-object structure, and the questions are not complex, containing the verb to be in the present simple. The content words snack, glass, orange juice, bread, chocolate spread are represented with concrete objects shown in the pictures. The distracter is easily understood as it makes a wrong reference to the picture: ‘I’m hungry and the fridge is empty.’ − the picture shows an open fridge with many items of food on the shelves.
Children in both studies found these two tasks the easiest, with success rate over 75 per cent in the ELLiE study (Szpotowicz and Lindgren, 2011) and over 83 per cent in the Serbian study. Their understanding was very much assisted by the pictures showing concrete objects. In Item 1, children had to respond to the prompt I’ll have a snack. First, a glass of orange juice., and to the picture showing a boy opening the fridge and taking out chocolate spread; the answer is There is some chocolate spread, yum. Item 2 was prompted by a picture of a boy looking at the table from which his food had disappeared, so the answer is Mmmmm this is going to be good! Where‘s the orange juice? Where‘s the bread?
Items 3, 4 and 5. Items that did not have a simple subject-verb-object structure or did not refer to the immediate context or objects in the pictures were more difficult. Item 3 was the most difficult one in both studies and only about a third of learners managed to solve it. In this item there is Tony in the picture opening the door and thinking Oh no! There’s somebody
178 at the door, and saying Coming!, and the learners had to respond to this written prompt and choose the answer consisting of three short sentences: Oh, it‘s you. Something strange just happened. I think we’ve got ghosts. This answer required knowledge of the world and the ability to make inferences from the context of the story. It also contained language unfamiliar to learners in the Serbian context (something, just happened).
Items 4 and 5 were less difficult, with success rates 55 and 52 per cent respectively in the ELLiE study and 65 and 67 per cent respectively in the Serbian study. In Item 4 children could rely on the picture showing Tina in the ‘thinking’ position (touching her chin with her hand), while in Item 5, Tony, who is saying Why do you need it? is even not visible, but the prompt is a simple polite request containing a cognate word (banana), Tony, can you give me a banana, please? Solving this item required making inferences about the situation of the text and the previous events. Establishing global coherence, i.e. using the context clues, seemed to be a problem for about a half of the learners in the ELLiE study, and for one third in the Serbian study.
Items 6 and 7. Item 6 was the second most difficult one for learners in both study contexts with success rates of 58 per cent in the Serbian study, and 52 per cent in the ELLiE study (though in the ELLiE study both items 5 and 6 got the same success rate of 52 per cent). It required activation of higher level discourse processing skills and the creation of the situation model of reading interpretation. Moreover, higher level processing required some type of executive control processing, a monitoring of information activation, text construction, and reader evaluation (within working memory): knowing that monkeys like bananas and that children were trying to play a trick on him, trap him and catch him red- handed taking the banana.
Some learners looked for repeated information and clause-level meaning, but lacked the skills to create a situation model of text interpretation, i.e. a mental model requiring
179 integration between adjacent clauses to establish local coherence and inferences about different events, actions and states – to establish global coherence, to make the text cohere as a whole (Kinch and Rawson, 2005). This is only possible through active, strategic processing and comprehension monitoring.
As for Item 7, it was the second easiest for learners in the Serbian study with success rate of 82 per cent, while it appeared to be very difficult for the children in the ELLiE study, with the success rate of only 62 per cent. One possible explanation of the success of Serbian children in item 7 may be a solid knowledge of narrative text structure, which is positively related to reading comprehension level and can help the reader to organize and relate events in a text, which then benefits memory and understanding (Cain et al., 2004). A plausible explanation is the cognate AHA, an exclamation that has exactly the same form and meaning in English and in Serbian, expressing satisfaction when something is understood or realized.
4.2.2. Discussion
The results of the present research study do not seem to confirm previous studies indicating that girls outperform boys in reading performance as a result of differences in attitudes (OECD, 2014). In respect of the somewhat higher reading scores of the Serbian young learners as compared to the ELLiE study results, it is necessary to consider several aspects related to the two studies. One of these aspects concerns some obvious differences of the two study contexts. First, the Serbian sample was smaller than the ELLiE study sample, involving 502 children, and second, it was more uniform: though aged 10-11, most of the Serbian learners were already aged 11, they all followed the same Serbian Primary Curriculum, with English being compulsory from Grade 1, with two 45-minute English lessons a week, and the EFL reading programme starting in Grade 3, were taught by
180 specialist English language teachers with a university degree, spoke only Serbian as their mother tongue, were drawn only from urban schools with more or less similar socio- economic status, and with some children starting learning English in kindergarten (45 per cent of the sample) and attending out-of-school English classes (40 per cent). On the contrary, the ELLiE study sample was larger, involving 1400 children, and less uniform, as children were 10 to 11 years old, came from seven different country contexts with different foreign language policies and national primary curricula (different number and duration of lessons), were taught by teachers having different qualifications, spoke different mother tongues, had different out-of-school exposure and socio-economic status, and apart from English as a foreign language in six country contexts, Spanish was taught as a foreign language in one country context (Enever, 2011).
The results show that the Serbian young EFL learners have developed reading abilities similar to their peers in the European countries that participated in ELLiE study. The reading scores of the Serbian young EFL learners indicate that they are rather good comprehenders of an unfamiliar narrative text accompanied with pictures, though there are big individual differences in reading scores among them. With English increasingly becoming a vital factor in academic and professional life of young generations, considerable attention must be paid to effectiveness of reading curricula, especially in teaching beginning EFL readers. To become more skillful readers and develop comprehension skill, learners should improve their strategic competence with the help of the EFL teacher. Since language learner strategies “offer the tools for learners to manage their own learning” and have the potential to make language learning more successful, “strategy work has a much greater potential to shape our pedagogy than has hitherto been grasped and that an overly utilitarian, ‘feed-in’ approach to strategies takes a far too limited view of that potential” (Grenfell and Erler, 2007, p. 6). Strategic reading involves both the knowledge about strategies and the
181 ability to apply them strategically (Anderson, 2005; MacKay, 2006). On the other hand, reading outcomes of the sample learners indicate that between a half and almost two thirds of the learners lacked comprehension monitoring and inference making strategies necessary for solving items 6 and 3, respectively. The failure might have been the result of both poor linguistic and strategic competence. So, although the learners in the Serbian study slightly outperformed the learners in the ELLiE study, they still had problems solving the same two most difficult items as the learners in the ELLiE study, i.e. items 3 and 6, and individual achievement scores were highly varied (from 0 to 7). A possible explanation may lie in the strategic competence area, which has been surveyed in the Serbian study through the post- reading reflection task, as described in the following section.
4.3. Post-reading reflection protocol
4.3.1. Results
Post-reading reflection task was used for a qualitative study of reading comprehension. Since protocol analysis is very time consuming, a sub-sample of 90 protocols (18 per cent of the whole sample) was chosen to be analysed. This group was drawn from the school with the best reading scores in the survey in order to find out which comprehension strategies might have contributed to the sub-sample’s above average reading scores in the Serbian context. Another reason for choosing this sub-sample was the fact that a very high percentage of its participants had accounted for their choices in the reading task (see Appendix 7). The sub-sample of 90 learners comprised 43 girls (48 per cent) and 47 boys (52 per cent), distributed into 4 classes in a school located in a big city in Northern Serbia. The sub-sample learners in School 6 (N=90) achieved the best reading results (M=5.54, SD=1.67)
182 and their post-reading reflection protocols were potentially rich in evidence of effective strategies they had applied. Considering the fact that “there is a strong relationship between strategy use and L2 proficiency” (Anderson, 2005, p. 762), the focus was on the sub-sample’s (S6) strategic competence as a variable considered significant for the reading test results. The reading task comprised seven items that were sequenced in terms of lexical (syntactic and semantic) difficulty, and also differed in scaffolding provided by the pictures and the text (became progressively more challenging), as already explained in the previous chapter. Figure 4.2 shows reading scores across seven items expressed in percentages of success achieved by the sub-sample (S6) and the rest of the sample (S1-S5 group).
Figure 4.2: Reading task achievement rates (in seven items) of the sub-sample (S6) and S1- S5 group
It can be seen that the scores for both groups differed considerably across the items, indicating a range of item difficulty and differences in comprehension success. However, the sub-sample’s scores were significantly higher, ranging from 57 per cent (Item 3) to 93 per cent (Item 2), as compared with the results of the S1-S5 group, ranging from 30 per cent
183 (Item 3) to 85 per cent (Item 2); these scores indicated that item 3 was the most difficult and item 2 the easiest for both the sub-sample and the S1-S5 group.
To determine the statistical significance of the above differences the independent- samples t-test was applied, the first sample being the sub-sample (S6, N = 90) and the second being the rest of the study sample (S1-S5 group, N = 412). In this kind of t-test, “scores from two different groups on the same measure are compared” (Larson-Hall, 2010, p. 267), using the means and standard deviations for each of the two groups. Table 4.11 shows the output from the independent-sample t-test.
Table 4.11: Output from the Independent-Samples T-Test (comparison of means and SDs) Comparison of
S6 sub-sample to S1-S5 sub-sample
N Mean Std. Deviation
Reading task scores S1-S5 sub-sample 412 4.57 1.83
S6 sub-sample 90 5.54 1.67
Independent Samples Test Levene's
Test for Equality of
Variances
t-test for Equality of Means
95% Confidence Interval of the Difference F Sig. t df Sig. (2- tailed) Mean Differe nce Std. Error
Difference Lower Upper
Reading task scores Equal variances assumed 1.657 .199 -4.643 500 .000 -.97406 .20978-1.38621 -.56190 Equal variances not assumed -4.924 139.731 .000 -.97406 .19782-1.36517 -.58295
It can be seen from Table 4.11 that there is a statistical significance of the difference in reading scores of the two groups as the p-value of Levene’s test is lower than α = .05, i.e. p = .000, which means that the variances of the two groups are not equal. To determine the
184 statistical significance of the above differences for individual test items, the independent- samples t-test was applied, with the null hypothesis stating that there is no difference between the two independent samples. Statistical significance of t-values obtained was determined on the basis of degrees of freedom (df) and a t-test significance chart. The obtained t-test values, degrees of freedom, a t-test significance chart with p-values, and the status of the null hypothesis for each of the seven items are shown in Table 4.12.
Table 4.12: Output from the Independent-Samples T-Test (comparison of achievement in seven items in percentages)
Items Scores from independent samples (%) Samples (N) t-test values Degrees of freedom (df) Significance value
intervals (p)* Null hypothesis (H0)