To a great extent, the results from large-scale cross-national comparative studies in mathematics education have aroused widespread concerns regarding the methodology used within academia, namely quantitative-led methods. Therefore, there are two aspects in particular that have been the foci of these concerns: quantitative approach and generalisation. Actually, they are interwoven as the quantitative approach is associated with the problem of generalisation. Here, it is notable that Shanghai students’ performance cannot present or be generalised towards the whole country, as they top the national group of pupils in China (Sellar & Lingard, 2013).
There are two main criticisms of quantitative research whose method dominated in PISA:
1. The reliance on instruments and procedures hinders the connection between research and everyday life.
2. The analysis of relationships between variables creates a static view of social life that is independent of people’s lives (Bryman, 2004, pp. 159-160).
Basically, these criticisms cause the demands for qualitative researches. These large projects indeed value quantitative measures of performance over qualitative ones. The results have given rise to two types of methodological concerns: whether the sample could be representative of all students within each country, and whether a valid conclusion could then be generalised to describe the education reality in each country. In other words, the debates surrounding these projects lay stress on the generalised students’ performances around the country, and the so-called several related major factors which influence students’
achievements. One typical enquiry into these projects focuses on choosing the factors related to students’ learning outcomes, what these factors are and why they are chosen.
In terms of the first concern, there was a debate within English academia regarding the disparate nature of English students’ performances in the results of PISA 2000 compared with that of TIMSS 1995 and TIMSS 1999. First, Prais (2003) argued that the fewer
representatives of schools in England would cause bias in reported average scores and representations of students; the strict 15-year-old criteria would exclude older students, so that it might impact on related factors, such as teachability. R. Adams (2003) then criticised Prais’ incomplete understanding of methodology in large-scale assessments. He argued that even non-responses from schools could be caused by a variety of factors, but there was no significant relationship between schools’ average score in GCSEs and whether or not they took part in the survey. Later, Prais (2004) still retained his previous judgement of poor reliability, due to the low response to these projects in England. These criticisms of the inadequate sample of respondents in fact reflect the issue of the representation of schools or individuals, because the purpose of a purely quantitative approach is to generalise the findings.
Particularly, several concerns for how to interpret the league tables have been made because PISA tested how to use knowledge instead of what certain mathematics knowledge
is. Moreover, the diverse foci of the curricula in each participating country influence students’ performance in some ways. Some curricula place emphasis on the solving of problems; some prefer the recall of mathematical knowledge; and others might concentrate on justification and proof (Clarke, 2003). The results have also been criticised as the higher scores that students achieve in these questionnaires only reveal that they could correctly answer more questions. Clarke also proposed the curricular alignment to form the assessment which will be discussed at Chapter 10: Discussion. Secondly, there is the public’s impression that whichever country achieves the higher ranking relates to the intelligence of their
students. Press coverage within the countries that achieved low results showed their refusal to accept their less successful position, such as Canada (Stack, 2006). Similarly, the England’s academic debate proved that the PISA outcome did not meet academic writers’
preconceptions (R. Adams, 2003). It has been increasingly demanded that the quantitative results should be explained in comparative research (Creswell & Clark, 2007) and that other contextual differences should be acknowledged (Lin, Bumgarner, & Chatterji, 2014). This desire of explaining more has resulted in the quantitative data being reused.
Then, those data collected from large-scale projects has been used or combined with other data for other research purposes. In terms of discerning whether USA teaching changes between TIMSS 1995 and TIMSS 1999 had an impact, Jacobs et al. (2006) used videotapes from these two projects to look at eighth-grade (age 13/14) classrooms. In comparing how teachers handle the same topic between Hong Kong and Shanghai, Huang (2002) used the data from Hong Kong classrooms from the TIMSS-Repeat Video study, while Shanghai classrooms were videotaped following the procedure suggested by TIMSS-Repeat. In order to figure out when the students’ performance gap between England and Eastern countries
increases, Jerrim and Choi (2014) compared English students’ and East Asian students’ (Japan, Hong Kong, Singapore and Taiwan) achievements between 10-year-old and 16-year-
old students using the data from TIMSS (4th grade, age 9/10 in TIMSS 2003 and 8th grade, age 13/14 in TIMSS 2007) and PISA 2009 (aged 15/16). In summary, the studies that re-used data within a more contextualised approach are another branch of the comparative study movement, because these studies offered another view of interpreting students’ performance. However, those findings are still not wholly satisfactory for academia, government, or public stakeholders who are eager to identify what can be done to improve students’ performance in a practical way. This study will further examine the students’ performance gap between England and Shanghai, but the gap shown in these large-scale projects is acknowledged as the background view to the present study.
In terms of the second concern, whether or not conclusions from quantitative research are valid for the reality of education has been a controversial issue. Firstly, it is questioned that the sum of each element, such as students’ performance and teachers’ questionnaires, could present the total of the educational reality in each country. Secondly, results from these questionnaires might lack consistency, namely in relation to how individual element operates with respect to others. Thirdly, the generalisation gained from quantitative results only explains the nature of the whole educational reality from a fragmental approach (Cohen, Manion, & Morrison, 2011) as opposed to considering education from a social science perspective.
In the next sub-section, narrowing towards curriculum comparisons, findings are able to provide more details about what has been identified so far in comparative studies.