Latecoming

Hardly anyone who is interested in the higher education sector can have failed to be aware of the annual rankings published by the Jiao Tong-university in Shanghai or by Times Higher Education – two of the very limited number of international ranking systems in the world.

Shanghai Jiao Tong’s Academic Ranking of World Universities In 2003, the Jiao Tong-university started to rank research universities around the world in accordance with its achievements in the Academic Ranking of World Universities (ARWU).74 _{The original aim was to study the gap between}

74. See Liu & Cheng (2005), Cheng & Liu (2007) and www.arwu.org for more information about the ARWU ranking system.

Chinese universities and world-class universities. The high degree of concen- tration on research in the ARWU tanking (see Figure 1, above) is because it is not possible to rank the quality if higher education all over the world due to the lack of comparable data (which, in its turn, depends on the considerable differences between counties and between universities).

The ranking group at the Jiao Tong-university claims that it is easier to find internationally comparable data for the quality of research. The strong point in their ranking is that they only employ data that can be checked. No infor- mation is obtained directly from the university concerned or from subjective surveys. This ranking is based on a limited number of indicators of two types: bibliometric citation indicators and scientific prizes (the Nobel Prize and the Field medals awarded for mathematics).

The ARWU ranking has been criticised on two grounds, right from the start. On the one hand, there has been criticism of its strong focus on research – its failure to cover any aspects of quality in higher education. A more relevant criticism concerns the selection of indicators to depict research quality. In the case of scientific prizes, this type of indicator has been criticised because the variation is extremely low (quite simply there are very few Nobel and Field awards). Another question that is often raised is what the number of Nobel Prize winners from a given institution actually says about quality. In addition, this is considered to be a far too superficial attribute, which can also be mani- pulated by employing Nobel Prize winners who do not actually participate in the institution’s activities.75

Considerable criticism has also been directed at the bibliometrically based citation indicators, in particular because this type of indicator is a poor way of depicting research quality, especially in the humanities and the social sciences, although medical and natural science research is published in a manner that complies rather well with the measures for bibliometric studies of this nature. Another common criticism is that bibliometric studies favour “Anglo-Saxon” universities, since they are accustomed to publishing their research in English- speaking journals.76

Times Higher Education QS World Universities Ranking

The Times Higher Education QS (previously the Times Educational Supple- ment) World Universities ranking is more of hotchpotch than the AWRU ran- king.77 _{The purpose of this ranking, ever since it started in 2004, is to offer an}

international comparison between various universities’ success in attempting to achieve world class, while simultaneously recognising that universities are multi-faceted organisations. Times Higher Education QS (THE-QS) identi- 75. Marginson & van der Wende (2006), Kälvemark (2007), Kälvemark (2008) and HEFCE

Issues paper 2008/14.

76. Rabow & Rabow (2005), Cavallin & Lindblad (2005), Usher & Savino (2006), Lund (2007) and Cunningham (2008).

fies four pillars on which it considers a world-class university is based: research quality, teaching quality, student employability and international outlook.

These four pillars result in six different indicators obtained from various sources. Research quality is measured by an academic “peer-review” in the form of a questionnaire survey in which academics state which institution, in their opinion, has the best research in their own particular discipline. The number of citations per academic staff member is also used as an indicator for research quality in this context. Teaching quality is measured in terms of the number of students per teacher, and student employability is also measured by means of a questionnaire survey (“employer review”) distributed to various potential employers. Finally, international outlook is measured by two indica- tors: the proportion of international teachers and researchers, and the propor- tion of international students.

The THE-QS World Universities ranking has also been criticised for failing to delineate education and teaching quality, and the indicator that is suppo- sed to measure research quality (academic peer review) also has a very heavy weighting in this ranking (40 per cent).

But the major criticism of the THE-QS ranking is that some of the sour- ces of data – both the questionnaire survey distributed to academics and the questionnaire for employers – are not transparent, and because they are purely a reflection of reputation and do not manage to chart quality. This is a typi- cal problem for this type of questionnaire survey, as clearly illustrated by the example of an American questionnaire directed at students asking which were the best law schools. Princeton was ranked as one of the 10 best, although Prin- ceton does not have a law school. This type of questionnaire appears to merely recycle reputations rather than rewarding genuine quality and, as a result, is only another form of popularity competition.78

The internationalisation dimension in THE-QS has also been criticised – partly because it is difficult to make comparisons between countries and partly because it tends to favour “Anglo-Saxon” universities. THE-QS has also been criticised for its citation indicator, on the same grounds as ARWU.79

The THE-QS World Universities ranking has modified its indicators and methodology over time, and this has affected the positions of higher education institutions in the ranking lists. Changes in ranking positions that might be interpreted as a dramatic shift in the performance of the institutions concerned may simply reflect a change in the way measurements are made.80

There are a few additional international rankings that cover universities in many parts of the world, but their focus is rather specific. The Leiden University’s various bibliometric rankings are exclusively based on scientific 78. Frank & Cook (1995), Guarino et al. (2005), Altbach (2006) and Marginson & van der

Wende (2006).

79. Cunningham (2008), Marginson & van der Wende (2006), Rabow & Rabow (2005) and Calavallin & Lindblad (2005).

citations and publications and, as a result, do not depict any aspects of the quality of education programmes. Webometrics ranks universities and higher education institutions on the basis of the extent to which they use the Internet, but with rather dubious results.81

Specific problems in international rankings

The designers of international ranking systems face two problems of a general nature that have consequences for the quality of such systems. Firstly, it is hard to find comparable measures on a worldwide basis, because higher education takes such different forms in different countries. Secondly, the quality of the gathering of information about the higher education sector varies considera- bly. One result is that the international rankings focus more on the quality of research than on the quality of education, since it is simpler to find compara- ble data for research. The attempt to find comparable criteria also means that the international rankings often apply relatively few and relatively superficial indicators.82

Variations in the classification of institutions, research and programmes in various countries also present problems for the designers on international ran- kings, and it is by no means always certain that the units compared by app- lying the same criteria are particularly similar. Even a simple matter such as the national names of institutions can be a problem. In Brussels, for example, there are two quite separate universities whose names, in both cases, would be best translated into English as the “Free University of Brussels”. One of the, Université Libre de Bruxelles, is a French-speaking university, while the other, Vrije Universiteit Bussel, is Flemish-speaking. The extent to which the international ranking designers are aware of the functions performed by the university concerned is far from clear. This also applies to university hospitals, which are sometimes ranked as separate units and sometimes as part of a larger institution. Another example of the difficulties in interpreting national langu- ages is the German CHE report on “excellent universities” at the second cycle level, in which the Swedish Royal Institute of Technology (KTH) is consis- tently referred to as “KTH – Sweden’s Largest Technical University”.83 _Some

mistakes (perhaps many) in data management and processing are probably the result of lack of knowledge about systems at the national level.

The United States – a wide range