existence of national higher education monitoring agencies; academics are not government employees and are free to move between institutions; the CEO is chosen by the university; and there is complete flexibility in the appointment of foreign academics.
(No more detailed information is given regarding calculation of the scores). Weight within E4 composite indicator: 24 9.55%
CONNECTIVITY indicators
Weights within the Measure 3Connectivity
C1 Proportion of international students in tertiary education, 2009 Not available 5%
C2
Proportion of articles jointly authored with international collaborators, 2005-2009.
The data is a weighted average for each country where the weights are the proportion of output from each higher education institution.40
Not available 5%
OUTPUT indicators
Weights within the Measure 4Output
O1 Total articles produced by higher education institutions, 2005-2009 (SCImago data used) 4 13.33%
O2 Total articles produced by HEIs per capita, 2005-2009 1 3.33%
O3 A mean-normalised citation impact indicator from the SCImago database, for 2005-2009. The country scores are calculated using as
weights the share of national publications contributed by each HEI. 1 3.33%
O4
“Depth of good universities in a country”. A weighted average of the number of HEIs listed in the SRC ARWU Ranking top 500 for 2011 divided by the country’s population.41 The weights used are
the scores out of 100 for each university.
1 3.33%
O5 Average score of the three universities with the highest scores in
the SRC ARWU Ranking. 1 3.33%
O6 Higher education enrolment as a proportion of the five-year age group following the completion of secondary education, 2009 1 3.33%
O7 Proportion of the population aged over 24 with a tertiary education qualification, 2009 1 3.33%
O8 Number of (FTE) researchers per capita, 2009 1 3.33%
O9 Unemployment rates among those with tertiary education aged 25-64, compared to unemployment rates for those with only
upper secondary or post-secondary non-tertiary education, 2009. 1 3.33% Source: Williams et al., 2012
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
Conclusions
While the development of a systems’ level ranking is an interesting new approach, as indicated in Part I there are many open questions. For example, as the weights of the indicators in the overall ranking have not been provided, it is very hard to determine which indicators have the greatest and least impact on the overall score, as the description of indicator weights is also confusing. The required calculations have been performed and the weight of each indicator added in the course of preparing the present report. While it has been assumed that the two “connectivity” indicators are equal in weight, nothing is said about them
either in the overall report (Williams et al., 2012) or on the U21 website.42
After calculating the real weights of each indicator for the purposes of this report, it emerged that the most influential single indicator is O1 (“total articles produced by higher education institutions, 2005-2009”), with a weight of 13.33%. Together with the other two bibliometric indicators, O2 and O3, this results in a total weight of 20% for publications and citations.
As regards indicator E4.1, the subdivision of higher education institutions into public, government- dependent private and independent private categories merely specifies their ownership status, and says little about their real diversity. So while readily accessible, the indicator appears to have little meaning. Indicator E4.2, which is derived from the World Economic Forum scores based on country responses regarding their own higher education systems, may vary in accordance with the national traditions or stereotypes determining whether countries regard their higher education systems favourably or unfavourably (see also Milot, 2012). Use of this indicator for the purposes of global comparison is therefore questionable. The secondary use of SRC ARWU Ranking scores in the indicators O4 and O5 strengthens the positions of big or rich countries with universities strong in medicine and natural sciences. Alex Usher (2012) argues that awarding points for the percentage of the population over 24 with a degree (indicator O7) privileges those countries that expanded their higher education systems some time ago (i.e. the US) and also criticises indicators such as researchers per capita (indicator O8).
Thus there is certainly room for methodological improvement, and there are also some more general concerns that should be raised, for example that the broad diversity of higher education systems is unduly condensed into a very few numerical characteristics, and that some indicators are calculated directly from the results of the most elitist existing rankings. Furthermore questions can be raised about the positions of some countries. It is not clear in the overall ranking table, for example, why Ukraine ranks higher than Czech Republic, Poland, Slovenia and Italy; in the sub-ranking on resources the UK is ranked below Iran, and the Ukraine higher than most EU countries; the sub-ranking on environment ranks Switzerland below Chile and in connectivity Canada is below Indonesia. Finally it would be helpful for users to know the weights of individual indicators in the overall score. This information is not available at present.
11. SCImago Rankings
SCImago is a research group engaged in information analysis, representation and retrieval. Its members are researchers from the Instituto de Políticas y Bienes Públicos, the Spanish National Research Council (CSIC), the University of Granada, Carlos III University of Madrid, the University of Extremadura, the University of Alcalá de Henares, the University of Porto and SCImago Lab.
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
43 See: http://www.scimagoir.com/methodology.php?page=cross_sectorial
44 This means that if the indicator value is greater than 1, the institution’s citation impact is higher than the world average; if the indicator value is less than 1
then its publications are cited on average less than elsewhere in the world (see also Rauhvargers, 2011, p. 39).
45 For further information on the Gini index, see: http://en.wikipedia.org/wiki/Gini_coefficient#Calculation