CHE (Zeit)
2. The purpose and target group of the ranking should be clearly stated. [4]
The purpose and target group is easy to find, but the third and fourth target groups are unclear.
3. The ranking should take the various missions and goals of institutions into account. [4]
They try to, but do not entirely succeed. The institutions are divided up by sub- ject, but no more than that. However, they refrain from using indicators that they feel are inappropriate for making comparisons between countries (e.g. research budget). They also have a primary aim (consumer guidance for master’s students), which reduces the significance of differences in missions and goals.
4. Several different data sources should be used for the ranking; these should be clearly stated and explained to the consumers. [4]
Several of the data sources are not stated, but they are probably fairly standard. The survey material is not accessible.
5. The linguistic, cultural, economic and historical contexts of the institutions being ranked should be specified. [1]
No information is given about this.
6. The methodology used to create the ranking should be transparent. [4]
Information is provided about the methodology used and the reasoning behind it is fairly detailed, but there are some issues that are unclear. However, this mainly applies to those indicators that have not been used as selection criteria for the ex- cellence group.
7. Indicators should be chosen according to relevance and validity, and it should be clear what they are meant to represent. [3]
Many indicators that are reasonably valid and that together cover a large element of the purpose. They have also refrained from using indicators that are not re- garded as being of sufficiently high quality, as well as from measuring subjects for which they do not feel there are any good indicators. However, there is a degree of overlap between the indicators and several aspects are missing. One major de- ficiency is that only EU projects, programmes and funding are counted.
8. Measure outcomes in preference to inputs whenever possible. [4]
Difficult to say exactly, since no complete explanation is provided of the link be- tween the indicators and the stars. However, the balance appears to be reasona- ble.
9. If several indicators are used, the weighting between them should be given a prominent position in the presentation, and changes to the weighting should be avoided. [3]
The weighting is complicated, but details are provided. It is not entirely clear whether the weighting has been changed, but if it has then it is only a matter of minor changes.
11. Only audited and verifiable data should be used.[3]
We know nothing about the survey material and some other areas. However, the sources that are used for the selection criteria for the excellence group are most likely OK (WoS and, probably, official EU statistics).
12. Only data that have been collected according to the proper procedures for scientific data collection should be used.
The student questionnaire is sent to an extremely small selection. The contact person questionnaire was carried out solely for the purpose of the ranking. How- ever, the sources that are used for the selection criteria for the excellence group are OK (WoS and, probably, official EU statistics).
13. The ranking process should be quality-assured. [1]
Nothing is mentioned about this.
14. The ranking should be organised in such a way that it enhances the credibility of rankings. [2]
Nothing is mentioned about this. However, CHE is independent.
15. Consumers should be given a clear explanation of all the factors used to develop a ranking, and they should be offered the opportunity to choose how the ranking is displayed. [3]
Considerable flexibility, but difficult to know what the factors actually mean (they are often presented in a group). It is far from clear that certain stars carry more weight than others.
16. The ranking should be presented in a way that eliminates or reduces errors in original data, and it should be organised and published in a way that makes it possible to detect and correct er- rors.[1]
A: 0. B: 1. C: 0. D: 0.
CWTS (Leiden)
2. The purpose and target group of the ranking should be clearly stated. [1]
Have been unable to find any information on this.
3. The ranking should take the various missions and goals of institutions into account. [2]
This is not really considered, but you can show the results per country, with some difficulty.
4. Several different data sources should be used for the ranking; these should be clearly stated and explained to the consumers. [2]
It is either Scopus or Web of Science, but it is not stated.
5. The linguistic, cultural, economic and historical contexts of the institutions being ranked should be specified. [1]
Not specified.
6. The methodology used to create the ranking should be transparent. [4]
The methods are reasonably well explained, but not in enough detail to enable someone to repeat the ranking.
7. Indicators should be chosen according to relevance and validity, and it should be clear what they are meant to represent. [1]
No score can be given for this since the purpose is not stated.
8. Measure outcomes in preference to inputs whenever possible. [5]
All indicators measure results. It is not clear what a ‘good balance’ is in this con- text, since the purpose of the ranking is not stated.
9. If several indicators are used, the weighting between them should be given a prominent position in the presentation, and changes to the weighting should be avoided. [5]
Weightings are completely clear and there have been no changes.
11. Only audited and verifiable data should be used. [4]
Either Scopus or Web of Science is used.
12. Only data that have been collected according to the proper procedures for scientific data collection should be used. [4]
Scopus or Web of Science.
13. The ranking process should be quality assured. [1]
No information on this.
14. The ranking should be organised in such a way that it enhances the credibility of rankings. [1]
Nothing about this is stated.
15. Consumers should be given a clear explanation of all the factors used to develop a ranking, and they should be offered the opportunity to choose how the ranking is displayed. [3]
You can sort the list according to the desired indicator. However, weighting can- not be chosen and the factors are not explained.
16. The ranking should be presented in a way that eliminates or reduces errors in original data, and it should be organised and published in a way that makes it possible to detect and correct errors.
[1]
HEEACT (Taiwan List)
2. The purpose and target group of the ranking should be clearly stated. [2]
The purpose is to ‘evaluate and rank the scientific research paper performance’, which is more of an operational definition than a purpose. However, one can as- sume that they mean quality of research.
3. The ranking should take the various missions and goals of institutions into account. [2]
This is not really considered, but it is easy to show the results per country.
4. Several different data sources should be used for the ranking; these should be clearly stated and explained to the consumers. [3]
Only Web of Science, which is clearly stated.
5. The linguistic, cultural, economic and historical contexts of the institutions being ranked should be specified. [1]
Nothing.
6. The methodology used to create the ranking should be transparent. [4]
The methods are reasonably well explained, but not in enough detail to enable someone to repeat the ranking.
7. Indicators should be chosen according to relevance and validity, and it should be clear what they are meant to represent. [2]
Assuming that they are trying to measure quality of research, the decision to only use bibliometric methods is not terribly appropriate.
8. Measure outcomes in preference to inputs whenever possible. [5]
All indicators measure results.
9. If several indicators are used, the weighting between them should be given a prominent position in the presentation, and changes to the weighting should be avoided. [5]
Weightings are completely clear and there have only been minor changes.
11. Only audited and verifiable data should be used. [4]
The data come from Thomson Reuters (exact products are stated), but since it is an indirect source that is not open it is difficult to know the exact kind of data used.
12. Only data that have been collected according to the proper procedures for scientific data collection should be used. [4]
The Thomson Reuters database is used in part, with the problems that entails, in combination with a proprietary name harmonisation process. It is not clear how this is done.
13. The ranking process should be quality-assured. [1]
14. The ranking should be organised in such a way that it enhances the credibility of rankings. [2]
Nothing. However, HEEACT is independent.
15. Consumers should be given a clear explanation of all the factors used to develop a ranking, and they should be offered the opportunity to choose how the ranking is displayed. [4]
Reasonably well presented, but the consumer cannot choose the weighting freely, neither are the characteristics of the indicators explained.
16. The ranking should be presented in a way that eliminates or reduces errors in original data, and it should be organised and published in a way that makes is possible to detect and correct errors.
[1]
A: 0. B:1. C: 0. D: 1.