3. ANáLISIS TERRITORIAL
8.2 Cartera de Iniciativas del Plan
The six RAEs discussed above are exercises that were undertaken over almost three decades. It is not surprising that the system is regarded as “tired”. Having survived its lengthy
lifespan, it is either the case that its intentions should have been achieved by now or that a trend is set that would be able to sustain itself for an extended period of time. If neither of the two occurred, then serious alterations are called for.
The entire system has been summarized below, into the sub-headings of origin and rational, purpose/objectives, processes of evaluation and benefits and challenges experienced in the exercises.
5.8.2 Origin and rational (dates and events are indicated in point form)
The table below summarizes the historical activities in relation to the initiation of the UK research assessment system.
Table 5.2 Dates and historical activities related to system initiation
Date Activity
1970s Evaluation initiatives by OECD/CERI programme – management of performance evaluation in HE, on voluntary basis
Before 1979 All public HEIs predominantly state funded with an academic self- regulatory system
1979 Reduction of public spending and move towards market economy > expectations of institutional accountability to service provision (efficiency and effectiveness) and the improvement of quality of teaching and research This led to the introduction of competition for the GUF funds
1980 SRHE/Leverhulme programme put performance on the agenda Government steering power compelled universities and Committee of Principals to base activities on performance evaluation
1980 UGC Introduced reduction of funds to universities
DES followed and reduced research funding to public sector institutions Universities resorted to non-governmental funding
1985 Jarrat Committee – on efficiency in universities led to DES Green Paper – Development of HE into the 1990s This introduced the ‘value for money’ philosophy
1985 Introduction of the UGC’ research selectivity exercise (first cycle) 1987 DES White Paper – HE, meeting the challenges
NB. Information on the series of exercises (including dates) is indicated in the subsections below.
5.8.3 Main funding bodies in the UK HE research system
- Government subsidy through block grants (GUF) > ex-post results - Research councils > ex-ante
5.8.4 Objectives, processes, benefits/good practice and challenges/lessons learnt
The tables that follow provide a summary of the variables according to funding organizations and periods of occurrences. Table 5.3 indicates the dates of the cycles, the period for which data was required, the purpose of evaluation and activities according to the cycle periods. Table 5.4, on the other hand displays benefits/good practice and challenges together with lessons learnt throughout the series of research exercises.
Table 5.3 Objectives and processes in the RAE exercises
Funding Org. Purpose Process
UGC – 1986 Data: 1980-1984
Resources provided by government for research purposes to be allocated on the basis of those achieving the highest rating receiving the highest share
-Informed peer reviews -Data:
Inputs on research grants
Brief account of university research performance
Five publications seen as representing unit research work Future research plans
-Rates 1-3 UFC – 1989
Data: 1984-1988
To best allocate resources – more refined targeting of research resources
-Peer reviews
-Response to criticism of the 1st cycle -Transparency in deliberations
-Output oriented data -Use of advisory groups
-Data: Output, reports on departmental achievements, summary of publications and research reports, future research plans, up to two publications per fulltime research staff.
-National and international standards introduced (level of excellence) -Rates: 1-5
HEFCs – 1992 Data: 1988-1992
Inform selective funding (allocate resources on the basis of rating received) with the following secondary intentions
-Improve quality of teaching and research
-Purposes of accountability to state funds
-Introduction of competition for resources
-Funding given according to perceived quality
-Subjective peer review based on available information -Audits, performance indicators
-Research units as units of analysis -Number of research active staff per unit
-Data: publications (two per research staff), other quantitative materials, description of research plan, research grants, post-graduate research activities, citation
-Quality of national and international standards -Report published by HEFCE
-Some of the panels provided confidential feedback
HEFCs – 1996 Data: 1992-1996
Rating of research quality for use by funding bodies in allocating funds to HEIs. Results were also used to advise government to make final decisions
-Mostly based on 1992 methods but improved
-Standard definition of research (gaining of knowledge and understanding invention and generation of new ideas relevant to commerce, industry, the public and leading to substantial improvements)
-Alteration of units of research (fewer)
-Concentration on quality and not volume (public available outcomes) -Operation within same framework with allowance for differences between subjects
-Data was submitted on software, making it easier to submit and easier to check -Block-grant funding (salaries of permanent academic staff, computing costs, support for basic research, research training)
-Chairpersons appointed by CEO
-Panels selected by academics, professional bodies, research councils and charity organizations
-Panels allowed to refer work to other panels -Period of panel meetings four X two days meeting -Secretaries provided by research organizations -Inquiry made telephonically or by email -Scale 1-5* (seven points)
-Data: names of active researchers, four publications, research students, research statement,
HEFCs – 2001 Data: 1996-2000 (Humanities data for seven years)
To produce rating of research quality to inform allocation of funds
-Ireland added (DENI/DEL)
-Five- year period between cycles recommended -Peer review
-Data (output): edited books, journal articles, conference proceedings, 4 items per research active staff, graduate students, research activity statement, funding (income)
-Definition of quality more consistent -More information sort from institutions
-7 point scale (1,2,3a,3b,4,5,5*), ‘4’ upgraded to international -Fourteen UoA, combination of some units, inclusion of sub-units -Umbrella groups for panels to share criteria
-Panel criteria drawn after consultation and released 16 months prior to exercise -Advisors allowed
-Seminars held to discuss framework and data collection -Ratings published and reports submitted to institutions -Calls for nomination of panel members to form pool HEFCs – Post 2001 Produce rates for the allocation of
funds
(need to provoke thinking beyond publications)
-Broader consultations conducted -Change to six-year cycle
-Further reduction in panels -Grading of joint submissions
-Indicators: research output (journals, conference papers, intellectual properties), research environment (students, income, structures), esteem (international recognition, fellowships and participation in advisory bodies) -Quality improved from 50% national/50% international to world class HEFCs 2008 Create quality profiles for each
submission in order to assist in the determination of grants for 2009 to 2010
The use of a two-tier panel structure for standardization and clarity on criteria to accommodate the assessment of applied, practice-based and interdisciplinary research
Table 5.4 Benefits/best practices and challenges/lessons learnt in the RAE exercises
Funding Org. Benefits and good practice Challenges/lessons learnt
UGC – 1986 -Systematic/comprehensive research evaluation had been introduced
-Results used to rate universities
-The use of input measures
-Bias in favour of large department (data) -Lack of clarity on quality criteria
-Anonymity of assessors, a problem to scholars -Methods not suitable to all subjects
-Different evaluation standards UFC – 1989 -Output used as a measure of quality (reliance on
journals)
-Introduction of level of excellence
-Introduction of national and international standards
-Questionnaire for data collection not adequate -Short period for evaluation
-Problems with definition of research units -Problems with the definition of outputs -Outputs introduced cheating
-Problems of inter-subjective differences
-No consideration of institutional size and type, benefiting the previously advantaged – Oxbridge effect (Culture of an institution depending on institutional decision context, with production of output serving as an input factor)
HEFCs – 1992 -Turning point of evaluation in the UK -Reduction of reliance on government -Diversification of income
-Promotion of elitism
-High volumes of submissions from units -Fewer units submitted than expected
-Securing increasingly more scares resources -Acceptance of the exercise
-Early preparations for the next exercise
-Strategic planning for future exercises, revisiting of missions and visions
-The drive to receive funds improved institutional performance (inspiration for better performance) especially in the not-so-well-performing institutions -Increase in research staff
-Improved system of research management -Establishment of research graduate schools -Encouragement of interdisciplinary research -Institutions of different cultures compete -Improvement in the quality of research -Increase in the funding avenue
-Creation of staff mobility (poaching)
-Less influence on established research institutions and researchers
-No encouragement for low grade researchers
-Conformity to expectations reduced diversity in research -Low or no correlation between RAE and other research measures
-Process of funding not transparent
-A small number of institutions continue to thrive -Separation of teaching and research
HEFCs – 1996 (Managed by HEFCE)
-Submissions had increased by11%
-Exercise more established and improved as the 4th in the series
-Early preparations of procedures and guidelines (published by 1995)
-Process piloted
-Some panels viewed the set standards to mean visibility and not quality
-Quality depended on subject acceptance of activities of a journal
-Software used not well prepared (corrupt data not detectable)
-Panel criteria improved through consultation with academics
-Basic and applied research submitted together -Institutions still made a choice on what to submit -Developmental feedback separately submitted to institutions and handled with confidentiality
-Seminars held to explain requirements (e.g. software)
-Rating not compatible across units -Varied submissions
-Standard definition of quality across units affect rating -Referrals (inter-panel) affected different criteria between panels
-Problems with classification of some variables (e.g. conference papers) by different panels
-Pooling affected data
-Sole authorship not considered in other panels
-Not enough (input) information from research councils HEFCs – 2001 -Exercise preceded by a broad consultation
-Planning started as early as 1996 -Definition of quality more consistent -More information sought from institutions - Humanities data of seven years
-Advisors allowed (including internationals) -Umbrella groups for panels to share criteria
-Panel criteria drawn after consultation and released 16 months prior to exercise
-Seminars held to discuss framework and data collection -Ratings published and reports submitted to institutions
-Number of submissions declined
-Increased finances to implement improvements (exercise more expensive)
-Poaching leading to movement of staff -Incorrect submissions
-Calls for nomination of panel members to form pool -Poached staff replaced with others that would also be trained (departmental arrangement)
-Data verified
HEFCs – Post 2001 -Further consultations conducted
-Quality of research is said to have improved -Improvement in research management
-Funding further reduced
General challenges: HEFCs
- Teaching suffered at the expense of research - Extreme pressure for institutions to deliver - Resultant hierarchy of universities
- Reduction of diversity as a result of conformity
General Benefits: HEFCs
- Introduced system of research management - Establishment of research graduate school - Encouragement of interdisciplinary research - ‘Improvement’ in quality of research
5.8.3 A summary of differences and similarities between the Dutch and the UK models
The following table sums up differences and similarities between the two models in relation to Campbell’s (Shapira and Kuhlmann, 2003) analysis:
Table 5.5 Differences and similarities between the Dutch and the UK models
Item Dutch UK
1. Responsible bodies VSNU (Association of major universities) HEFCs (Intermediary institution responsible for funding) 2. Linkages Self-evaluation of universities
Autonomy within the system of generous public funds
Close to public decision making concerning public funds
3. Duration/length of cycle
Six to seven years Lately six years
4. Number of disciplines
34
Less and reducing all the time
Inter-disciplinary research encouraged
1989 – 92 1996 – 69
2001 – 68 (too many, have institutional assignment problems)
5. Units of assessment Research programmes (smaller than a department) University department.
Problems of inter-disciplinary research Sub-units introduced to solve problem
discipline panels -Panel chair is Dutch and nominated by institutional academic and selected on consensus
-Other panel members are local and international -Evaluation carried out in English except for domestic oriented disciplines
-Procedure standardized across disciplines and published in advance
-Disciplines may adapt but not change methodology, therefore results may be compared across disciplines
-Sub-panels used where necessary
-Chairpersons nominated from panel members of previous exercise and appointed by CEO of funding organization -Panel members representing geographical areas and different characteristics of universities
-Procedure standardized and published prior to evaluation
7. Content of submission
-General information: inputs, PhD thesis -Five best academic publications per research programme over a predetermined period
-Faculty of departments (or institutions) forward information from internal assessment to panels
-General information: staff information, research output, textual description and related data
-Up to four items of research output per researcher over a predetermined period (output accessible to public
-Units submit to panels 8. Components -Academic quality
-Academic productivity (efficiency) -Relevance
-Academic viability
The multi-dimensional definition highlights complexity
-One quality dimension as an overall quality for research (reliance on decisions of journals and international standards) Funding dominates university research
137
Chapter 6 The New Zealand research assessment exercise
6.1 Introduction
Information on this model was obtained from the 2003 and the 2006 periodic research exercise reports and only discusses operations of the one exercise. There is more discussion on the 2003 exercise. The 2003 periodic research performance exercise is the first
comprehensive evaluation of ‘research quality’ in New Zealand. The exercise is said to have been costly, demanding and time consuming but systematic and authoritative. Activities of the 2006 exercise on the other hand were based on the experiences of the first with some added improvements.
The first section provides the background, based on the history and rationale for the model, followed by the purpose for the evaluation. The evaluation processes are provided under the section on methodology and these are followed by the challenges encountered in the model. The chapter ends with a conclusion and a summary.
6.2 Background on the exercise
The New Zealand tertiary education system has since the 1980s undergone several changes in relation to the changing economy of the country and to some extent changes in the ruling party (McLaughlin, 2003). The former resulted from the dependency of tertiary education on national funds. Changes in the economy were affected by the funding of the tertiary education system. In the 1980s, there was a demand for the tertiary education system to ensure “a knowledge economy” and “the need in the changing economy to provide tertiary education to a high proportion of the population” (McLaughlin, 2003:13). The increase in tertiary
participation compelled government subsidy policies to shift from the “elite system” of subsidizing smaller numbers with higher amounts to a “mass system” of providing for more students with lower rates. This led to the reduction in the differentiation of institutions. Large numbers of very diverse students would be expected to have quality implications and as a result, the need arose for government to develop strategies on dealing with at-risk groups.
138 According to McLaughlin (2003), these demands have been imposed regardless of political party, philosophy or even policy approach. Whereas the idea of a broader access was initiated by the Labour-led government in the 1980s, the National-led government carried out its implementation, with a few additions such as the involvement of the private sector in the funding process. The National government also moved towards a competitive, market-based approach. When the pressure of student numbers and funding increased, government looked for alternative ways to solve the problems. One thing led to another, as government was caught up in the absence of accountability and capacity for governance issues. These challenges led to several tertiary education policy reviews in relation to the funding of the system.
A 1997 discussion paper and the White Paper of 1998 were formulated to guide policies to be more responsive to students needs. This would mean, among other things, that there would be improvement in accountability, improvement in governance and capacity building measures (for the competitive model), the introduction of quality assurance and audit mechanisms (for quality teaching and research) and alterations in research funding. Although government made repeated requests for inputs from the universities, the reviews were mainly government- driven, leaving the state with high steering powers.
In the late 1990s, New Zealand was characterized by changes in government, in which the National-led government gave way to the Labour-Alliance government (Tanczos, 2002, PBRF, 2003, Wrath, 2005). Prior to this change of government, the HE system had already experienced problems (stated above), which according to Tanczos (2002) would hopefully be solved by the new system.
Among other things, the system was blamed for policies that were incompatible with access to tertiary education (Tanczos, 2002). In research in particular the government was criticized for what was referred to as brain drain, which was interpreted as neglect “to nurture ‘a research culture with the right mix of funding, incentives, and devotion to the spirit of intellectual inquiry’.” (Wrath, 2005:1). Pressure was mounting at a time when government was losing power to the left-wing, the Labour Party.
139 According to Wrath (2005:1) the movement of tertiary education research staff to universities abroad was the result of the National-led government staff-exchange policy which was later interpreted as “the plight of ‘academic refugees’ driven abroad by economic need”.
Unfortunately the policy opened access to information about the advantages of being a researcher abroad and created a path for academics to search for greener pastures. The solution to the brain drain problem would therefore have to reward those that participated in research in order to retain them.
Government was compelled to review its policies and in 1999 when the Labour Party took over, the Party used this opportunity and worked on a vision to strengthen the system of tertiary education research (PBRF, 2003). It is obvious that the changes in party leadership had some effect on the system. The competitive spirit of the parties nevertheless seemed positive in that the intention was attached to votes and thus meant to make an impression. The new government established the Tertiary Education Advisory Committee (TEAC) to help formulate long-term strategic direction for tertiary education (McLaughlin, 2003). The Committee’s are said to be more inclined to tertiary education systems than issues of access and student support. A manifesto was published that related HE research to national
economic and social development. Another initiative of the new government was the Tertiary Education Strategy 2002/07. The following were advocated:
- The establishment of seven centres of research excellence
- The introduction of a programme to build research capacity in the Social Sciences - The creation of a Performance-based Research Funding (PBRF) exercise
The PBRF is the centre of discussions in this section because it discusses issues of research evaluation. In the plan to implement the PBRF, government would rely on the results of investigations and advice by different bodies, for example; a series of audits commissioned by the Ministry of Education, the 2002 results of the PBRF working group, the Transition
Tertiary Education Advisory Commission in 2001 and 2002 and a wide consultation with the Tertiary Education sector.
140 The Tertiary Education Commission (TEC) was assigned responsibility to manage the PBRF exercise. This was done through a constituted project team, responsible for policy, technical and administrative support. Panel chairpersons (sometimes referred to as ‘Chairs’) and panel members were appointed to take responsibility for evaluation. The reviewers received “some briefing” prior to the commencement of the exercise.
6.3 Purpose for the evaluation exercise
The multiple-purpose nature of the model is observable. This may be the result of the many reviews that were undertaken on the tertiary education system. For example, the report on the 2003 PBRF assessment exercise (PBRF, 2003) states that the aim of the exercise is to increase the average quality of academic research and to encourage research to support teaching. This is despite the fact that (as highlighted by Wrath in 2005) government’s intention was to