Based on these assumptions (and the theoretical logic established in Chapter Three), the empirical dimension of this research was centred on CHAT-based action research case studies in two distinctive environments in the College of Law at the Australian National University (ANU). The College of Law is one of the five colleges of the university and offers a broad range of undergraduate and postgraduate coursework programs, as well as higher degrees by research. In 2011, it had 1573 full-time equivalent students (with roughly half being undergraduates) and 126 staff (76 of whom were academics).
As introduced in Chapter Two, ANU was an early adopter of student feedback based evaluation. In the early 1980’s, the ANU Office for Research in Academic Methods (ORAM) developed a student feedback system, which was broadly based on the work of Falk and Dow (1971) and TenBrink (1974) (Miller, 1984). A series of quantitative, ratings-based student questionnaires on teaching and courses were developed, which offered the opportunity for academics to choose questions from a question bank. The voluntary system was designed to be administered either by ORAM or individual
academics. The explicit objective of the system was to improve the quality of individual teaching and to counter the rising negative teaching reviews offered in student
alternative handbooks (Miller, 1984, 1988). In early 1994, a new and expanded student evaluation system (the ANU Student Evaluation of Teaching) was introduced. This coinciding with the rise of the broader quality assurance demands for the higher
education sector and the introduction of the CEQ described in the previous chapter. This more automated and centrally managed system remained voluntary, but offered an expanded range of questionnaires for large and small class teaching, courses and an open-ended question form. For the first time, the system produced computer-generated student feedback reports that, over the following years, generated a longitudinal database (which aggregated data in discipline clusters). Areas of the university were encouraged to use this aggregated data to review performance and undertake planning. However, individual data was to remain private and the system not compulsory (despite a number of subsequent internal debates on these matters). The system also was
administered and supported by the recently formed ANU academic development unit, the Centre for Educational Development and Academic Methods (CEDAM).
A major review of the ANUSET system in 2006 found a growing use of the student feedback system over the preceding five years, with around half of all courses offered by ANU being evaluated (Wellsman, 2006). Based on a series of interviews with key university leaders, this review concluded that the main driver behind participation in the ANUSET system was academic promotion and school positioning in difficult student markets, rather than course improvement as such. The Review also discovered some ‘impatience’ amongst these leaders as to the:
non-compulsory nature and inconsistent levels of use of the system lack of broad access to feedback data on individual academics
limited ability to undertake comparative analysis of academic performance (Wellsman, 2006)
In 2008, with a plateauing in the use of ANUSET system, a further review was undertaken. This review, which was one of the catalysts for this study detailed in Chapter One, radically overhauled the fourteen year-old ANUSET model. As a result, a new Student Evaluation of Learning and Teaching (SELT) system was introduced in 2009. It was broadly based on two online student questionnaires:
a) a compulsory Student Evaluation of Learningquestionnaire, largely modelled on the national CEQ survey. Its outcomes were to be made public within the
university and subject to comparative analysis against other outcomes in the university
b) a voluntary Student Evaluation of Teaching,whose data remained private unless agreement was given for its release for such things as performance management, promotion or teaching grants and awards
The comparative ANUSET and SELT questionnaires are compared in Appendix One. This comparison demonstrates the retention of a dual motive in the new system with the collection of teacher-only student feedback data on teaching the objective of quality improvement, as well as for the first time (internally) public feedback data on student opinions on the affordances and constraints to their learning, orientated to quality assurance. This latter questionnaire replaced a previously private series of ANUSET course questionnaires aimed at particular forms of teaching groups and a stand-alone open-ended answer questionnaire. This change was also accompanied by a new policy framework around student feedback, which introduced for the first time institutional requirements for reporting to Academic Board where numeric averages were not achieved on the public forms of data (this policy is included at Appendix Two).
Although this new policy framework (further revised in 2013) noted for the first time that student feedback was for both quality assurance and quality improvement purposes, it significantly required ANU Colleges to formally report to the University Education Committee where ‘overall satisfaction agreement level (is) below 50%’, outlining the ‘specific actions and timeframes to improve the student experience’ (ANU, 2013, p. 2). It is also notable that shortly after the introduction of the new SELT system in 2009, the administration and support for the student feedback system was moved to the ANU Statistical Services Unit, from its home of the preceding three decades in ANU research and academic development centres (ORAM and CEDAM).