Research on student evaluations identifies several recommendations to ensure that course evaluations can provide valid data for formative and summative evaluation of teaching:
Set clear evaluation goals, including clear definitions of what constitutes effective teaching at your institution and ensure that questions reflect these goals
Section 4.C.i: Defining Effective Teaching describes the importance of ensuring that evaluation questions match institutional teaching priorities and provide adequate information to make the kinds of summative assessments for which the instruments are being used. The identification of teaching measures to be evaluated and the development of evaluation questions should be viewed as an opportunity to encourage an institution-wide discussion about teaching goals and evaluation uses. To ensure that questions can provide meaningful feedback to instructors and can be used in the summative evaluation of teaching, the questions that are ultimately selected should measure aspects of teaching that reflect these conclusions.
Design and test instruments according to rigorous theoretical and psychometric standards The development of evaluation instruments should be a serious and substantial process involving many members of the institutional community. Questions should be selected carefully according to well-developed theoretical and research-based constructs. Scales must be logical and clearly explained. Instruments should be approved by an appropriate committee or governance body through a transparent and consultative process. Approved instruments should be evaluated by experts in survey construction and continuously investigated through institutional research (See Section 5.C.iv: Ensuring Utility for Institutions). If an institution cannot devote the time or expense to developing a rigorous in-house instrument, it may wish to consider licensing a validated instrument from another institution.
Student Course Evaluations: Research, Models and Trends
46
Establish appropriate and standardized policies and processes for the administration of courseevaluations
Clear and consistent policies and processes must be developed to ensure that the ratings collected are not compromised. This includes ensuring that:
Policy and practice about the administration of evaluations is standardized at the administrative level at which comparison between instructors or courses (if employed) is made
Many threats to validity are introduced through inconsistent administration of evaluations. This might include issues such as instructor presence during evaluations, inconsistent evaluation forms, or conducting some evaluations online and others on paper, among others. By ensuring that policies about the administration and reporting of evaluations are equitable and are applied consistently, institutions can make dramatic strides towards improving evaluation validity.
Each course achieves an appropriate response rate
Cashin (1990) recommends collecting feedback from at least 10 students and at least two-thirds of the class, whichever is higher. As described in Section 4.D.i: Reporting of Evaluation Results, to further ensure that evaluation results are representative, several scholars suggest averaging some or all of an instructor’s evaluations to ensure that the responses collected provide an accurate representation of their teaching.
The anonymity of student responses is protected
There is little data to suggest that anonymous responses are any more or less accurate or valid than non-anonymous student responses.
Wright (2006) has argued that anonymous ratings absolve students of responsibility for their statements and opinions, and that “[w]ith no possibility for follow-up, students need not think through their decision” (p. 419). Wright notes that anonymous evaluations are intended to ensure that students are not reprimanded by faculty for negative comments. However, he argues that while the intentions behind protecting student anonymity may be positive, such a system effectively places more trust in students than faculty. He further raises concerns that students may use evaluations to vent anger or disappointment regarding low grades (notably, Wright does not point to any specific studies to support this theory).
Wright also suggests that anonymity may encourage abuse of evaluation instruments and the process of administration, hypothesizing that “students could enter the room and fill out evaluations who were not even in the class” (p. 419). To address this problem, Wright
recommends that evaluations be confidential, with names stripped from the data before being viewed by faculty, so that students can be tracked by the administration to allow for follow up (e.g. to investigate an extremely high or low ranking or to identify variables that contribute to high or low rankings).
However, research does indicate that students may be uncomfortable providing non-anonymous data and that non-anonymous student responses yield somewhat higher ratings (Wachtel, 1998). Consequently, policies protecting anonymity should be applied consistently and uniformly as there is much to lose by jeopardizing the already minimal student trust of the evaluation system. Practice should also ensure that students understand that and how their anonymity will be protected.
An appropriate amount of data is distributed to appropriate populations and that appropriate and consistent policies for access to and storage of data is developed
Students, faculty and instructors each benefit from and require different data derived from course evaluations. Wachtel (1998) argues that students deserve to see the result of their input in the form of publicly distributed evaluation results. Many institutions who do share evaluation results
Student Course Evaluations: Research, Models and Trends
47
publicly (see Section 3.F.iii: Publication of Results) choose to highlight a small number of globalquestions to distribute to students to assist with course selection. A number of institutions publish evaluation results, primarily to provide students with information to assist in the course selection process. See Section 5.C.i: Ensuring Utility for Students for further recommendations about sharing results with students.
Administrators should receive appropriate individual and comparative data that matches how they will use evaluation data. Administrators who are not providing diagnostic or formative feedback may require only data from the summative global survey items (see Section 4.D.ii: Challenges to Interpretation and Use for Summative Purposes). Instructors may receive further results that can be used for formative purposes. The data that administrators receive should match their facility with statistical and data analysis. Evaluation results should be accompanied by any additional information necessary to adequately contextualize the data (for example, interpretive guides, comparative means, or written narratives by faculty members; see Section 5.C.iii: Ensuring Utility for Administrators).
Individual faculty members should have access to all course evaluation data collected about their teaching, including anonymized student written comments. Instructors should also be provided with appropriate data summaries (see Section 4.D.i: Reporting of Evaluation Results) that help to contextualize the data they receive.
Institutions should maintain centralized records of teaching evaluations (see Section 3.F.i: Who has Access? To What? for examples of how evaluation data is maintained at several institutions). Originals should be retained for a limited amount of time but long enough to verify any contested results. Processed data should be retained confidentially by departments, divisions, or in a centralized database for as long as they may be used by instructors and institutions.