Scholars have long pointed out the inherent flaws and potential biases of student evaluation systems — and the fact that these scores can possibly unfairly determine whether faculty members receive tenure or a raise. While these long-used assessment metrics were created with the intention of improving instruction, many institutions are starting to recognize that these largely opinion-driven standards do little to inform administrators on how well a faculty member is performing, which can be negative for the integrity of the business model.
That's why the University of Oregon decided to turn the evaluation system on its head, with administrators developing an evaluation system that would rely more heavily on qualitative measurement, rather than having students rate instructors with arbitrary numbers.
The problem: Biased performance metrics based on opinions
Sanjay Srivastava, a psychology professor at the University of Oregon, was on the original task force working on the new evaluation system.
"As a psychologists, other experts look at my work and form expert opinions. Of course, that’s open to bias, but if the institution cares, you can mitigate bias," he said. "With the typical quantitative system, the numbers have the appearance of being objective and precise, and so people latch on to them. Then the number is the thing you care about, and the expert judgment takes a backseat, and often times with people not taking time to ask about the validity of those judgments."
Where this becomes an issue, is if students perhaps use evaluation systems in a retaliatory manner. Sierra Dawson, associate vice provost for academic affairs at the University of Oregon, has been working extensively on this project. She said the traditional evaluation system can be a problem because "you can have a professor who is a very knowledgeable person who doesn’t construct a very good learning environment, or you can have one that's organized but is not putting together a very effective learning environment."
The lack of appropriate insight means a professor could persist in a classroom to the detriment of students' outcomes, if his or her teaching has not been measured appropriately. The University of Oregon's website on evaluations is pretty transparent about this, noting, "studies show that students’ unconscious and implicit biases generally cause women and minorities to have lower evaluation scores than their white male counterparts, and that the level of the course and other factors may also affect the results."
The innovation: A comprehensive, qualitative evaluation process
"In some ways, what the University of Oregon ... saying is we’re going to evaluate the work of teaching based on the way we are evaluating other parts of scholarship," Srivastava said. "That itself can be done well or poorly and that’s also under development." But, he added, "it’s a disruption because we are actually returning to the way academia has historically worked."
The University of Southern California is moving away from this type of system in favor of a peer-review system. But certainly, the mindset shift around how evaluations must be done is new and necessary, said Dawson, who added that the current way of assessing teacher performance is lacking in evidence and transparency.
The system the University of Oregon is trying to adopt is intended to be more comprehensive. "We found the questions that have been used here for a long time weren’t very compelling and weren’t getting to what we know forms scholars for teaching and learning," said Dawson. "We are asking deeper questions that are based in the scholarship."
The university's proposed evaluation system also includes reflections from professors. "These reflection questions help faculty put their own voice, decision making and thought process right into the hands of the person reviewing," she said. "Often the reviewer doesn't have enough information."
The goal: A holistic teaching performance evaluation system
"Whereas most institutions ask students more open-ended questions that don’t necessarily get to the core of that, we are trying to de-risk bias," Dawson. "We are creating a system through which we can provide feedback that is relevant to teaching excellence."
Dawson said the goal is to produce engaged teaching and learning, as well as professionalism within the craft. Through the new valuation system, she said, the university will provide data sources and examples of teaching behaviors so someone can see whether those behaviors were happening or not. These factors are critical, she said, otherwise "how would we know someone is meeting expectations, not meeting expectations or exceeding expectations."