I have been wondering lately about teaching evaluations: how they are best structured and analyzed, disseminated, and used to make decisions, and, in the larger scheme, how differing interests should be weighed as we address these issues. I have no answers, but I have a lot of questions (they follow after the jump).
I would love to hear people’s thoughts on the answers to these questions, or suggestions for more questions to add to the list. Also, I’m sure there has been a tremendous amount of research on all of these subjects, but unfortunately I’m entirely ignorant of it, so among other comments, I’d be very curious if anyone had particular reading they would recommend on these subjects. It would also be great to hear how other law schools approach these issues now, and how other law schools arrived at their decisions about to address these issues.
Creating and Understanding Evaluations
What questions should be on teaching evaluations, and how should they be framed?
If we ask students to evaluate teachers on a numerical scale, what specific statistical concerns should we take into account when analyzing the raw evaluations numbers? For example, it has long been noted that there are problems with averaging ordinals. That is, on a scale of five (where five is the best), there may be a bigger gap in quality in the minds of students between someone who earns a four and someone who earns a three than between someone who earns a five and someone who earns a four. But if we simply average numerical scores, these differences have the same effect on the average, which is not good. (Side note: It makes me incredibly happy that there is a journal entitled Quality and Quantity. Yes please!) Or, to give another example, is there some particular way we should treat outliers when analyzing evaluations? (The answer might be no.)
Should teaching evaluations be released publicly?
If so, which portions? The entire evaluation? Raw numbers? Averages? Written comments?
And if some or all of this information should be released, how should it be released? On a website available to the general public? On a website available only to members of the law school or university community? Or should it not be available electronically at all—for example, should the information be available only in, say, a book that sits in the registrar’s office? Or, on the other extreme, should the information be released in a downloadable spreadsheet, to make analysis easier?
Finally, how long should the public portions remain public? Permanently? Or for some limited amount of time?
How should evaluations be used in hiring and tenure and promotion decisions?
How should internal law-school evaluations be compared to external evaluations (e.g., when hiring lateral candidates)?
Do student evaluations show consistent bias against teachers of a particular gender, race, age, attractiveness, ability/disability, or any combination thereof? (I.e., do teachers with particular characteristics that seem not intrinsically related to teaching ability receive consistently lower numerical evaluations or consistently more negative comments?) How, if at all, should the use of evaluations for hiring or tenure and promotion decisions take this bias (if it exists) into account? (And should the answer to the bias question affect any questions in any of the other categories?)
Finally, the most important question of all: When making these decisions, how should we compare and weight net utility within and among affected groups, including students, faculty, and the institution of the law school itself?