« The Best Commentary on that Idiotic New Yorker Cover. . . | Main | A Model Blog Operating Agreement? »
July 16, 2008
Teaching Evaluations
I have been wondering lately about teaching evaluations: how they are best structured and analyzed, disseminated, and used to make decisions, and, in the larger scheme, how differing interests should be weighed as we address these issues. I have no answers, but I have a lot of questions (they follow after the jump).
I would love to hear people’s thoughts on the answers to these questions, or suggestions for more questions to add to the list. Also, I’m sure there has been a tremendous amount of research on all of these subjects, but unfortunately I’m entirely ignorant of it, so among other comments, I’d be very curious if anyone had particular reading they would recommend on these subjects. It would also be great to hear how other law schools approach these issues now, and how other law schools arrived at their decisions about to address these issues.
Creating and Understanding Evaluations
What questions should be on teaching evaluations, and how should they be framed?
If we ask students to evaluate teachers on a numerical scale, what specific statistical concerns should we take into account when analyzing the raw evaluations numbers? For example, it has long been noted that there are problems with averaging ordinals. That is, on a scale of five (where five is the best), there may be a bigger gap in quality in the minds of students between someone who earns a four and someone who earns a three than between someone who earns a five and someone who earns a four. But if we simply average numerical scores, these differences have the same effect on the average, which is not good. (Side note: It makes me incredibly happy that there is a journal entitled Quality and Quantity. Yes please!) Or, to give another example, is there some particular way we should treat outliers when analyzing evaluations? (The answer might be no.)
Disseminating Evaluations
Should teaching evaluations be released publicly?
If so, which portions? The entire evaluation? Raw numbers? Averages? Written comments?
And if some or all of this information should be released, how should it be released? On a website available to the general public? On a website available only to members of the law school or university community? Or should it not be available electronically at all—for example, should the information be available only in, say, a book that sits in the registrar’s office? Or, on the other extreme, should the information be released in a downloadable spreadsheet, to make analysis easier?
Finally, how long should the public portions remain public? Permanently? Or for some limited amount of time?
Using Evaluations
How should evaluations be used in hiring and tenure and promotion decisions?
How should internal law-school evaluations be compared to external evaluations (e.g., when hiring lateral candidates)?
Do student evaluations show consistent bias against teachers of a particular gender, race, age, attractiveness, ability/disability, or any combination thereof? (I.e., do teachers with particular characteristics that seem not intrinsically related to teaching ability receive consistently lower numerical evaluations or consistently more negative comments?) How, if at all, should the use of evaluations for hiring or tenure and promotion decisions take this bias (if it exists) into account? (And should the answer to the bias question affect any questions in any of the other categories?)
Weighing Interests
Finally, the most important question of all: When making these decisions, how should we compare and weight net utility within and among affected groups, including students, faculty, and the institution of the law school itself?
Posted by Sarah Lawsky at July 16, 2008 09:14 AM
Trackback Pings
TrackBack URL for this entry:
http://www.concurringopinions.com/movabletype/mt-tb.cgi/3895.
Comments
To me, the overriding principle is to have a clear understanding of who the intended audience for an instrument is. Is this a tool for the school to evaluate professors, for students to learn about professors they might take classes from, or for professors to improve their teaching? The more of these bases an instrument tries to cover, the less well it'll do at any of them. For that reason, I can't recommend highly enough the practice of using different evaluations for these different purposes. Three may be overkill, but I've found that two is quite reasonable:
We hand out course evaluation forms near the end of the semester. They're good, but they've got some, perhaps unavoidable, issues: First, by asking for numerical ratings, they put the class in an evaluative frame of mind: this was good, that was bad, let's play Simon Cowell for a few minutes. The qualitative responses end up being a little thin. Second, due to the timing, it becomes impossible to ask questions about the exam, even though the exam is the focal point of the course (due to the cockamamie law-school grading system).
I get around these issues by also sending my students a SurveyMonkey poll a little after the grades are in. The timing lets me ask things like whether the exam was fair. And the less evaluative nature of the instrument means I get longer, more helpful responses to questions like "What topics do you feel could have been taught better?" The answers have been very useful. For example, in my Copyright class, I'm going to completely revise the way I teach substantial similarity, My SurveyMonkey told me pretty strongly that that week was a disaster, a fact that was hidden in the noise of lesser concerns on the official school evaluation.
Posted by: James Grimmelmann at July 16, 2008 09:52 AM
James, if it makes you feel any better, I think most copyright professors feel like they chunk the "substantial similarity" section. The law is way too squirrelly for anyone to teach it cleanly.
Sarah, this is a terrific topic. Let me offer up a couple of marginally useful generalities:
* there is an enormous amount of scholarly research on teaching evaluations--probably thousands of articles
* at the law schools I've been affiliated with, precisely ZERO of the scholarly research is consulted in the design and implementation of teaching evaluations.
I can't solve your broad problem, but I've repeatedly blogged on literature about teaching evaluations, both in the law school context and beyond:
http://blog.ericgoldman.org/personal/archives/2008/05/arthur_best_on.html
http://blog.ericgoldman.org/personal/archives/2007/03/merritt_on_teac.html
http://blog.ericgoldman.org/personal/archives/2007/05/student_evaluat.html
http://blog.ericgoldman.org/personal/archives/2006/08/sexy_professors.html
http://blog.ericgoldman.org/personal/archives/2006/03/i_need_to_get_t.html
http://blog.ericgoldman.org/personal/archives/2007/04/law_professor_t.html
http://blog.ericgoldman.org/personal/archives/2005/11/are_you_hot_or.html
Regards, Eric.
Posted by: Eric Goldman at July 16, 2008 11:03 AM
"Should teaching evaluations be released publicly?"
Yes, partially. The results (but not the individual survey responses) should be released to students, at a minimum.
Students want information about professors—who is a good lecturer, who isn't, who's a tough grader, who they'll learn from, who is old-school Socratic and who is more laid back, and so on. In the absence of official evaluations, students go to outside sources of information. Other students. Ratemyprofessors.com. Internet message boards. Bad information sources, most of them, but if that's all a student can get, that's where she'll go.
Course evaluations aren't perfect, but they're better than the available alternatives. An official report on student satisfaction with courses and professors is the best way to deal with sites where professors are rated by a handful of self-selected students.
Posted by: Jim G at July 16, 2008 01:06 PM
There is a lot of literature on evaluations. One of my favorites is this relatively recent one:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=963196&high=%20Deborah%20Merritt
Posted by: Anon. at July 17, 2008 10:44 AM
Professor Merritt's Article (referenced above) appears in the St. John's Law Review:
82 St. John's L. Rev. 235 (2008).
Posted by: anon. at July 17, 2008 10:19 PM
