On teaching evals

Ms Choi (I believe is her name) needs to be reminded (at Harvard, even) that the plural of anecdote is not data. In particular, numerical eaching evaluations are a joke, and yield nearly no useful information to anyone. Any faculty thinking of adopting such systems should also prepare to pander to their students in order to bring their evals up.

Tom in Raleigh, at 8:45 pm EDT on July 2, 2008

There is some value

The problems of popularity and content are quite real. My evaluations tend to be better when I teach higher level courses, but that may just as easily be the maturity of students who have had to pass required courses (and thus spent, in theory, more time in the college classroom) as it might any other factor.

Another key element in student evaluations is the content. I have seen anywhere from four to fifteen questions on student evaluation forms, and I have a preference for the longer ones. Students have to respond to more specific questions, often giving those who review the results, including me, much more detailed information about areas that could use improvement.

Maybe it’s time to drain the tub a bit and add a little hot water, but the baby seems otherwise reasonably healthy.

Andrew Purvis, at 7:00 am EDT on May 4, 2006

I am pleased that the faculty at Harvard are concened about student evaluatons of faculty, since they are often misused and overused. But they have been shown to be reliable and valid indicators of one’s teaching from the perspective of the students. To me the intended use of the evidence is the important factor to consider. When I make presentations on the utility of student ratings, I like to start with a summary of the research on excellent teachers: “Effective teachers, identified by both students and faculty peers, have a hardness of the head and a softness of the heart.” And it is important that is is not the other way around.

Larry Braskamp, Professor emeritus at Loyola University Chicago, at 7:50 am EDT on May 4, 2006

The issue concerning the appropriateness or value of student evaluations of teaching performance in postsecondary education is likely to continue to be debated with good arguments on both sides. My observation is that using an instrument that provides opportunity for standard general information as well as discipline or course specific items can be useful as one component in the evaluation process. The best indicators can include, in addition to appropriately constructed student survey instruments, a) peer observations, b) focus group discussions, c) information from exit interviews by faculty teams or department chairs/deans, and d) other anecdotal types of documentation such as student or former student letters, notes from students, samples of innovative instructional materials, records of student performance on licensure or certification examinations, teaching awards, or recognition by institutional, student or professional groups. If accountability is to be considered a factor and if teaching performance is valued, then it is reasonable to expect there to be multiple forms or tools for evaluation. The tools or methods can be determined on a departmental, college, and institutional level but unless accountability is to be unmeasured, use of multiple forms of evidence can be a valuable tool as both a basis for monitoring improvement of teaching as well as overall indicators of teaching performance.

For an evaluation process to work, there needs to be an open discussion and collaborative plan developed that includes faculty and administrative input using multiple indicators as well as specific consideration given to discipline or situational circumstances created by teaching large sections, required courses, grade distribution effects, and other factors that often influence student feedback. There will be no perfect system or combination of evaluative tools but there is a body of information to reflect that teaching effectiveness can be measured and from my perspective, should be evaluated using multiple indicators.

Tom Rakes, Provost and Vice Chancellor for Academic Affairs at University of Tennessee at Martin, at 8:50 am EDT on May 4, 2006

And what are we evaluating?

Yes, student evaluations are great predictors about what students thought in the first place. I once heard a report of a study that took close to 1000 course evaluations and actually split the evaluations between beginning of class and the end of the class. Guess what, the student expectations at the beginning of class predicted the student evaluations at the end to a very high level of significance.

If we want to evaluate teaching, then we should use indicators appropriate to the purpose in the first place. What should count is what is learned and how it is used, not how students feel about the teacher. Unwilling to check on a student five years down the road, we should call our current system instructional performance assessments. Then we could have questions like, “Was the instructor entertaining?” (I really meant performance in the theatrical sense) and “Did you get the ‘A’ you paid for??” Believe it or not, students will answer these questions just as honestly as they answer the question, “Was the instructor knowledgeable about his or her field?” I have few students who could answer this one fairly, yet it is a very common question on these “evaluations.” Worse yet, these are often anonymous. No corporate employer would dare use an anonymous evaluation to determine an employee’s retention, termination, or salary increments. What ever became of that mantra, education should be more like a business?

mdg, at 9:50 am EDT on May 4, 2006

This was good for a laugh:

“Irene Choi, a Harvard psychology student and member of the Undergraduate Council, said that she understands that faculty members are concerned that students who do poorly in a class might lay into a teacher in their evaluation.”

What is poorly? Getting a “B+” ? Since 2/3 of undergrads leave with some type of honors from Harvard, that would hardly hurt any instructor.

ppaul, ppaul, at 10:15 am EDT on May 4, 2006

Most “evaluations” are just smile sheets

I’m with mdg: what are we evaluating? Most “student evaluations” are what trainers refer to as “smile sheets” or Level 1 of Kirkpatrick’s oft-quoted rubric (see e.g. http://coe.sdsu.edu/eet/articles/k4levels/index.htm). They can only report on students’ reactions to the course/instructor. They can’t measure actual instructional effectiveness.

And as others have stated here, there are several issues in the process — which should be well known to positivists: - Does the instrument measure what we think it is measuring?- If so, how reliable is the instrument?

Level 1 student evaluation smilesheets can be very useful, as long as we (and administrators, trustees, legislators, and the public) understand their limitations. Unfortunately, the popular perception of standardized testing in K-12 does not make one optimistic about this.

Joe Clark, at 10:50 am EDT on May 4, 2006

What A Bunch Of Wimps!

I am sooooo tired of you academic girly-men ... whine, whine, whine. You’re worse than my five-year-old grandson.

I taught my first college class more than thirty years ago, I had spectacular student evaluations then, and I have continued apace ever since. If you’d like to learn a bit about my instructional techniques and exploits – and I don’t teach wussy courses, I teach mathematics to “soft” science wannabes – check out the following Uniform Resource Locator ...

http://www.creationismstrojanhorse.com/Trojan_Horse_Intro.pdf

Hoping to be your mentor, I remain ...

Dr. Fox, at 12:00 pm EDT on May 4, 2006

Evals

They can pinpoint glaring problems with a professor. But some of the best “teaching” is giving students awkward, open-ended problems that don’t lend themselves to cookie-cutter answers. Students HATE those questions. They make them uncomfortable. But they learn the MOST when they are forced out of their comfort zone.

Professors who force their students to stretch are often misunderstood and generally feared and hated. It might take years for the student to realize that the professor who NEVER let them enjoy a comfort zone were the ones who taught them the most.

Having said all this, I generally get solid evaluations in most of the GE Math courses I teach. But at the service course level and even the upper division level, students who lack some of the tools needed (as spelled out in the prerequisites for the course) tend to blame the instructor for simply upholding standards. As a rule, the better the student, the better the evaluations. I can give identical presentations and assignments to two different sections and one will think I’m great and the other will think I’m from another planet. The strongest correlate to high evals is high incoming student aptitude and work ethic.

Harry Mills, at 12:00 pm EDT on May 4, 2006

Job

Its part of your job to teach. I can’t think of any job where the quality of service (or product) doesn’t matter.

As for those with bad grades, my school has a box to check current grade in course and current overall GPA, and can then adjust the importance they give to various comments accordingly.

Kevin, Undergraduate, at 12:50 pm EDT on May 4, 2006

One question I would like to see added to all student evaluation forms is: “What grade are you currently receiving in this class?”

In my experience, students receiving failing grades uniformly answered evaluation questions negatively, in some cases as negatively as possible. Students receiving a’s and b’s gave a’s and b’s. The tougher the grades I gave, the tougher the “grades” I received on our evaluation forms. But there has always been direct correlation between what grades they received and what evaluation I received.

I especially agree with the previous posts about student anonymity. This doesn’t work because there are no consequences for that small segment of the population that responds inappropriately and abusively. I believe that student evaluations can be extremely helpful, and have modified my teaching style because of thoughtful comments received. Students should have to be identifiable and accountable for their evaluations after the course’s final grades have been issued, just as professors must back their grades.

grad03, at 2:15 pm EDT on May 4, 2006

Kevin, Individuals grades and overall GPA are a function of how well the student can interact with the school: both in terms of convincing teachers to give (deserved or undeserved) higher grades, or simply not taking challenging courses. There is a proud tradition amongst many (including some of your current teachers) of raising GPAs by taking easy courses and dropping harder ones. I am sure that you know this, and therefore, I wonder why you think that lower GPAs should result in evaluations being taken more seriously.

But, here is a better solution. Videotape and transcribe randomly selected courses. Sure, this might interfere with the “sovereignty” of the classroom, but who cares? (Classrooms are not countries, and local, state, and federal laws generally apply, anyway.) Then, have the courses reviewed by 1) faculty at that school; and 2) faculty at other school; and 3) faculty in other disciplines. If necessary the professor’s face can be pixilated.

These inside and outside reviewers would not have a stake in the outcome of the evaluation, and would presumably be more mature than undergrads. The reviewers would also have some idea about what goes on in other disciplines.

Larry, at 2:15 pm EDT on May 4, 2006

Ask the right questions!

End of semester evaluations by students that inform instructors of how much the student thinks he/she made progress ON THE IMPORTANT AND ESSENTIAL goals of the class are very useful. They help instructors gain feedback to evaluate their own focus on goals THEY SET for the class.

Furthermore, once selected class goals are aligned with specific instructional methods (there is much educational research on this!), feedback from students about how much they observed these specific behaviors can help instructors make changes.

We use the IDEA Center evaluations and they supply USEFUL information, not just the old “I like...I didn’t like...” Many of our best professors are requesting use of this in all classes they instruct. Check out www.IDEA.ksu.edu for some no-nonsense, useful evaluations.

You have to ask the right questions, colleagues!

Randi Hagen, at 3:45 pm EDT on May 4, 2006

Face it, student ratings of instruction are not going to disappear from Higher Education any time soon. They have been around since the early 1960’s and research shows they are used by most accredited institutions today.

Certainly, the effectiveness of student ratings are directly tied to the quality of the evaluation system itself. As well, none of us would expect a freshman undergrad student to provide expert evaluation of a Ph.D. tenured faculty’s content knowledge of chemistry. At the same time though, students can provide valid and reliable reactions about their learning experience – and I’ll tell you a secret, this can impact learning. For example, a student’s evaluation of the instructor’s ability to clearly present and communicate information (in the language used in the course) can certainly impact a student’s ability to understand what the instructor is saying, and in turn impact student learning. There are numerous well studied dimensions of a student’s learning experience that can provide valuable data to faculty for self-improvement and to administration to manage faculty performance.

There is nothing wrong with a well-developed, well-intentioned student rating of instruction process – especially if it’s used along with other assessment and evaluation methods.

Moreover, take a moment and look at the for-profit higher education institutions. Why have they become such a significant threat to traditional higher education? Why do the growth rates of their enrollments continue to blow away those of traditional institutions? Perhaps it’s their focus on the student, student learning, and the student’s learning experience – and less on the “intrusion on the sovereignty of the classroom”.

University Admin, at 6:00 pm EDT on May 4, 2006

We sure can get rid of course evals if we want

Face it, student ratings of instruction are not going to disappear from Higher Education any time soon. They have been around since the early 1960’s

By the same reasoning we should have faced it and been “realistic” about the impossibility of getting rid of smallpox, Jim Crow, and various other ills.

Faculty have always been evaluated by their peers—sitting in on classes, assessing course materials and so on. Perfectly legitimate.

Students are not in a good position to evaluate faculty: they don’t know the field—that’s why they’re students. Poor students are likely to blame faculty for their performance. And in the aggregate students tend to be highly conventional, much more concerned with appearance, manner and personal charm than faculty and, because they aren’t accountable base their judgements in such soft areas as “ability to communicate” on these factors—and (I’ve gotten this) such things as instructor’s proclivity for using “big words.” Are you suggesting, Admin, that we tailor our style, and our wardrobes, to suit student preferences, avoid big words that they don’t know, big concepts that they don’t understand, controversial issues that they don’t want to deal with and play to the gallery?

But here’s another question for you: what would happen if we faculty just said no—simply voted to stop administering course evaluations? Would students take their marbles and go home—transfer to universities where they got to do evaluations? Would teaching suddenly degrade? Seems things were going ok before course evals were instituted 40 years ago. Oh, incidentally, I just went to a workshop on grade inflation and it seems grades started going up at exactly the time that course evaluations started becoming popular.

If we chose, if we got together and exhibited some solidarity here, we could jolly well vote to stop the course eval business. So, what would happen? Let’s talk turkey.

LogicGuru, at 9:15 pm EDT on May 4, 2006

Correlation v. Causation

Let’s ignore the specious analogy of course evaluations to “smallpox, Jim Crow, and various other ills,” all of which had demonstrable and documented negative effects. What concerns me here about LogicGuru’s comments, particularly under the banner of that name, is the point that grade inflation and course evaluation began at “exactly the time.”

I would like to see this evidence, including the proof that they correlate in place as well as time (if school A begins course evaluations, and school B, 800 miles away, demonstrates grade inflation without evaluations, it doesn’t help this insinuation much). I am not saying that the evidence does not exist for causation, only that people commenting here, on this and other pieces, have yet to bring evidence of causation.

If we’re going to have someone using “logic” in the name, let’s see some logos, please.

Andrew Purvis, at 4:35 am EDT on May 5, 2006

Evals

20 pages of writing is not really a whole lot for a one-semester course, ML. 100 pages of reading may or may not be a whole lot. It depends on the content density and complexity. Certainly 100 pages of mathematics is more than 100 pages of history.

And I repeat: There is a very strong correlation between better students and better evaluations, especially when the content matter is something to which students are generally averse (such as mathematics).

I think evaluations are good for pinpointing serious problems, but peer review should be expanded, and student evaluations of teachers should not play as significant a role as it does.

I believe that there is not enough peer review and too much student review. I talk to a lot of students outside of class, and the things they say about my own peers tells me a lot about the disconnect between student perceptions and faculty actions. I KNOW what Professor X is driving at in Abstract Algebra, and I know Prof. X is doing a solid job of it. The disaffection among her students is more a product of their growing pains as they are forced to construct logical arguments in order to earn a decent grade. Is it X’s fault they’re struggling? Nope. Are these students likely to give X poor evaluations? Yup.

Harry Mills, at 11:00 am EDT on May 5, 2006

research, anyone?

I’m always amazed by how many faculty are willing to self-servingly base their arguments in favor of student evaluations on nothing more than their own anecdotal experiences—"I have good evals despite the fact that I assign 20 papers, etc, etc.” —even when they come from disciplines that require at least a modicum of quantitative literacy from its students. There’s a great deal of research that shows teaching evaluations are very strongly shaped by judgements of a professors attractiveness, nonverbal style, and other things that have nothing to do with teaching effectiveness. For a review of some of this, see: http://home.sprynet.com/~owl1/sef.htm. (For a more cynical and entertaining review and instructions on how to use the research to manipulate your student evaluations scores, see: http://mtprof.msun.edu/Fall1997/HOWTORAI.html). See also Hamermesh and Parker’s demonstration of the correlation between attractiveness and eval scores in the Economics of Education Review 24 (2005) 369-376. Also, I’ve heard there’s an entire issue of American Psychologist from a few years back that demonstrates the correlation betwen grade expectations and evaluation scores. No wonder that the self-congratulatory teachers who support student evaluations never support their anecdotes with any evidence that their students leave class knowing any more than they would have if they’d simply stayed home or taken the class from a different professor. Aren’t such measures far more adequate measures of effective teaching? Perhaps not if what you are trying to effect is the creation of an amused and sedate audience who strokes your ego. Sad that it seems to have come to that for so many in academe. Sad, because the direct effect of this is extreme grade inflation and the concommitant watering down of higher education—see Valen Johnson’s “Grade Inflation: A Crisis in College Education” (Springer, 2003).

ex-prof, at 12:15 pm EDT on May 5, 2006

Let’s Talk About Variance

Three things ... and I’m assuming I’m addressing this post to individuals who are truly concerned about both teaching excellence in general and their own instructional expertise.

First, I want to emphasize a point not made strongly enough in my earlier post; to wit, there is so much common-cause variation in the processes we attempt to measure with student evaluations, it would be a miracle if they revealed anything meaningful that would be of interest to anyone other than the individual teacher. Differences in academic discipline, level of the course, targeted audience, previous academic experience of the students, instructional objectives, teacher’s personality and instructional style, and the teacher’s experience constitute but the tip of the iceberg of evaluative relevance.

Second, the enormously important factor left out of the list above—I suppose it’s part of the iceberg below the water line — is the culture of learning that defines the college or university. My most recent faculty position was at a university whose culture for learning was an embarrassment to the concept of “higher” education. To have high expectations for students’ performance put a faculty member in such a difficult position vis-a-vis student evaluation it almost defied imagination. Students were used to being “passed through” and any teacher whose expectations deviated from that had to overcome almost insurmountable barriers just to be taken seriously to begin with. After a couple of years spent establishing a reputation as a teacher with specific, if minimal, expectations, I got terrific student evaluations. By then, the VPAA had targeted me as an insensitive professor. May I repeat that ... a teacher who had high expectations for his students’ performance, worked with them to accomplish his objectives, and had high (4.5/5.0) evaluations was considered to be INSENSITIVE TO STUDENTS’ NEEDS AND EXPECTATIONS.

Granted that some evaluative strategies and tools are much, much better than others, anyone who thinks even the best of those tools – and let’s focus attention on only the measurement instruments for the moment – captures anything more than a tiny fraction of the variation in “teaching ability” is ... well, a potential vice president for academic affairs. In the hands of academic administration (and there’s a lot of variation there too), the results of these quite awful evaluative strategies and measurement tools is something no conscientious teacher – and especially no young assistant professor aspiring to be an excellent teacher – should have to endure.

Third – and this is especially for Harry Mills and Mike – I have a very close friend who is on the mathematics faculty at a “better than average” liberal arts college in the Midwest. At one time we were colleagues, so you will have to take my word for the fact that he is a superb teacher (I would have loved for either of my youngsters to take more than a few of his courses). In any event, the culture for learning at his college was such that he was soon seen to be an instructional outlier (his expectations were too high) and his student evaluations reflected that.

After a couple of years on the faculty, he took matters into his own hands. He (1) dumbed down his courses, (2) greatly lowered his expectations, (3) greatly decreased the amount of out-of-class work expected of students, and (4) significantly decreased the difficulty of his assignments and tests. Within a couple of years, his student evaluations were the envy of his colleagues and he is now a tenured full professor. It is noteworthy that he has not changed the instructional strategy that got him there. But – and this is my point – he is confident he decreased the quality of his courses and his teaching in order to “accomplish” promotion and tenure. Talk about “teaching to the test.”

Is his behavior unethical? Perhaps. But any college or university professor who has been in that situation – and I have — knows, it is virtually impossible for an individual professor to change the culture for learning that defines one’s academic institution.

By the way, more and more these days we see academic mush-heads referring to students as customers. One might argue, from Ed Deming’s perspective, that my friend was merely satisfying his customers’ needs and expectations.

RWH, at 3:10 pm EDT on May 5, 2006

John Anderson,

I think it is pretty clear the numerical averages student evaluations are used in making tenure decisions and for calculating some part of annual merit pay raises. That sets the atmosphere.

Although there is wide spread concern that even well designed evaluations produce unintended consequences (and many evaluation forms are not well designed), administrators want numbers that are quick and easy to use.

In mathematics it has long been known that there are sets that cannot be measured and spaces were distance cannot be defined.

Mike, at 5:30 am EDT on May 6, 2006

Evals, Self esteem and Academic atmosphere

Some of the best ideas on evals, in one place. And 360 degree range— Only with what I must add !! ( Otherwise, all that is said above becomes of little value)

There are two kinds of everything in human affairs: coming from good idea; or coming from bad idea; Evals are no exception to this Axiom of academe.

Now, if administration & students & faculty want to play games in the name of education, they’ll find ways—for sure, God gives that much enterprise to lowest among us, what to say of the brighter ones!!

IT SURELY IS A PITY WHAT EVALS DO TODAY. Only, it is just the SYMPTOM, not the DESEASE. And calls for a radical stand to start cleaning of the entire education system. Nothing short of a movement ushering renaissance in higher education will meet the challenge.

Wiseman, Dr, at 6:10 am EDT on May 10, 2006

I have newly entered teaching in the university environment, and I was shocked at the comments in the student evaluations. I was (silly me) excited to see what I could learn from my students through their evaluations. I did not expect glowing praise — I have much to learn — but some of the comments were, as some above have mentioned, angry ventings. I have alternated between trying to see what information I can salvage from their ‘essays’ and thinking that I should never teach again! The thing is, I have instructed in other environments to receptive and thoughtful responses. People who value their learning experience invest more in the process and are willing to take some responsibility for themselves, both in terms of behaviour towards other people and the effort they put into their studies, critical thinking and reflection on the subject material. Regarding the students at the “working class” school mentioned in an above comment, you can be damn sure that those students are there because they want to be there and have taken time out from their jobs and family time to learn. In comparison, most of my students have come from wealthy backgrounds — as tuition rises, so does the economic situation of the average student. I say this based on informal conversations about jobs (they don’t need them), the expensive clothing they wear, missed classes due to vacations to exotic locales mid-semester, and the average age of my students. Their parents are paying and many (not all, but many) of these students see education as a commodity. A colleague had a student who was shocked to fail a test because she had “paid for the course.” I contacted some of my most beloved professors, now retired, to ask their advice about my teaching skills and what I can only describe as mean-spirited evaluations. I have heard back from several of them that in their final years of teaching, they stopped reading evaluations. Countless awards, respect from colleagues and a long history of students who idolized them faded as they were left with a new generation of students who were only at university to buy a diploma. Evaluations for instructors who care about sharing knowledge, who seek answers to tough social questions, who think learning is a right that carries responsibility— these kinds of professors seem to be the ones who do not fare well in such an environment. I have decided to carry on teaching courses modeled after the ones that shaped my academic perspective and changed my life. I will put that much more care into teaching for the students who want to be there and spend less time worrying about the ones who don’t, or what they might say on their evaluations.

new instructor, at 5:40 am EDT on May 27, 2006