News, Views and Careers for All of Higher Education
July 11
The question of how to measure the quality of college teaching continues to vex campus administrators. Teaching evaluations, on which many institutions depend for at least part of their analysis, may be overly influenced by factors such as whether students like the professors or get good grades. And objective analyses of how well students learn from certain professors are difficult because, for one, if based on a standardized test or grades, one could run into problems because professors “teach to the test.”
A new paper tries to inject some rigorous analysis into the discussion of how well students learn from their professors and how effectively student evaluations track how well students learn from individual instructors.
James West and Scott Carrell co-wrote the study, which was released by the National Bureau of Economic Research. “Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors” examines students and professors at the U.S. Air Force Academy from fall 1997 to spring 2007 to try to measure the quality of instruction.
The Air Force Academy was selected because its curricular structure avoids many of the pitfalls of traditional evaluation methods, according to the report. Because students at the Air Force Academy are randomly assigned to sections of core courses, there is no threat of the sort of “self-selection” in which students might choose to study with easier or tougher professors. “Self-selection,” the report notes, makes it difficult to measure the impact professors have on student achievement because “if better students tend to select better professors, then it is difficult to statistically separate the teacher effects from the selection effects.”
Also, professors at the academy use the same syllabus and give similar exams at about the same time. In the math department, grading is done collectively by professors, where each professor grades certain questions for all students in the course, which cuts down on the subjectivity of grading, according to the report. The students are required to take a common set of “follow-on” courses as well, in which they are also randomly assigned to professors.
The authors acknowledge that situating the study at the Air Force Academy may also raise questions of the “generalizability” of the study, given the institution’s unusual student body. “Despite the military setting, much about USAFA is comparable to broader academia,” the report asserts. It offers degrees in fields roughly similar to those of a liberal arts college, and because students are drawn from every Congressional district, they are geographically representative, the report says.
Carrell, an assistant professor economics at the University of California at Davis, attended the academy as an undergraduate and the University of Florida as a grad student, and has taught at Dartmouth as well as the Air Force Academy and Davis. “All students learn the same,” he said.
For math and science courses, students taking courses from professors with a higher “academic rank, teaching experience, and terminal degree status” tended to perform worse in the “contemporaneous” course but better in the “follow-on” courses, according to the report. This is consistent, the report asserts, with recent findings that students taught by “less academically qualified instructors” may become interested in pursuing further study in particular academic areas because they earn good grades in the initial courses, but then go on to perform poorly in later courses that depend on the knowledge gained from the initial courses.
In humanities, the report found no such link.
Carrell had a few possible explanations for why no such link existed in humanities courses. One is because professors have more “latitude” in how they grade, especially with essays. Another reason could be that later courses in humanities don’t build on earlier classes like science and math do.
One of the major points of the study was its look at the effectiveness of student evaluations. Although the evaluations can accurately predict the performance of the student in the “contemporaneous” course — the course in which the professor teaches the student — they are “very poor” predictors of the performance of a professor’s students in later, follow-up courses. Because many universities use student evaluations as a factor in decisions of promotion and tenure, this “draws into question how one should measure professor quality,” according to the report.
“It appears students reward getting higher grades,” Carrell said.
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
One of the safeguards for ensuring high quality instructors are minimum accreditation standards for faculty.
Yet these very important standards have never been fully implemented (see link below).
The same situational pressures (at least for two-year colleges) that have produced the out-of-field teaching problem at the secondary level also operate at the postsecondary level. OOF faculty are far less likely to teach effectively, and this negatively impacts students and the credibility of the administrators that consistently bend the rules for their friends.
Part of the problem in addressing this is that the “black box” of hiring practices at the institutional level is poorly understood, especially by scholars. What actually happens — how high quality instructors are hired — also deserves greater attention, in addition to the problem of the missing faculty minimum standards (HEA 1992).
Glen S. McGhee, Dir., at Florida Higher Accountability Project, at 9:05 am EDT on July 11, 2008
Well, duh.
Hoosier Prof, at 10:05 am EDT on July 11, 2008
Frizbane Manley:
I, too, am struck by the observation that science and math students taught by less well qualified professors at lower levels (seem to) do better early on, only to run into difficulty in upper level work. See my “Finally” question below.
I’m guessing this is what Prof. Ethan was quite legitimately concerned about in our earlier conversation, one based on a fundamental misunderstanding between him and me. Yet without addressing key distinctions I was trying to make, he insisted that I was professing to teach disciplines for which I was not formally trained.
Not so. I was, quite rightly, claiming not just the right, but arguing the NEED, to incorporate interdisciplinary material, NOT disciplinary methodologies per se, into literary studies: texts, literary movements, the culture which gives rise to literary texts in the first place, and the social-historical IMPLICATIONS, including the notion of competing narratives of history itself—all for student consideration, not to impose dogmatic conclusions on them.
What, Frizbane Manley, do you think, are the implications of this study where our previous discussion on teaching students to work with necessarily ill-defined problems is concerned?
By the way, I’m wondering if “poststructuralism” and “postmodernism” are fancy terms for what cognitive psychologists have called “fuzzy boundaries” amidst well-defined problems. Perhaps the whole Sokal’s Hoax episode was in part the failure of Social Text’s editors to recognize the well-defined versus ill-defined distinction.
For a further discussion (and necessarily rough draft—I’m still working on it), see
http://www.insidehighered.com/news/2008/06/26/qt
where you and I are discussing well- and ill-defined problems. I use a couple poems to illustrate why art cannot be sealed off into a realm unto itself: it’s social and historical.
Why students need to make the distinction goes right to social issues in technology: Whether we can build a nuclear reactor is a well-defined problem. Whether we ought to is ill-defined, and political. Whether our global economic institutions can maximize growth (as growth is presently defined) is one thing, whether they ought to another, and so on.
Finally, where does Teaching Evaluation stand with regard to Writing across the Curriculum?
James W. Gettys, at 10:20 am EDT on July 11, 2008
The problem here, as ever, is grades. That is why I do not give grades on papers, tests, etc. during a course (only comments) but wait until after the final examination to grade students, based on a dossier of all the written work, class discussions, attendance, etc. Course evaluations therefore precede final grades and just might be more accurate.
jay halio, professor emeritus of english at u or delaware, at 10:20 am EDT on July 11, 2008
Sorry, folks. I had meant to respond to Frizbane Manley in other of today’s articles in IHE: “Confidence Gap for New Faculty.” I hope it’s (mis)placement here, however, is not too far off topic.
James W. Gettys, at 11:20 am EDT on July 11, 2008
I have cross plotted my student ratings vs class size. There is a significant decline in my evaluations as a function of class size. For adjunct professors who teach the larger classes, this is a significant problem on our evaluations.
George Kuck, CSULB, at 12:05 pm EDT on July 11, 2008
Mr. Halio, Hopefully, your post-facto analysis of student work is conducted objectively. I assume that you have third parties scrutinize random portfolios (without knowing what grade you gave them) and those third parties come to the same or similar grades that you came to. If not, your grading system is purely subjective and you are just playing favorites.
Larry, at 4:05 pm EDT on July 11, 2008
Liping Liu
In my opinion, the present evaluation process used in the U.S. too often tends to depend on the data generated only by self-selected students’ evaluation as if it were some sort of objective truth, rather than a general indicator of student opinion and tends to be biased by the self-selection aspect. It seems that the system used in some schools in China could add an important new dimension to the processes now in use here, so as to ultimately improve the overall quality of instruction at a university. Here is what they do in China. First, students must participate in the evaluations online; otherwise they cannot take the final. The online record gives them permission to take the final. American schools do not have this requirement. Second, the students’ evaluations are only one part of the system of assessing the teacher’s teaching effectiveness. Another part is composed of retired professors from the same school. Three or four of them visit the teachers’ class, roughly twice a semester. They report their evaluation to the provost’s office, and work with those whose teaching is not effective. Since they are retired, and theoretically without any personal agenda, they are not afraid of speaking about the situation as they see it, and these evaluators are chosen because they are very concerned about education and are selected as perceived very responsible types. Therefore, a teacher’s performance is evaluated not only by his or her students but also by some retired senior academics.
Liping Liu, Dr. at CNU, at 8:15 pm EDT on July 11, 2008
I wonder if the survey examined foreign language courses as part of the “Humanities” rubric. I have noticed very similar trends to those listed as in Math/Science: lower-degreee/experience types encourage continuing study, but the students usually do less well in later courses.What courses are included under “Humanities” and are the generalizations true across the board? Or are there enough courses to judge?
LM, at 8:15 pm EDT on July 11, 2008
When a person is hired, commttees often haven’t done thorough research on the candidate to determine whether the person is qualified and whether the person can teach. They rely on resumes, letters of recommendation, and interviews.
I’ve sat on several hiring committees, and it’s seemed to me that committee members would rather not spend time hiring someone. They’re often distracted, tired, and prone to get the process over with as soon as possible. Reading resumes isn’t a favorite task either.
So we resort to consensus, hoping someone comes up with an idea of some sort. One of those ideas is to form a list of questions or topics for the interviewee. He or she gets the stuff ahead of time. To me, this is ridiculous. Giving someone questions/topics ahead of time gives that person time to go to someone for plausible answers/responses. With today’s easy internet access, a person could research the questions/topics.
But how do committees evaluate a prospective hire? We don’t know how, truth be told. It seems that because we’re teachers, we should know. But too often, the process becomes a personality contest. People are content with someone if that person is pleasing, both in appearance and demeanor. We are pleased that the interviewee can respond at length. But we aren’t sure how to evalate the response, much less whether the person actually knows what he or she is talking about. Follow up questions are few. But because the interviewees have the questions/topics in advance, they’ve been given a tremendous advantage. As a result, we can’t see how the person responds at the moment to questions about pedagogy, class management, difficult students, currency, curriculum, diversity, and administrative machinations. We hear prepared answers, answers that may have originally come from the intellectual resources of the interviewee.
On the other hand, so-called student evaluations are superficial at best, no matter how they’re conducted. To ask students whether a teacher is competent, whether that individual knows subject matter and is current and whether he or she is an effective teacher is an exercise in futility, another word for administrative non-sense.
Evaluations, in my experience, especially when it comes to part-time faculty, are essentially tools with which get rid of someone. It’s a big mistake, as well, to think administrators, often former teachers themselves, somehow become experts in evaluating teachers. A title doesn’t enhance one’s abilities, but it does enhance one’s ego.
If faculty were trained by people who have expertise in hiring, who do this as their profession, it would seem that the results would indicate that qualified and experienced people would be hired. More training would also help committees to become alert to personality quirks, to people who are “good” at interviews but not too adept at doing the job.
To ask students to evaluate faculty, no matter how it’s done, is to ask them whether they like or dis-like a teacher. Pure and simple. If they like someone, as has been pointed out, the person gets high marks and kudos from administrators. If the students don’t like someone, administrators assume that the teacher is suspect or even incompetent, all based on the subjectivity and whim of students who rush through evaluations so they can get out of class early.
Let’s keep this in mind: adminstrators are rarely competent to conduct evaluations, much less interpret them. They too are hired on whether they’re pleasing and look like they could represent the institution well on television. Board members? They are even less competent. They’re often wealthy, and they’re often business owners, which they think gives them expertise in matters of colleges and universities.
Throw the evaluations out, as they’re done now, and hire professional personnel groups to train faculty on hiring, on-going professional development, and evaluationsand retention.
Leonard Adame, Instructor, at 5:30 am EDT on July 12, 2008
I followed the link in McGhee’s comment and, even after fixing the typo, could not find any evidence that the USAFA has faculty that do not meet the minimum accreditation standards. Glen, the USAFA is not a secondary school with HS instructors teaching CC classes. You owe them an apology, Glen.
One of the math faculty at our CC once taught at the USAFA and he has the highest standards of any of our math faculty. Just as this article states, students from his class might have lower grades but they know more than the ones that come out of the classrooms of some of our new faculty with similar credentials but less experience.
What interests me the most is that the exams were graded in common in the math department. That means the differences must arise from other factors, such as extra credit work, curving grades, or tipping students off so a cram-and-forget approach will produce higher grades but less learning. That last detail, prerequisites needed for later courses, is why they don’t see it in humanities classes. No one ever asked me about Greek history in my Russian literature class.
CCPhysicist, at 11:25 am EDT on July 12, 2008
This study assumed that students tended to go on to higher level courses after having younger professors because they got better grades from younger professors. But there are other possible explanations, such as that younger professors tend to better motivate students.
John, at 9:30 am EDT on July 14, 2008
“We find that the academic rank, teaching experience, and terminal degree status of mathematics and science professors are negatively correlated with contemporaneous student achievement, but positively related to follow-on course achievement. Across all subjects, student evaluations of professors are positive predictors of contemporaneous course achievement, but are poor predictors of follow-on course achievement.” [abstract of USAFA report]
So, “contemporaneous student achievement [sic]” is low when professorial expectations are high—a differential between the two exists—and students actually benefit from those high expectations (unbeknowst to them at the time) at a later time in the technical curriculum. All of which goes to show that the faculty in technical curricula know what they are doing and focus on the longer-term curriculum while students cannot see past the current course noses (nor frequently do they want to be held accountable for actually being expected to have learned, know, or review previous course content on their own but rather have it reviewed to their blissful pleasure to the detriment of current course content; water has never wanted to flow uphill, after all). The distortion at play is that the phraseology “student achievement” isn’t achievement at all, but a euphemism that for all practical purposes refers rather to “student short-sighted satisfactions” while high expectations or being held accountable on the part of a learned and far-sighted professoriate who are trying to educate students for the future and not simply please them for the now are simply “unsatisfactory” to students. “Satisfaction” = “holding higher principles hostage to lower” = “dissatisfaction.” When will higher education wake up and stop catering to the worst of tendencies found in every person and rather take responsibility for daring to teach to the highest of possibilities—however dissatisfying that may be to those who indulge in the narcissism implied in smug “satisfaction"? Hats off to some members of the USAFA who evidently do so—against increasing pressures.
Senior Professor, Professor of Chemistry, at 5:00 am EDT on August 11, 2008
Why do bad teachers get hired and stay at major universities?
1. Teaching ability is not the major factor when hiring a professor. On the contrary, universities hire based upon the reputation of the institution the candidate earned his/her degree and their ability to publish.
2. One of the major factors in earning tenure is the ability to publish, not teach.
3. Professors are rewarded with Academic Grants, which in turn, takes them out of the classroom.
“The views expressed in this BLOG are those of the author and do not reflect the official policy or position of the Department of the Air Force, Department of Defense, or the U.S. Government.”
Richard Carrell, Major at USAF, at 3:50 pm EST on November 13, 2008
Advertisement
or search for jobs directly.
The Carroll College community invites applications and nominations for the position of Senior Vice President for Academic ... see job
Walden University is focused exclusively on providing a superior university experience for adult professionals through the ... see job
Want to redesign the way Pharmacy is practiced? Lead a multidisiplinary department dedicated to the challenge of creating the ... see job
Western Carolina University’s Hunter Library invites applications and nominations for the position of Associate Dean of ... see job
Founded in 1898, and affiliated with what is now New York-Presbyterian Hospital since 1927, Weill Cornell Medical College ... see job
Located just north of Houston, Texas, our five campuses serve 1,400 square miles. Our student enrollment is nearly 50,000 in ... see job
Saint Mary’s College of California invites applications for a full time Director of the Center for International Programs ... see job
Founded in 1898, and affiliated with what is now New York-Presbyterian Hospital since 1927, Weill Cornell Medical College ... see job
The North Carolina Center for the Advancement of Teaching (NCCAT), a center of The University of North Carolina, invites ... see job
Civil and Environmental Engineering Department Head Position Civil & Environmental Engineering Department Worcester ... see job
Want to Improve Your Evaluations by Students?
Then do as a colleague advised me thirty years ago in my first year of community college teaching of the humanities:
“Give lots of praise and lots of A’s.”
Bob Schenck, at 8:40 am EDT on July 11, 2008