News, Views and Careers for All of Higher Education
Aug. 8 Outside the Circle
You see it all the time, in the brochures and advertisements from liberal arts colleges and other non-gargantuan institutions. “Small class sizes,” they promise, and for good reason, because everyone knows that small classes are better than large. No cavernous lecture halls where the professor is little more than a distant stick figure, they say — raise your hand here, and someone will stop and listen. Plus, he or she will be a real professor, the genuine tenure-track article, not a part-timer or grad student but someone who really knows his or her stuff. Because everyone knows that real professors are better than the other kind.
Except, they don’t.
Nobody actually knows whether small classes are better than large. Pascarella and Terenzini’s How College Affects Students, the bible of such matters, says “We uncovered 10 studies that focus on the effects of class size on course learning. All of the investigations are quasi-experimental or correlational in design …. Unfortunately, five of the studies used course grade as the measure of learning … the conflicting evidence and continuing methodological problems surrounding this small body of research make it difficult to form a firm conclusion.”
The text of How College Affects Students occupies 650 pages; class size consumes just over one of them. In the long history of higher education up to 2005, when the latest edition of HCAS was published, there had never been a single truly experimental study of college class size. Not one.
Similarly, the American Association of University Professors recently released a report criticizing the big regional accreditors for failing to enforce standards related to the growing use of part-time professors, which the AAUP regards with much dismay. Like class size, the full-time / part-time issue is generally treated as a given. U.S. News & World Report uses both measures in its methodology to rank colleges, and they’re among the few such ranking measures that don’t cause college leaders to erupt in fits of consternation. Yet the AAUP report contains no references to research proving the underlying premise of full-time professorial superiority — because, I’m guessing, it more or less doesn’t exist. A few studies have examined full-time / part-time status and completion rates, but when it comes to actual student learning — basically nothing. This is not a standard of evidence that university professors would tolerate in their own research.
In other words, when it comes to the central enterprise of higher education — teaching students — we don’t know if the reigning professional qualification system works, or how many professors we actually need. And this is true for all kinds of other basic elements of college teaching and learning — curricula, training, pedagogy, and much more.
It’s not like these questions couldn’t be answered. Millions of students attend college every day in classes that vary greatly in size — much more so than in K–12 schools, where the issue has been studied exhaustively. Many college courses are taught by full-time professors, many not. It wouldn’t take the world’s greatest social scientist to design an experiment to get at what those differences mean.
To be sure, there are sui generis courses at every college that wouldn’t lend themselves to such analysis because they combine the unique perspective of a particular scholar with his or her subject expertise and time-honed approach to teaching. But that’s often not the case. Transcript studies indicate that 20 percent of all course credits earned by college graduates come in just 13 introductory courses like English Composition, Calculus, and Introduction to Economics. Seventy-one percent of all college graduates take some version of Psychology 101. Calculus is pretty much calculus wherever you go (or should be). And even in cases where curricula vary between institutions, larger universities routinely teach many sections of the same course every semester. Why doesn’t anyone ever study how much learning varies between them, and why?
In part, because that would require some kind of objective measure of how much students learn in different sections and/or institutions, i.e. some kind of standardized test. Such assessments are often considered anathema to what the AAUP somewhat breathlessly terms the “sacred principle” of academic freedom. But I don’t really understand why. There’s a great deal of logic behind academic freedom for scholarship. When scholars are bound by conventional thinking and political pressure, their research can suffer immensely. But it’s not clear why giving professors license to say and think what they like necessarily translates into an absolute license to teach how they like, at a level of quality that’s more or less up to them. Academic freedom shouldn’t immunize anyone from scrutiny of how much their students learn between the beginning of the semester and the end. Yes, there are student evaluations, which mean more than they once did. But they are poor substitutes for objective measures of student achievement. Basing education research and instructor performance assessment entirely on student evaluations is like basing clinical drug trials entirely on patient reports of how they feel.
And there’s one type of class that’s certainly amenable to standardized testing: developmental courses. If colleges choose of their own volition to give all incoming students a test to determine whether they need remediation, it seems reasonable to give developmental students a similar test once they’ve completed the course, and use evidence of (what one hopes is) increased learning to judge which kinds of developmental approaches — and instructors — are best.
The underlying cause of this remarkable information deficit is pretty clear: Colleges and universities don’t really need to know — or want to know — the answers to these questions. They don’t need to know because student learning results are peripheral to the core incentive system in which they operate. University success is measured in terms of dollars raised, high-achieving students recruited, and prestigious scholarship produced—period. Even less selective institutions are highly influenced by these values. They may not have the research mission of the academic giants, but they share organizational models, practices, and ways of thinking, all of which cut against rigorous self-evaluation of teaching and learning.
And they don’t want to know because, well, what then? I’ll wager dollars to donuts that any well-designed study of the relative effectiveness of full-time vs. part-time instructors would find far more variation within those populations than between them. That would have profound implications for the way college teaching is supervised and evaluated — like, for example, that it should be supervised and evaluated in a meaningful way. And what if someone actually proved that low-paid part-time professors and large, highly profitable freshman lecture courses had a demonstrable negative impact on learning? The money to support financially hemorrhaging sports programs, rapidly expanding administrative budgets, and light teaching loads for senior faculty doesn’t exactly grow on trees.
In fairness, there are some positive signs. The popularity of surveys like the National Survey of Student Engagement, the Community College Survey of Student Engagement, and the Collegiate Learning Assessment indicate a desire for self-study (as long, in the case of most four-year institutions, the results are safely hidden from public view).
But the overall lack of needed information about the core educational mission remains profound. The American higher education system is, we are often reminded, unmatched in diversity. It contains thousands of institutions and hundreds of thousands of college instructors, each granted significant autonomy to teach in different ways. Yet instead of learning from these differences we often ignore them. In our colleges and universities we’ve constructed the greatest engine of inquiry the world has ever known. That penetrating gaze is too infrequently directed inward, to know itself.
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
“But it’s not clear why giving professors license to say and think what they like necessarily translates into an absolute license to teach how they like, at a level of quality that’s more or less up to them.”
I grateful, Kevin Carey, for the all-important qualifier “absolute license” in the statement above.
James W. Gettys, at 8:10 am EDT on August 8, 2008
Well, let’s see. Faculty salaries are tied to student evaluations. Students don’t want to do any homework. Teachers give less and less homework, have more parties in class... and how much do students learn? Who cares? All that’s important is that everyone is happy and students have their piece of paper saying they’ve done the class time. Professors give exams out in advance and go over the answers, then let the students take them till they pass with an A...But you can’t do that with standardized tests!!!Good luck.
LM, at 8:45 am EDT on August 8, 2008
I think it would be a great think to do the kinds of studies the author proposed. But not until we have assessment metrics that do not replicate the problems with standardized testing. It will not help anyone learn to have higher ed faculty teaching to the test; for students to continue believing that all questions have right and wrong answers; and for the black-white test score gap to contribute to continuing devaluing of HBCUs.
ML, Sociologist, at 9:30 am EDT on August 8, 2008
Good essay, thanks. It’s likely a Herculean labor to introduce profession-wide metrics beyond the recruiting-oriented college rankings.
However, recording and analyzing data at the level of individual instructors or programs shouldn’t be too difficult. When I teach and evaluate student essays in lit or comp classes, I keep digital copies of all student work with my comments. At the end of the class, I’m able to identify learning and areas for additional work. This information, along with a series of grades, is the data I can draw on as I update my syllabi for future courses. With more experience, I’ll be able to track student learning (as opposed to student grades) and assess my performance at three levels: as an instructor, as an instructor of a particular class, and when teaching specific skills linked to specific assignments.
The catch is that I’m the profession’s migrant worker. Without the control of a permanent appointment, my data are useful only to me. But again, it wouldn’t be too difficult to design and implement a system to track student progress course over course and year over year at the sub-department level. Beginning with graduate instructors might offer benefits (good professional habits, but also no need to worry about inequality of workload) that introducing a system to an entire department couldn’t.
Humanities Grad Student, at 10:20 am EDT on August 8, 2008
Great article! This for me was the money line (no pun intended): “University success is measured in terms of dollars raised, high-achieving students recruited, and prestigious scholarship produced—period.”
I think that reason this has not changed has less to do with colleges’ desire to avoid unpleasant facts or the difficulty of measuring the outcomes than it does with the lack of any evidence that consumers care about “student learning outcomes” in a way that is orthogonal to reputation. Until that changes — until there is evidence that consumers will pay for learning outcomes as a factor that is independent from reputation — I don’t think you’ll see many colleges take up this responsibility.
Admittedly this is partly the result of the lack of available comparative student learning outcomes data, which is what the regulators are trying to change.
Market forces aside, though, I do think this is an ethical responsibility of the professoriate, as you point out. The continuing willingness to tolerate — and among many a determined insistence on the maintenance of — a virtual data vacuum on teaching effectiveness in this most basic of roles *should* be pretty embarrassing to the profession. In theory, faculty should be the party most aggrieved by the “hollowing out” and “commodification” of higher education — goodness knows they stand to gain the most from addressing it...I remain hopeful on this front — since I don’t think its going to change unless from this direction.
Mark Freeman, Director, Institutional Research, at 10:55 am EDT on August 8, 2008
There are two issues conflated here which need to be carefully distinguished: the research on pedagogy and effectiveness, which would add to our general store of knowledge and give us some tools to refine our approaches; evaluation of individual faculty, for salary or promotion/retention. The first, research, is absolutely a great idea, though I strongly suspect that our collective intuitions about teaching are close to correct. The second, though, raises terribly complex issues about the disparity between the measurability of different disciplines, the post-tenure review question, and the “achievement v. progress” question.
Jonathan Dresner, at 11:40 am EDT on August 8, 2008
“And what if someone actually proved that low-paid part-time professors and large, highly profitable freshman lecture courses had a demonstrable negative impact on learning?” What if indeed? The author seems to think that this would cause some change in the trend toward bigger classes and more part-timers. I’m not at all convinced that would be the case, partly because some administrators would simply say that even if there are costs in terms of student learning, they are outweighed by the benefits of having more money available for other things.
I also believe that both of these questions (either class size or full-time/part-time) are similar to the question of school choice for K-12 — people already believe they “know” the answers so unless (perhaps even if) every single study came out with the same results, I’m not sure data would change anyone’s mind (and I say this as an empirical social scientist, not a data-hater!). This is made even worse by the fact that I don’t think even controlled experiments can truly control for everything. In economics, there have been several studies of class size that used standardized pre- and post-tests (usually the Test of Understanding in College Economics), and the results have been mixed. Although I’m not aware of any where the assignment was truly random (since students still choose their section), the assignment of choices is fairly random (i.e., early choosers may choose smaller classes but who is an early chooser is random) and there are also statistical ways to control for this selection. Still, some studies find positive effects, some find no effect (thankfully, none find negative effects!). In economics, the finding of no effect is generally explained by the fact that professors do not adjust their teaching style for the class size (and economics is notorious for the ‘chalk and talk’ approach which is rarely exciting in any size class). But to provide definitive evidence that class size COULDN’T matter is a different question, and one that I think you will never convince people of, no matter how much data you throw at them.
Jennifer, San Diego State, at 11:40 am EDT on August 8, 2008
“In economics, the finding of no effect is generally explained by the fact that professors do not adjust their teaching style for the class size (and economics is notorious for the ‘chalk and talk’ approach which is rarely exciting in any size class). But to provide definitive evidence that class size COULDN’T matter is a different question, and one that I think you will never convince people of, no matter how much data you throw at them.”
Jennifer of SDSU (my institution!) is absolutely right to point out that how one teaches should differ according to class size, and therefore, so will the result. In a literature class, smaller class sizes mean more writing, more evaluation of the writing by the professor, more discussion in class, likely more discussion outside of class, more back and forth between professor and student, and between student and student. In short, better teaching. In large classes, one is reduced to using standardized tests and one piece of writing, if that. Interactions between professor and student are rare, especially for those who sit in the back of the very large class. In short, poor teaching.
And I’m not sure one needs a study to prove that.
Peter C. Herman, Professor at SDSU, at 12:55 pm EDT on August 8, 2008
Class size: If the data does not exist, why not err on the side of caution? If I am teaching difficult concepts to a group of 50 (or more), and if I can tell that they are having trouble—I sometimes can—then it would *seem* to follow that I could reach 15 students more successfully, data or no. At the very least, I would be able to address myself to their needs more directly.
As far as the full-time/part-time debate is concerned: how about we forget about whether f/timers are better teachers and focus on getting the p/timers the respect they deserve by moving them into f/time jobs?
The Grumpy Academic, at 1:10 pm EDT on August 8, 2008
Peter C. Herman commented, “In large classes, one is reduced to using standardized tests and one piece of writing, if that. Interactions between professor and student are rare, especially for those who sit in the back of the very large class. In short, poor teaching.”
There are several unverified assumptions in the above statement:
1- Standardized tests [and perhaps one piece of writing] = Poor teaching; ergo standardized tests are automatically bad.
2- Rare interaction = poor teaching; ergo interaction is automatically good, with the more the better.
3- Sitting in the back of large classes = not interacting; ergo the teaching is poor.
Hunh?
The underlying ideas here are that large courses are automatically bad because
1/they require standardized tests [who says?],
2/they disallow interaction [as if lectures that do not allow interaction are not valid pedagogical practices], and
3/ that students who sit in the back cannot interact, which means the teaching is poor [even though the student could move to the front, not interact in small courses, or just be a bad student].
Anecdotal evidence from decades of college experience refutes these ideas to some degree. To claim that lecturing to sizable groups of people is always and already bad runs contrary to the point of the article that real social scientific research needs to be done to assess learning on the part of the student [which is quite different from the the quality of the teaching by the instructor, which could be impeccable in any size classroom].
hunh?, at 2:10 pm EDT on August 8, 2008
“Yet the AAUP report contains no references to research proving the underlying premise of full-time professorial superiority — because, I’m guessing, it more or less doesn’t exist.”
Could this be because even AAUP doesn’t believe it? I’ve talked to both my local AAUP chapter and, briefly, the AAUP President, and the concerns stated about adjunct faculty were around their conditions of employment — lack of access to resources, lack of decent evaluations, low pay, no benefits, etc (with all the attendant consequences for students and the quality of education offered at institutions of higher learning). If there is a performance gap between full-time and part-time faculty, maybe we should look at the conditions adjuncts labor under, rather than assume anything about the skills of those doing adjunct work.
Additionally, the stated concern of AAUP, as far as I know, has been that drastically shifting the ratio of full-time to part-time faculty has at least two other negative consequences: 1) Shifting power from faculty to (increasingly corporate) administrators, with the attendant negative consequences for education quality; and 2) depriving students of the out-of-classroom benefits they get from having full-time faculty teach their courses. I’m again referring to, of course, the disparity in working conditions between full-time and part-time faculty. If an adjunct doesn’t have an office on campus, isn’t also doing service work with students, which experience is more likely to be better for the student? Which person is more accessible, more knowledgeable about the institution, etc?
I don’t think there’s much of a problem with having some part-time faculty, especially when it comes to specialized classes and unforeseen circumstances, but the combination of terrible pay/working conditions and a massive shift toward part-timers is not good — not for students, not for the professoriat, not for anyone.
Dennis, at 2:30 pm EDT on August 8, 2008
Right, and the check is in the mail!
Lets try an experiment, lets put all 15,000 students at my college in one “really big” classroom with one professor and see if any learning occurs.
Sometimes an intellectual exercise is just that and sometimes more statistics are just...more statistics. Lets leave staffing, curriculum development and room scheduling to those who know how.
dundermifflin, at 2:30 pm EDT on August 8, 2008
Hey, while we’re testing the impact of class size, let’s also test if the impact of large classes is any different from classes taken online. If not, let’s find the best professors in the country and have them webcast their courses to tens of thousands of students, at low cost. I can’t imagine that sitting at home watching a lecture is any less educational than sitting in the last row of a 500-student lecture hall.
One more thing: has anyone ever empirically tested whether liberal arts education has a better impact than practical education on things like morality, creativity, or any of the other qualities that liberal educators purport to impart in their students?
Author, No Sucker Left Behind, at 2:35 pm EDT on August 8, 2008
Not much to quibble with here. I would simply add that one kind of “evidence” would be “data” concerning what students preferences are. Of course, this “data gathering” could get tricky.
For instance, let’s take the class-size issue. If you ask students whether they prefer smaller classes, that’s one thing. But what about context? What if you tell them smaller class sizes will cost them more? What will their response be then? Are they willing to pay the price to obtain these preferences? Or will the assessment provide opportunities for discriminating between courses that warrant small classes and those for which, in some students’ minds, size doesn’t matter?
Same goes for the full-time vs. part-time issue. And this one is further complicated by questions about whether students feel a campus culture is more or less enriched by a more permanent full-time professoriate, than by a cadre of underpaid though dedicated adjuncts whose services keep costs down.
As we think about how we “justify” our “assumptions” by using “empirical evidence,” or “data” I hope these sorts of complications will be addressed.
Finally, there is always the “it all depends” response to questions that try to do “assessment.” As an undergraduate over 4 decades ago, I had a course in Shakespeare: it was a small class and the full-time professor was a disaster. The next term, the second semester continued the course. It was taught by a different full-time teacher via TV hookup on campus. Though there were no discussion sections (as often happens now in such courses) this course was fabulous.
George T. Karnezis, at 2:50 pm EDT on August 8, 2008
Humanities Grad Student, I want to learn more about your method of tracking student learning. It sounds really useful. Could you communicate directly with me at South Florida Community College? Thanks.
Charlotte Pressler, Dr. at South Florida Community College, at 4:05 pm EDT on August 8, 2008
Where ARE the data? Did someone say higher ed?
Bill, at 8:05 am EDT on August 9, 2008
Our Community College uses an exit test for our developmental reading class that is separate from the course grade. Performance on that test as well as improvement compared to the numerical score on the placement test would offer a chance to compare f-t and p-t instructors.
CC Professor, at 9:20 pm EDT on August 9, 2008
I expect it helpful in formulating generalizations, Kevin, to distinguish between K12 public schools and boarding schools as well as between top tier research universities and other forms of higher education. Their student academic performance increases appear on first blush markedly different. Seems data here would help clarify relevance of class size as well as instruction style.
Also, an ed policy doc student at Illinois State University in the early 1990s found no significance differences in K12 outcomes after conducting a meta review of over 1500 empirical studies. Wish I could remember her name, but you can find it with a little snooping.
Bob Heiny, at 9:20 pm EDT on August 9, 2008
datum = singulardata = plural
Therefore, where data the data is correct.
Michael @ Dunder Mifflin, NE, at 7:00 am EDT on August 10, 2008
Our students may be happy with classes taught by part-time faculty. However, they find themselves in a bind when they ask part-time faculty for letters of recommendation to grad school and then discover that such letters are not weighed as heavily by admissions committees as those coming from tenure track faculty. If they have never taken a course from a full-time faculty member (as happens) and do not know the tenure track faculty well enough to receive a strong letter, they are disadvantaged in their applications. Our university does not explain this to our students, since it would impact their course choices. This part-time full-time divide has more consequences than simply quality of teaching.
Because most part-time faculty have not published (or they would be competitive for full-time jobs) and are not engaged in active research, such faculty are less well-equipped to teach methods to undergrads. They are also not paid to do mentoring toward grad school and ill-equipped to do it. At our institution, part-time faculty do no advising. This, coupled with the low investment of part-time faculty in anything beyond their courses on our campus, precludes development of student-faculty relationships beyond class. If teaching is only a matter of assigning grades in classes, then the rest of this doesn’t matter, but is that all there is to education?
Perry, at 11:55 am EDT on August 10, 2008
It is important to consider the total education experience not just compare a single large class v. a similar small class; or a single class taught by a part-time v. a tenure-track faculty. If students are at a college where most classes are small, where they get to know their professors and where they interact with their professors and fellow students outside of the classroom, they are getting a different experience than students on a campus where most classes are large and often taught by part time lecturers who are not otherwise available on campus. There is potentially more to a college education than 40 3-unit courses; but the entire experience is affected by class size and who is teaching those classes.
Vince, at 5:00 am EDT on August 11, 2008
So many comments, and none mention that the sciences, led by physics, are taking precisely the called-for approach. Google the Force Concept Inventory, or Carl Wieman’d Science Education Initiative, to get a flavour of how large well-controlled studies of what works and what doesn’t are changing how science is taught.
Rosie Redfield, at 5:05 am EDT on August 12, 2008
Rosie Redfield, in her comment of 12 August 2008 on Kevin Carey’s piece, wrote: “So many comments, and none mention that the sciences, led by physics, are taking precisely the called-for approach.”
Right On, Rosie!! I’m glad to discover that there’s at least one Inside Higher Ed reader who’s aware of what’s going in physics education research and it’s stimulation of direct formative pre/post testing of higher-level student learning in college courses.
I recently transmitted to many discussion lists a response to Carey’s piece. The abstract reads:
***********************************************ABSTRACT: Gina Hiatt (2008) of the POD list has called attention to Kevin Carey’s (2008) provocative “Inside Higher Ed” report “Where’s the Data?” Carey wrote:
“. . . .when it comes to the central enterprise of higher education — teaching students — we don’t know if the reigning professional qualification system works, or how many professors we actually need. And this is true for all kinds of other basic elements of college teaching and learning — curricula, training, pedagogy, and much more. . . . . Why doesn’t anyone ever study how much learning varies between [courses], and why?”
Carey, along with most of academia, appears to be either unaware or dismissive of the fact that formative pre/post testing is being successfully employed to enhance student learning in many science, math, and engineering, and economics (but not psychology!) courses. For introductory physics courses it’s been found that pre-to-post-test average normalized learning gains for “interactive engagement” courses are about two standard deviations greater than those for traditional passive-student courses.***********************************************
Those who have nothing better to do may wish to click on http://tinyurl.com/5t5law to bring up the complete 19 kB post as it appears on the OPEN and secure AERA-L archives.
Richard Hake, Emeritus Professor of Physics, Indiana University 24245 Hatteras Street, Woodland Hills, CA 91367 Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands. rrhake@earthlink.net http://www.physics.indiana.edu/~h...http://www.physics.indiana.edu/~sdi/
Richard Hake, Right On, Rosie!! at Indiana University Emeritus, at 8:10 pm EDT on August 12, 2008
Advertisement
or search for jobs directly.
Founded in 1898, and affiliated with what is now New York-Presbyterian Hospital since 1927, Weill Cornell Medical College ... see job
HACC, a leader in education in Central PA, is a comprehensive, multi-campus community college, providing quality instruction ... see job
University of Rochester seeks Assistant Dean of Academic Affairs for the Eastman School of Music see job
Located just north of Houston, Texas, our five campuses serve 1,400 square miles. Our student enrollment is nearly 50,000 in ... see job
The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job
The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job
Under the direction of the Cuyamaca College Dean of Instruction, this position will manage the operational and programmatic ... see job
The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job
Financial Aid — Functional Systems Consulting. Take your career to the next level with CedarCrestone. see job
We are expanding, and currently seek an experienced Senior Director to join our Mission Support cluster. see job
Amen!
With the exception of a few picky — and mostly trivial — objections ... Amen!
Frizbane Manley, at 8:10 am EDT on August 8, 2008