Hidden in Plain View

A lot of ink has been spilled over the years about “America’s Best Colleges,” the annual rankings published by U.S. News & World Report. We read periodic complaints that the rankings improperly rely on inputs, resources, and a reputation survey of questionable validity. The rankings are in the news again, with fresh complaints about punitive actions against institutions that withhold data, and a fresh call for collective acts of resistance against the rankings juggernaut.

Surprisingly, a fundamental aspect of the rankings has gone relatively unquestioned -- the composition of the comparison groups. Because the groups are derived from the Carnegie Foundation for the Advancement of Teaching’s classification of colleges and universities, and because I direct that work, I have a special interest in the question of who gets compared to whom. It turns out to be a questionable use of the classifications that can have significant consequences.

The U.S. News rankings framework asserts that there are two kinds of higher education institutions -- universities and colleges -- and they compete in national or regional markets. Consequently, there is not one set of rankings, but ten: national universities and (national) liberal arts colleges, four regionally differentiated sets of master’s universities, and four regional sets of comprehensive colleges. To a considerable extent this reflects the reality of the higher education admissions market. It also makes good business sense: multiple rankings mean multiple sets of “top” performers who will be happy with the results and proclaim their distinction, and who will be eligible to display the America’s Best Colleges award badge on their Web sites and in their promotional materials.

But it leads to big definitional challenges. What’s a university? What’s a college? What’s national and what’s regional? There is no consensus, to be sure. Name alone is not sufficient for distinguishing universities and colleges: Boston College and the College of William and Mary, with their extensive graduate offerings and research orientation, are clearly universities, while exclusively-undergraduate Illinois Wesleyan University and McMurry University are clearly colleges in the conventional sense of the term. Considering the presence or absence of graduate degrees is also unsatisfactory: most would not list Bryn Mawr, Lewis & Clark, Middlebury, or Williams Colleges among universities, graduate programs notwithstanding. Defining national and regional is even trickier.

So what do they do? Easy -- they rely on the Carnegie Classification! (Now that there are several classifications, I should be specific: they use the basic classification, meaning the latest version of Carnegie’s “legacy” classification.) Any institution classified among doctorate-granting universities is deemed to be a national university, and all baccalaureate colleges -- arts and sciences are (national) liberal arts colleges. All master’s universities and the remaining baccalaureate colleges are ranked by region.

It’s reasonable to use the Carnegie Classification to define universities and colleges, though there are cases where the distinction is far from clear-cut. But there’s no basis for inferring national versus regional focus, because it’s not a factor in the classification criteria. So it should come as no surprise that the national and regional lists contain a great many inconsistencies and bizarre placements.

Let’s consider some examples. I’ll take a simple, straightforward approach to defining regional and national, using freely available data on the geographic origin of entering students. Among the roughly 270 institutions that U.S. News will label national universities in next year’s rankings (because they award at least 20 doctoral degrees annually), 82 draw students from fewer than 30 states or territories (for example, Cleveland State University, Illinois State University, University of La Verne, and University of Massachusetts-Lowell -- fine institutions, but undeniably regional in focus). By contrast, among about 660 universities that will be ranked regionally (because they don’t award doctorates, or award fewer than 20 per year), more than 100 draw from at least 30 states or territories (including Hampton University, Ithaca College, James Madison University, and Villanova University).

It’s even worse with respect to the colleges. Of almost 300 that will be ranked nationally because a majority of their graduates major in the arts and sciences, more than 120 enroll students from fewer than 20 states or territories (e.g., Ave Maria College, Burlington College, Penn State Abington, and the University of Wisconsin-Parkside), while about 100 of the 460-odd regional colleges have greater geographic diversity.

If we substitute a measure of admissions selectivity as the metric for regional or national focus, we get similarly bizarre contradictions: national universities and colleges with open or near-open admissions, and regional ones that are quite selective.

This isn’t just a technical issue. Ask any president of a regionally- or locally-focused college that -- due to students’ majors -- gets ranked against wealthier, far more selective colleges that truly compete nationally. If she calls U.S. News to request a correction, the answer will be, in essence: “Our comparison groups are based on the Carnegie Classification, so if you don’t like it, call them.” So she calls the Carnegie Foundation (read: me) to explain that hers is a regional institution. But because that’s not the basis for classification, and because misuse of classifications should not drive classification decisions, she doesn’t get satisfaction from us, either. She -- and her college -- are caught in the middle: stuck in the wrong comparison group, with potentially serious consequences for recruitment and fund raising as well as board and alumni relations, and no recourse.

If U.S. News thinks the national-regional distinction is important, they should define it. It’s preposterous to use degree offerings (for universities) and the proportion of majors in arts and sciences (for colleges). I told them so last November, and demonstrated how they could do it better. They heard me out, but ultimately decided that consistency (or convenience) is more important than accuracy and rationality. By continuing to rely on the Carnegie Classification, they avoid the tough job of defining their terms -- and in so doing, they can dodge a whole category of complaints by shifting responsibility to a third party.

If they aren’t willing to do it right, they should either stop making the distinction or they should adopt a much more flexible stance for institutions that legitimately challenge their comparison group placement.