Measuring What?

NEW YORK -- The New York Times kicked off its higher education conference here Monday night by releasing what it called a "revolutionary college index" that ranks institutions that enroll students from low-income backgrounds.

The rankings are derived from a formula based on the proportion of undergraduates who receive Pell Grants and the net price (what students actually pay as opposed to sticker price) paid by those with family incomes of $30,000 to $48,000. But the Times applied this formula only to institutions with a four-year graduation rate of at least 75 percent. That's a bar that only about 100 colleges meet, and all but three of them are private institutions. (The full Times methodology may be found here.)

So while the leading three (Vassar College, Grinnell College and the University of North Carolina at Chapel Hill) are all institutions that don't normally top the likes of Harvard University (ranked sixth), they are also not institutions that serve as many low-income students as many public institutions that don't meet the 75 percent standard. Of the 100 colleges in the Times rankings, the top percentage of Pell-eligible students is 25 -- at Susquehanna University. (The full list of colleges, in order, may be found here.)

But consider some of the public institutions that weren't ranked. For example, the University of California at Berkeley, which just misses the four-year cutoff (71 percent), has more than 27 percent of its students on Pell Grants, a larger percentage than any that were ranked and about twice the rates of Dartmouth College and Duke University, which made the list. (And Berkeley's student body is much larger than that of most of the colleges in the new ranking, such that its Pell students have typically been close in number to the combined total for the eight Ivy League institutions.)

Baruch College of the City University of New York has a lower four-year rate (39 percent) so it too was excluded, yet its percentage of Pell students is 46 percent -- twice that of Vassar. The Times also did not find any historically black colleges that met its standard, although those institutions are strongly committed to serving low-income students. Spelman College, where 70 percent graduate in four years, also has 46 percent of its students receiving Pell Grants. And these colleges don't include community colleges and other open-access institutions, many of which are majority Pell-eligible. (All the figures in this paragraph and the proceeding one for institutions not ranked by the Times come from the U.S. Department of Education's College Navigator site.)

In introducing the new rankings, David Leonhardt, managing editor of The Upshot column of the Times, acknowledged that there were no perfect measures and hinted that the Times might in the future examine more institutions. Bernie Machen, president of the University of Florida, was on a panel here to discuss the rankings, and he quipped that the newspaper might find many interesting things "if you ever get down to the real world" of institutions that don't have 75 percent four-year graduation rates.

Leonhardt said that it was important to note that among the colleges that were ranked, there was wide variation in how they fared, and that this variation wasn't entirely related to wealth. Susquehanna, for example, outperforms institutions for which its endowment would be a rounding error. And the Times rankings draw attention to institutional wealth, listing endowment per student. Vassar, for example, in coming in first, beats out 27 colleges and universities on the list with higher endowments per student.

"It's clear that colleges have very different levels of commitment on this issue," Leonhardt said.

For example, the article he published on the new rankings takes Washington University in St. Louis to task. "Maybe the starkest example is Washington University in St. Louis, one of the hot colleges of recent years, having climbed to No. 14 in the U.S. News rankings last year," he writes. "Only about 6 percent of the freshman class in recent years at Wash. U., as it’s known, have received Pell Grants, even though it is one of the country’s 25 richest colleges on a per-student basis."

A statement from Washington University late Monday said that the university has a higher percentage of Pell students this fall (8 percent) than reflected in the Times rankings, and that the share has been moving up, in part because of additional aid provided to low-income students. The university is pushing to raise more money for financial aid, the statement said, with a $400 million target for such funds in a current campaign.

Speakers here said that institutions needed to spend more on low-income students. Vassar's 23 percent Pell-eligible student body is nearly double what it was in 2008. Catharine Bond Hill, president of Vassar, said here that if all the elite colleges received twice as many eligible applicants from low-income families next year, she didn't think many would have the resources (or commitment) to admit them.

She noted that it is a distinct minority of American colleges that are need-blind. She said that "we have known for a decade" that there are many more qualified students "out there" than are enrolling at elite colleges. (While Hill was among those commenting Monday, Leonhardt said that she did not know in advance how Vassar would fare.)

The reality that the institutions that serve the greatest number of Pell-eligible students weren't in the rankings bothered some experts (who were not on the panel here).

"When your peers are doing terribly, doing better than your peers isn't the same as doing good. The idea that Harvard is 'accessible' to poor people when such a small number pass its gauntlet for admission to earn the nearly free net price on a campus where hidden expenses abound -- it's laughable -- is a sure sign of a flawed methodology," said Sara Goldrick-Rab, director of the Wisconsin HOPE Lab and professor of educational policy studies and sociology at the University of Wisconsin at Madison. "This is a set of metrics designed to praise the wealthy for doing a little philanthropy -- not a set of metrics aimed at helping spur colleges to become more accessible or affordable. You only win on this metric by being rich. The New York Times has helped perpetuate inequality. And it has added absolutely nothing to the college rankings or ratings conversation."

Paul Glastris, editor in chief of The Washington Monthly, which also ranks colleges in part on their commitment to low-income students (and many other factors), had more praise for the effort. "Good for them for doing this," he said, adding that "it means a lot coming from The New York Times." But he noted that his magazine's methodology, which lacked the 75 percent four-year bar, resulted in 9 of the top 20 being public institutions. Further, 6 of his top 11 are at or above Pell proportions of 30 percent.

F. King Alexander, chancellor and president of Louisiana State University, has written about the need to encourage colleges to admit large numbers of low-income students. He said that it was "a good first step" for the Times to encourage elite institutions, but questioned the impact. Alexander was formerly president of California State University at Long Beach, which when he left had a 60 percent graduation rate and enrolled a student body in which 40 percent were Pell eligible. That university's 14,000 Pell students exceeded those of the Ivy League, he said.

Alexander said he would like to see ratings that only include those colleges and universities that meet some minimum Pell level, such as 20 percent, rather than only analyzing those with the highest graduation rates. "I think that the 75 percent grad rate kills most of any scale that could make the new index better," he said.

The Challenges Facing the Publics

Only three public institutions -- the University of North Carolina at Chapel Hill, the University of Virginia and the College of William and Mary -- made the Times list. In the discussion here, however, there was more attention paid to publics, especially to flagships.

Machen of the University of Florida (another public whose 29 percent Pell-eligible students tops all institutions rated by the Times) talked about how most of the money given out for financial aid at his university is not based on need, but on various definitions of merit. He said that state mandates give him no choice, and that he has been rebuffed by lawmakers when trying to get more money for need-based aid.

Anthony W. Marx, president of the New York Public Library and former president of Amherst College, said that, as a nation, "we are busy under-funding and dismantling our public universities."

And Leonhardt noted that UNC-Chapel Hill -- which fares well in the ranking -- is actually facing criticism for being generous to low-income students. The University of North Carolina System Board of Governors recently voted to limit to 15 percent the share of tuition revenue that campuses may spend on financial aid. Chapel Hill is one of six campuses in the system that currently spend more than 15 percent of tuition revenue on financial aid.

The new rankings were released at the opening of the Times's Schools for Tomorrow conference. Inside Higher Ed is a media sponsor of the conference, meaning that it traded advertising with the Times. But Inside Higher Ed played no role in the development of the rankings or any conference content.

Those Other Rankings

The Times rankings were released at 6 p.m. Six hours later, U.S. News & World Report released its rankings, which did not contain surprises. There were no methodology changes in those rankings this year. The magazine did add information about campus crime to its database about colleges, but did not factor those statistics into its rankings.

Even before the new U.S. News figures were out, at least one college was scrambling to deal with an error. The News-Press reported that Florida Gulf Coast University found itself unexpectedly on top of the magazine's list of high debt colleges with an average debt for graduates of $56,208. The real figure is $21,394, and the error was due to a typo by the university in submitting information.