Ratings Plan Arrives, Details Scant

WASHINGTON -- After nearly a year and a half of public debate over its proposed college ratings, the Obama administration on Friday provided the first glimpse into how it will structure such a system, including the criteria it will use to judge colleges.

The administration’s “framework” identifies nearly a dozen metrics that officials are planning to use -- but leaves a host of important questions unresolved (highlights here; full release here). The department published more details Friday morning.

The most straightforward set of measures looks at the rate at which colleges enroll low-income and first-generation students -- and the extent to which institutions are affordable. Those measures include the percentage of students who receive Pell Grants, the financial need of students based on the federal aid formula, the students' family income levels, and the number of first-generation college students on a campus.

In addition, the department is considering the average net price of an institution, which is the actual amount that families end up paying after they receive aid. The Higher Education Act, which is the federal law that governs financial aid, already requires the U.S. Department of Education to publish an annual list of the highest-net-price institutions each year.

Among the most controversial elements of the ratings system has been its plan for measuring student outcomes. The outline released Friday says the department wants to judge colleges based on their completion rates (as well as transfer rates and how many students go to graduate school) as well as on graduates’ earnings.

“The outcome metric has always been the most complicated,” Undersecretary of Education Ted Mitchell said. “We don’t intend and ought not to intend to capture every outcome of higher education."

Over all, the department’s approach in the outline appears to be moving away from a system that lets students and families draw comparative value judgments between colleges and closer to something that resembles a set of minimum standards for institutions.

Mitchell said, for example, that the department thinks setting a "threshold" or a “floor” on the earnings metric would solve an issue of perverse incentives. Critics have warned about judging colleges based on graduates' earnings data, which could encourage institutions to graduate more bankers and engineers rather than teachers or public servants. Such a threshold would work, he said, by rating a college based on the share of students who go on to earn salaries that exceed some multiple of the minimum wage or poverty line rather than publishing median salaries.

In a paper released Friday morning, the department called such an approach a “substantial employment” measure. It would essentially answer a yes-or-no question about whether enough graduates meet a minimum standard for earnings in the short-term, which the department defined as six years after enrollment or two years after completion.

The goal, the department said, is to pair the short-term earnings metric with a calculation of graduates’ salaries over a longer-period (10 years after enrollment or six years after completion). In calculating the long-term metric, the department said it might use graduates’ average or median salaries. The department floated the idea of adjusting the measure to account for outliers (such as the highest earners, like corporate executives’ salaries), students in graduate school, those with a permanent disability, or who had died. 

The goal of the earnings outcomes data, Mitchell said, is to produce ratings that identify which institutions "equip a student to graduate, go to work and in that work be able to pay their loans and get started with those big things in life like saving for a mortgage."

Mitchell said the department acknowledges that the ratings system doesn't "aim to be comprehensive" or seek to "measure everything." 

The framework offers few concrete details other than 11 data points that department is considering. Mitchell said he expected the rating system would include fewer metrics than the ones outlined in the framework.

Officials said they will rate only degree-granting undergraduate institutions based on three different levels: high-performing, low-performing, and “middle.” But the department has still not decided whether a handful of metrics will feed into one single rating for an entire institution or whether it will issue separate ratings scores for each metric, according to Mitchell.

Regardless of whether there is a singular rating or a set of them, Mitchell said, the goal is to keep the number of institutions in the high- and low-performing categories relatively small, with the bulk of institutions falling into the “middle” category.

Perhaps the most definitive part of the framework was what the department said would be left out of the ratings. Among the metrics the department has ruled out: levels of civic engagement by students and alumni, job placement rates, and student satisfaction with their college or some aspect of it. 

In addition, the department said it did not want to include average loan debt in the ratings. That information is currently part the federal College Scorecard. But the measure, the department said, should not be a part of the ratings because it reflects a student’s financial resources more than it does college affordability. 

The framework also leaves unanswered how to compare student outcomes among institutions.

The department has not decided -- though it is possible -- whether to create a statistical model to control for different institutional types (and perhaps colleges' location) and the population of students that an institution serves (such as low-income, first-generation students versus wealthier students who come to college better prepared).

So far, the department has settled only on one way to group colleges in the ratings system: separating two-year institutions from four-year institutions. But officials said they are exploring more granular ways to compare institutions of similar types. One possibility, they said, would be to group colleges by selectivity or the type of programs and degrees they offer. That might mean grouping liberal arts colleges together or looking at institutions that share a specialty, like engineering.

Higher Ed Remains Skeptical 

Terry Hartle, senior vice president for government relations and public affairs at the American Council on Education, praised the department for producing a “thoughtful status report” but said the lack of details showed officials do not have a clear direction for how to proceed.

“For many institutions, this will heighten concerns that what matters to the agency is the need to get it done by the deadline and not necessarily the need to get it right,” Hartle said. “Does this increase our confidence that they would do this [rating system] well? The answer is no. They would get an incomplete on this.”

In announcing the plan in August 2013, the Obama administration said it would publish a first version of the ratings by the 2015-16 academic year. 

David Warren, the president of the National Association of Independent Colleges and Universities, was among the higher education leaders briefed on the outline by the department on Thursday. He said he was concerned the department had not made any significant changes to the ratings plan since it was first announced. 

“They’ve had 16-and-a-half months to come to where we are," he said. “And they’ve not changed the rudimentary metrics. I’m not clear where the feedback they received has altered the proposal.”

Warren said that although it generally would be acceptable for the department to publish more data about student outcomes, he remains concerned that combining multiple data points into a rating that carries the weighty “imprimatur of the federal government" would be reductive.

“It is a limited, arbitrary and narrow set of variables they picked,” he added. “Our concern is that it’s not going to provide a prospective student and parent with a very good picture of what it means to go to college.”

David Baime, senior vice president for government relations and research at the American Association of Community Colleges, said he was surprised by the "tentativeness" of the plan released.

"It looks a lot less specific than what we would've expected," he said. 

M. Peter McPherson, president of the Association of Public Land-grant Universities, praised the proposal for incorporating some of his organization's recommendations, such as rating institutions based on a three-tier system.

"There should be consequences for the very bad performers and rewards for excellent ones," he said in a statement. 

But McPherson also said the proposal fell short in relying on incomplete graduation data and not fully endorsing a "student readiness adjustment" to account for the demographics that an institution serves. 

"The absence of such an adjustment as part of the ratings system could create perverse incentives for institutions to shift away from taking in more low-income and first-generation students, since it could negatively impact a school’s rating."

The document released by the department on Friday, after McPherson’s statement, did float the idea of adjusting outcomes based on student and institutional characteristics. The department said, for instance, that it was considering adjusting student outcomes to account for family income, age, gender, enrollment intensity, and other factors. It may also, according to the framework, adjust outcomes based on how selective a college is in admissions. 

Timeframe

The release of a draft ratings system has twice been delayed, but officials insisted that they are still on track to publish actual ratings for the first time next year. As part of that process, the department also said it wants feedback on how best to present its ratings system to the public. It said, for instance, that it is considering a ratings website as well as publishing an Application Programing Face, or API, which is a set of web protocols that would allow developers to build third-party services based on the ratings.

The department also said it would host “structured discussions” about its framework next year. Those public events are expected to be announced in “early January.”

The department will solicit public input on the framework released Friday over the next several months, with a comment deadline of Feb. 17.