Behind the Ratings

Friday’s release of a college ratings “framework” was a relatively anticlimactic milestone that capped months of speculation, delays and sometimes-fierce criticism from higher education leaders.

The U.S. Department of Education published just a handful of pages of information, much of which underscores what officials have been saying publicly for months. There are, however, some new details about how the administration plans to approach the ratings. And the department released an expanded version on Friday.

The document is the product of more than a year of scrambling by several dozen officials and staffers at the Education Department and other federal offices to meet President Obama’s August 2013 directive to create a ratings system. A core challenge, several people working on the project said, was how to translate several lines of presidential rhetoric about a system to measure the value colleges provide into a concrete policy proposal.

Department officials and their staff have been gathering weekly -- and at times daily -- since last August to try to pull together ratings that overcome a number of significant hurdles, said several people who worked on the project.

In addition to various Education Department teams, the ratings effort has involved several White House offices and other federal agencies, like the U.S. Department of the Treasury, which assisted with statistical modeling.

“It was a very, very difficult problem,” said Tom Weko, who worked in the department’s policy and budget office until March, when he moved to the American Institutes for Research.

Weko said the ratings presented a unique policy challenge because the department wasn’t working within the confines of a specific law or regulation, but rather had a largely amorphous goal. 

“If you can’t figure out what the problem is that you’re trying to fix," he said, "you’re at high risk for wheel spinning."

A widely discussed challenge -- both inside and outside the department -- was how to develop ratings based on the existing quality and availability of federal data.  

"There were a lot of smart, hard-working people trying to figure out how to address this problem," said Weko. Beyond surveying the sets of data already available to the department, he said, staffers also tried to come up with creative ways to combine or mix other federal sources of data, such as the Internal Revenue Service or Social Security Administration.

Another challenge was trying to adjust existing sources of data to meet the needs of the ratings system, Weko said.

"A typical discussion was: 'Gosh, the University of Chicago bachelor's grads don't have high earnings after five years, but that's because they're all pursuing Ph.Ds.,' " he said. "How do you exclude those people from the cohort of people you're including in earnings?"

Series of Delays

The release of the ratings system was delayed several times. Last year, Arne Duncan, the education secretary, said he expected the department to produce draft metrics by spring 2014, with a "first cut" of ratings due out about now. The department’s public timeline for producing an outline of the system slipped from spring to summer to, finally, “by fall,” a deadline that technically will be met, as winter officially begins on Sunday.

"They have now spent longer to get to the conceptual ratings than they took to draft, debate and pass the Affordable Care Act," said Terry Hartle, a senior vice president at the American Council on Education. "This cake has been in the oven for a very long time, and that is because this is not easy."

In the meantime, the lack of a concrete proposal hasn’t stopped college leaders or members of the U.S. Congress from slamming the administration’s proposal. For their part, department officials have said repeatedly that critics should refrain from judging the ratings system before it actually exists. Duncan last September dismissed higher education group's criticism of the ratings plan as "premature and more than a little silly" since the department had not yet published a plan. 

Ted Mitchell is a former college president who took the helm of higher education policymaking at the department this summer as the ratings were already being developed. He acknowledged that it’s been difficult to have a productive debate without more information about the department’s plan.

“We are probably more eager than anyone else to have a draft out in the public,” Mitchell said in an interview earlier this month. “The anxiety of not having something concrete to discuss has distracted the field and has made conversations difficult.”

Listening Tour

After the ratings plan was first announced last August, department officials fanned out across the country for a listening tour about the proposal. They held full-day public hearings in northern Virginia, California, Louisiana, and Iowa.

Denise Horn, a department spokeswoman, said Thursday that officials had received feedback on the ratings plan from more than 9,000 people at some 165 events.

Department officials also regularly delivered what became an evolving campaign-like stump speech about college ratings.

“I sometimes think I might as well just wear a T-shirt that says 'ratings' because that is something that people are very interested in hearing about," Deputy Undersecretary of Education Jamienne Studley quipped before a group of Ohio college presidents at a gathering on Capitol Hill earlier this year.

As was typical of department officials’ public remarks on the ratings, Studley sought in the speech to reassure college leaders that the government had heard their concerns about the possible pitfalls of a ratings system.

"Let me acknowledge first that this is complicated,” Studley told the Ohio college presidents. “We know it's a tough challenge. And we know that it's controversial."

Those speeches had a few recurring themes, such as justifying the need for a ratings system for accountability purposes (the department spends $150 billion in taxpayer money on grants and loans each year). They've also argued that students and parents feel overwhelmed by the process of applying to and choosing between institutions and need access to better data.

But for all of the department’s outreach, some in higher education have said they worry that their concerns aren’t necessarily being heard.

"You have to give the department great credit for meeting with anyone who wanted to talk about it,” said Hartle. "But this has been a very closed process. The department has been very unwilling to talk about their specific plans for a rating system."

Tod Massa, director of policy research and data warehousing at State Council of Higher Education for Virginia, was among the people whom the department turned to for technical advice on how to build a rating system. He said the department kicked off the ratings development well by soliciting a broad range of input, "but then it kind of fell apart."

"After that, all we heard was rumors and promises," he said. "They owe it to the community to have an open and transparent process all the way through."

Massa questioned why department officials did not work more closely with outside data experts as the process moved forward.

"Lots of us intimately know the available data. We know what can and cannot be done," he said. "It's not like they could have come up with anything that really hadn't been done before without new data sources."