What's 'Good Enough'?

SAN FRANCISCO -- Four years ago, as a member of a panel convened by then-Education Secretary Margaret Spellings to negotiate new federal regulations on accreditation, Ralph A. Wolff was among several accreditors forced to respond to criticism that they must do more than simply require the colleges they accredit to measure whether their students are learning, as they had been doing for years. With some colleges reporting graduation rates in the single digits and evidence that students were emerging from college with inadequate skills, wasn’t it time, the critics argued, to focus on more than process and hold colleges more accountable for their actual results?

The Spellings panel went so far as to put forward a proposal in early 2007 that accreditors identify a “core set of student achievement measures, both quantitative and qualitative,” and define an “acceptable level of performance” that all colleges should meet.

Wolff, president of the senior college commission of the Western Association of Schools and Colleges and a negotiator on the panel, bristled at the idea. Accreditors, he said at the time, have traditionally put the onus on “an institution to define its learning outcomes and to assess the achievement of those outcomes, and through that assessment to determine whether improvement is needed…. We should keep that locus of responsibility at the institutional level.” It would be a mistake, he said, to identify “quantitative ‘bright line’ indicators that ... if you fall below them” the institution is in trouble. Facing pushback from Wolff and many others, the administration decided not to put forward new rules for accreditors.

Last week, at his agency's Academic Resource Conference here, Wolff and WASC's other leaders sought to sell presidents, provosts, and faculty members from the region's four-year colleges on the idea that, with policy makers in Washington increasingly questioning the future of accreditation, the time has come for major changes in the agency's peer review process.

Many of the alterations in the "redesign" would represent a significant break with how WASC and the other regional accrediting agencies have historically operated, including by making public the commission’s letters of findings about individual colleges and using an outside auditing firm to review the finances of publicly traded higher education companies during WASC’s accreditation process. Currently, WASC announces the actions it has taken on colleges, but does not release the evaluations themselves.

But perhaps no change would be more dramatic than a proposed requirement that colleges benchmark their own learning outcomes and measures of student success (i.e., retention and graduation rates) against those of their peers. One WASC document summarizing the commission's goals suggested that each institution would work with WASC to set a "target graduation rate" (and potentially different rates for different subgroups) that it would "be expected to meet or exceed."

“Our institutions have done an enormous amount of work on the process of assessment” of student learning, “but we need to do better at answering what the results mean,” Wolff said in an interview at last week’s meeting. But amid increased questioning about how much college students are learning, he said, WASC officials believe the time has come “to work with institutions to decide how you would validate those results in a responsible way…. What’s ‘good enough’ has to be more transparent.”

As Wolff and other WASC officials floated the proposed changes to groups of presidents and other senior campus administrators, pushback against them was relatively muted, probably because scores of campus officials have been involved in drafting them over the last year. Several presidents applauded Wolff and WASC for getting out front in contemplating major changes in accreditation, to help ward off potentially increased political interference.

But the criticism grew sharper when WASC officials ran the proposals by faculty members and academic administrators closer to campus classrooms, with several people expressing concerns that requiring colleges to compare their outcomes to peers' would force them to use measures that are readily comparable to other institutions -- and hence more likely to be standardized and simplistic.

Others said they feared that too much emphasis on measures like retention and graduation rates would prompt colleges with lower rates than their peers either to raise their admissions standards on the front end (to keep out lower-performing students who might drag down their performance) or “pass people who should not be passed,” as one person put it.

Aiming Higher

WASC's senior commission (the changes would not apply to the Western accreditor's two-year-college commission) did not set out to undertake such a thorough overhaul. The accrediting association, which represents about 160 colleges in California, Hawaii and several U.S. territories, began its federally mandated once-a-decade revision of the handbook that guides its (and its members') work with a relatively modest goal: reducing the costs and demands of the process while retaining its focus on student success.

But at the same time, WASC notes in its description of the process, “serious challenges have arisen to the effectiveness of accreditation in addressing low graduation rates, evidence of ineffective learning of graduates, and questionable practices by for-profit institutions.”

Those concerns have led to intense scrutiny of accreditation in a variety of venues. The Obama administration has picked up where the Bush-era Education Department left off in undertaking a review of whether the peer-led process is a sufficient guarantor of quality, with some of its advisers going so far as to suggest breaking the link that makes accreditors the arbiter of whether colleges can qualify to award federal financial aid. The Senate, meanwhile, has gotten into the act by calling regional accreditors onto the carpet to defend their regulation of for-profit colleges.

In the interview, Wolff acknowledged that the external pressures on accreditors motivated WASC to consider a more thorough overhaul of its policies, and that the questions he endured from the Spellings-era panel in 2007 "have stayed with me." But he rebuffed the suggestion that the changes the commission was weighing have been driven by the external pressures.

"There is a convergence here, but we wouldn't be doing this unless we thought it was the right thing to do," he said. "We can't be sure that any action we take will stop [potential government intervention into accreditation policy] or forestall it, so we have to do this because we believe it's good for students."

The redesign process WASC has undertaken is wide-ranging, including outside consultants and five panels made up of 80 college officials, and it examines many aspects of the commission's work. The overarching goal, as commission officials describe it, is to put student learning at the center of the accrediting process, in part by strengthening how the agency looks at those issues and in part (here's the tradeoff for colleges) by streamlining the time and cost that institutions spend on other parts of the review.

Changes Ahead

If they are ultimately embraced later this year by the 25 college administrators and others who make up the senior college commission, some of the changes would be significant but relatively straightforward.

One set is designed to make WASC's process more transparent, by making public the "action letters" the agency issues at the end of each college's accreditation process and establishing a "report card" that would include information about how an institution scored on the various criteria used to judge colleges. The opaqueness of the accreditation process has been a major bone of contention to many critics.

The Western accreditor may also consider a system in which it would rate institutions not just by whether they are accredited or not, but also by "categories of distinction" in certain areas that colleges might apply for, along the lines of the LEED program in environmental sustainability.

WASC is weighing several steps aimed at strengthening its oversight of for-profit colleges. Wolff said the agency was considering using "secret shoppers" to explore the institutions' recruitment, admissions and financial aid practices. And the accreditor is testing out a process aimed at assessing the finances of for-profit colleges, particularly complex publicly traded ones.

For an unnamed for-profit college that is seeking its accreditation now (almost certainly Bridgepoint Education's Ashford University), WASC is contracting with accounting firm KPMG to lend its expertise. This step would be meant to acknowledge to the critique that accreditors do not have the wherewithal and expertise to understand and judge complex for-profit companies, but also to keep such a review under the accreditors' umbrella, rather than peeling it off as some commentators have urged.

"It's not entirely clear that it's profitable to try to train a bunch of nonprofit folks in the workings of for-profits," said Harold Hewitt, executive vice president and chief operating officer at Chapman University and a member of WASC's for-profit task force. "This may be a better way to handle it."

WASC is also experimenting with the Lumina Foundation for Education's recently released Degree Qualifications Profile, which aims to define the types of knowledge that degree holders at various levels (associate, bachelor's, etc.) should have and what they should be able to do.

Perhaps the biggest change that WASC is contemplating, though, is the idea of forcing institutions to identify levels of learning and student success that would be externally validated through comparisons with peer institutions.

The goal, said Linda Johnsrud, provost and vice president for academic planning and policy at the University of Hawaii System and chair of WASC's senior college commission, is to make accreditation less about setting "minimum standards" and more about ensuring quality.

"It's a natural evolution," she said. "We've been measuring [student learning], but to what end? We've never said, 'Is it good enough? Are the outcomes we're getting what we want them to be?' "

Wolff and Johnsrud were quick to note that WASC is not looking to impose "bright line" metrics on institutions, and that in the accreditor's vision at this stage, individual colleges and universities would set their own goals in tandem with WASC's visiting teams of peer reviewers.

But the fact that the measures would need to be "externally validated" -- which, to some, means compared statistically to other institutions -- almost inevitably means that WASC will have to lean toward metrics that are readily comparable from one institution to the next. When it comes to student learning outcomes, that could mean an emphasis on standardized exams over e-portfolios and homegrown assessments; for measures of student success, it might favor numerical outcomes like retention and graduation rates over more qualitative measures.

That specter was raised by some members of the audience during one of Wolff's presentations at last week's WASC meeting. "A whole lot of bad things" happen when institutions become "obsessed with retention and graduation rates," said one questioner. "There is a field full of land mines in front of us; I hope we can navigate it without getting blown up."

Another said WASC's proposed transition to "the arena of external comparisons" was a "fundamental shift" that would be "a hard sell" to faculty members on his campus, who will be concerned that a common set of metrics will straitjacket their own assessment efforts.

Wolff said he understood that that might be the case -- but that accreditors have to answer the questions being asked of them. "We will contextualize new metrics to the type of institution, but we have to have very clear indicators" to satisfy the accountability demands. "I don't know if it will work, but it would allow us to make a much more defensible case than, 'We all go through a process and it leads to a result we won't tell you about.' "