The Challenge of Comparability

It's hardly a coincidence that the push for colleges to measure and report how successfully their students achieve certain learning outcomes has been least embraced by elite, traditional institutions already perceived to be atop the higher education pyramid, and championed most by colleges with less name recognition or nontraditional approaches. The former would arguably have the most to lose, and the latter the most to gain, if evidence about student learning were to upend longstanding perceptions about which colleges did the best job.

So it's also not surprising that of all the accountability systems that have emerged from within higher education in the last few years, perhaps the most aggressive and expansive, a project known as Transparency by Design, has come from a group of primarily online institutions that serve adult students.

The system, which officially gets off the ground today with the launch of its Web site, CollegeChoicesforadults.org, differs from the accountability tools developed by groups of public (College Portraits) and private (University and College Accountability Network) colleges primarily because it aims to provide tangible information about students' learning outcomes in specific fields and programs -- in organizational leadership at Arizona's Rio Salado College, business administration at Kaplan University, and accounting at Franklin University, to cite three examples.

The site already does or soon will provide nationally comparable information for all participants in such areas as undergraduate student engagement (using nationally normed surveys), and alumni satisfaction (using a commonly agreed upon set of questions the colleges have agreed to ask their graduates).

Despite the lofty goals of Transparency by Design's leaders, though, the program has been ensnared in exactly the same kind of concerns that have imperiled or at least impeded other discussions of college learning outcomes -- difficulty in defining common outcomes among diverse groups of institutions, and worries about whether disclosure of some information will make some colleges look worse than others, hurting them competitively. After much discussion among the participants, leaders of the effort opted not to ask institutions to use common measures of program-level learning, but to be fully forthright about the measures they use and the results they produce -- information typically found buried deep within accreditation reports or institutional research Web sites, if public at all.

"We probably would have spent the next 30 years getting" to agreement about common outcome measures, says Michael Offerman, vice chairman of Capella University and chair of the project's executive committee. "We opted instead to lay things out that have been hidden or secret, and that's scary. It's not perfect, and for now it's the best we have. But we're going to figure out how to do it even better."

Accountability for Adult Students

As political and other pressure has built on colleges to prove their performance and value, prompting various efforts in response, the participants in Transparency by Design (a list of participating institutions to date can be found at right) were motivated by their desire to provide information about colleges' performance for a very specific set of prospective students: adults (rather than traditional college-aged students) who were interested in studying via distance education.

While comparable mechanisms like College Portraits and U-CAN focused primarily on traditional undergrads, those behind Transparency by Design (the Presidents' Forum of Excelsior College and the Western Interstate Commission on Higher Education's WCET, backed -- what isn't? -- by the Lumina Foundation for Education) sought to differentiate it by emphasizing information most important to older students, who are likeliest to be viewing college as a way to bolster their experience and grounding in specific workplace fields.

The participants began with a working consensus (disputed by some in higher education) that, wherever possible, common definitions and measures were preferable. So they agreed, for instance, to use the National Survey of Student Engagement or the Community College Survey of Student Engagement for the section on student engagement, the Educational Testing Service's Measure of Academic Proficiency and Progress for the section on general education learning outcomes, and the Priorities Survey for Online Learners, from Noel-Levitz, to assess the satisfaction of online learners. (Some of these data aren't on the site yet, but will be in the coming days and weeks.)

In addition, the colleges agreed to ask their alumni a common set of questions ("Is the education you received relevant to your current career?" " How satisfied are you that this institution helped you meet your program outcomes?") to try to produce comparable information about after-the-fact satisfaction.

But when it came to defining sets of common learning outcomes for specific degree programs -- Transparency by Design's most distinguishing characteristic -- commonality was hard to come by. There was widespread agreement, says Offerman of Capella, that the members wanted to provide "proof that you're really teaching people what they're there to learn." But defining that was difficult because of the wide diversity of institutions in the consortium, which includes for-profit universities (such as Kaplan and American Public University System) as well as public (Connecticut's Charter Oak State College) and private nonprofit (Southwestern College in Kansas and Franklin University) institutions, and the full gamut from two-year (Rio Salado) to primarily graduate (Capella University) institutions.

That mix, says Offerman, is "our blessing and our curse," the latter in terms of posing barriers to a common view of mission and approach. "You might think, how different can one accounting program be from another," he says. "But it was unbelievable the level of defensiveness that people had about the ability to differentiate among programs."

Instead of continuing to disagree over that, says Russell Poulin, associate director of WCET and project director for Transparency by Design, the group's members opted to require each institution to make public both the measures it uses to gauge whether students are learning what they're supposed to in specific programs, and its results. Most institutions use such information for internal purposes only, and "we are hopeful that just by the whole act of sharing, we'll bring the conversation about what people are doing out into the open," says Poulin.

The initial information produced is not readily comparable, not surprisingly. Western Governors University's master's in business administration, for instance, uses a set of "objective," "performance," and "essay" assessments to judge what its students have learned, and lists its overall pass rate as 71.4 percent; Regis University's MBA, in turn, uses a 3.0 scale to rate student work from capstone courses on a set of rubrics, finding in fall 2008 that students averaged a 2.16 on communication and 2.50 on critical thinking.

A potential student may not have a lot to go on in judging his or her potential success in one of those programs based on that information, but the mere process of sharing results with one another has helped some institutional participants. "I've seen things in some of the other reports that make me think we're not performing at the level I think we should," says Karen Pedersen, vice president for professional studies at Kansas's Southwestern College, which is participating in Transparency by Design.

Going forward, the leaders of the consortium say they plan to add significantly more members and to continue to work through the program level outcomes and other issues on which they've yet to reach full agreement, such as how to report the proportion of students who make it through a given program -- a problem complicated by the fact that some of its institutions (like Charter Oak) focus specifically on degree completion, so that far larger numbers of students finish, but in many cases by completing a relatively small number of credits at the college.

They are also exploring a mechanism for the "missing piece" of the puzzle for adults -- employer satisfaction with the products of colleges' programs.

The site is a work in progress, and much remains to be done to improve it, says Offerman. But as with the other accountability systems that have emerged, he adds, "it's a step forward from where we are today."