Alternative Measure of Success

Get any group of college presidents, assessment experts or education researchers together, and it’s not hard to get a consensus that the federal graduation rate is seriously if not fatally flawed.

According to the U.S. government, graduation rates are measured by the proportion of students who earn a degree within 150 percent of the expected time -- six years for a bachelor’s degree and three years for an associate degree. The formula counts only one group of students: first-time, full-time students. Not surprisingly, elite, residential colleges that serve well-prepared students do amazingly well by this methodology, routinely having rates in the 90s. But for many other colleges, the graduation rate is both irrelevant (they may have very few first-time, full-time students) and infuriating (the institution that takes full-time, first-time students that other institutions pass over may well be working harder and more effectively, but looks lousy by comparison to the wealthy institution that serves the wealthy.)

Much of the complaining comes from those institutions that believe the federal rate suggests that they aren’t doing well when they are. But some experts say that the real problem with the rate is the educational problems it potentially ignores. At a time when more and more students are part time, enroll in multiple institutions, and drop in and out of colleges all the time, no data effectively measure how institutions are doing with this cohort.

While such complaints are widespread, and a few institutions periodically suggest alternative measures, few of those who complain have committed to putting forth their own measure of accountability -- and using it.

The University of Alaska at Anchorage has just done that, and in the interest of getting critiques and inspiring others to develop their own new measures, the university wants to share its approach. Unlike most four-year institutions, Anchorage awards a range of associate degrees in addition to bachelor's and graduate degrees, so its efforts were designed with two- and four-year students in mind. It will continue to report its federal graduation rate, but for efforts to track its own performance, it is now using its alternative rate as well.

The Anchorage rate differs on just about everything from the federal rate: what counts as success, who is counted, and for how long.

If you want to understand why the federal rate is so irrelevant, according to Gary Rice, director of institutional planning, research and assessment at Anchorage, consider this statistic: Anchorage’s low federal rate (about 18 percent) is based on cohorts that represent only 3-5 percent of each year’s new students. Like many public colleges (and plenty of private ones), Anchorage’s students simply don’t fit the old-fashioned federal measure.

With the federal rate, “we’re looking at one tree instead of a forest,” said Rice.

So how to figure out a new system? Rice started off by seeing how long it takes Anchorage students to either succeed or fail at their educational goals. He determined that in a 10-year period, 95 percent of students have either had success (as they define it) or not. So the first decision was to use 10 years as a measure, not 6.

The next decision, Rice said, is to add back in everyone the federal rate excludes. Transfer students? Count ‘em. Part timers? Count ‘em. The cohort to be tracked over 10 years is everyone who has enrolled, not just those who fit the federal definition.

But perhaps the stickiest question is what to count as success. Here Anchorage has decided to track each new student on five questions:

• Are you back the next year?
• Did you transfer?
• Did you graduate with a degree?
• Did you graduate with an interim degree (short of your eventual goal)?
• Are you achieving grades that qualify as a success to stay on track to earning a degree?

The approach, Rice said, reflects the many reasons students enroll at an institution like Anchorage. For students who enroll with the purpose of earning a bachelor’s degree, that’s a measure of success. But a student who comes for that purpose and earns an associate degree has also been helped. And a student who enrolls with the goal of transferring and does transfer is also helped -- “and should be considered a success,” he said.

By also tracking what students’ goals are when they enroll, the Anchorage system comes up with two primary measures: 34 percent of those admitted in the last 10-year cohort measure met their educational degree goal, and an additional 50 percent “made progress” toward a goal.

Whether those are good figures is debatable, and Rice said that the Anchorage system could benefit from having other institutions -- especially those serving similar student bodies -- ask similar questions and run similar analyses of their data. He also acknowledged that other institutions or education researchers could differ on some of the measures -- such as the use of 10 years for the time frame, or some of what counts as success. (And Rice said he would be happy to consider other measures that different colleges put forward.)

But by creating a new system, Rice said, the university can now use the rates in meaningful ways. It may, for example, now look at various subgroups -- do part timers who want bachelor’s degrees succeed? Do students who have enrolled elsewhere first have more or less success than other students? Do new students who hope to transfer actually do so? Only by building data around assumptions that fit their students can colleges hope to ask such questions and to identify weaknesses where new approaches may be needed, he said.

Institutions like Anchorage have duties to all of those cohorts, he added, and so should be measured on them.

“Any single metric can’t recognize what we are charged with doing as an institution within society,” Rice said. “We’re not afraid of being held accountable, but we need to get on the national agenda the idea of creating new metrics.”

Until there is one, Anchorage will attempt to use its metric to focus attention where needed, he said.

Randy L. Swing, executive director of the Association for Institutional Research, said his organization has been briefed on the Anchorage approach and he hopes it extends the national discussion about measuring success. Swing said that he thinks some institutions with more traditional student bodies and educational goals might think Alaska has gone too far in broadening the measures of success. He also warned that getting this discussion going may not be comfortable for everyone. He noted that for some institutions, alternative approaches may expose very low graduation rates or success rates.

But he said that what Anchorage is doing advances a cause on which there is broad agreement: “Everyone knows the current definition doesn’t fit higher education.”