Measuring the Dreaded 'P' Word

Continuing its efforts to identify and encourage new ways to measure higher education performance, the Delta Project on Postsecondary Education Costs, Productivity and Accountability issued a new report Thursday designed to gauge how successfully public colleges in various states use their available resources to produce graduates with credentials that are valued in their markets. The report, "The Dreaded 'P' Word: An Examination of Productivity in Public Postsecondary Education," also ranks states using the measure.

Like most of the project's work, the new study aims to give policy makers and university leaders more tools to rebut the argument that it is difficult if not impossible to measure the effectiveness of colleges and universities. The report, by Patrick J. Kelly, senior associate and director of the National Information Center for Higher Education at the National Center for Higher Education Management Systems, won't be appreciated by all; the author acknowledges many of its limitations, notably that it focuses only on colleges' degree production (a critique leveled against the last Delta-produced study) and that it makes no effort to gauge the quality of the credentials produced.

But at a time when policy makers, led by those in the Obama administration and in Congress, are focusing intently on higher education's need to significantly increase the number of Americans with a higher education credential of some sort, the focus seems appropriate, Kelly argues. " 'Business as usual' -- at current costs per degree -- would require substantial new investments," Kelly writes -- investments that are unlikely to emerge, despite the federal government's current plan to provide some additional financial incentives to help states in their efforts.

So "[h]ow well higher education institutions perform with the resources they have, and how they can improve performance with few or no new resources, are uncomfortable questions that are here to stay," he adds.

Kelly's approach aims to improve on the traditional methods of measuring public colleges' (and, by extension, state higher education systems') productivity, such as graduation rates (which ignore part-time students and those who transfer without having earned an associate degree) and the number of credentials produced per full-time-equivalent student (which ignores the value of the degrees and certificates produced and does not account for how much the institutions spend to produce those credentials).

Using publicly available data (in hopes of making it easy for policy makers to replicate), the report starts with the total funding for each state's public colleges, combining state and local appropriations and tuition and fee revenues, which account for the vast majority of operating funds for state institutions over all. The analysis then weights the numbers of degrees and certificates that a state's colleges award (by level) by the median earnings associated with them in the state's employment market. Higher degree levels are weighted more heavily, as are credentials and science and technology fields.

An average "weighted" value of a credential is calculated for each state, and divided into the total public college funding, producing a "productivity" figure that amounts to the total amount spent per certificate and degree.

Based on Kelly's analysis, Florida, Colorado, Washington, Utah and North Dakota have the lowest costs per weighted credential awarded, while "the least productive states -- those with the highest cost per credential -- are Alaska, Wyoming, Delaware, Rhode Island and Connecticut."

Importantly, he writes, "there is no evident relationship at the state level between resources and performance: higher levels of resources do not result in more credentials awarded per student." Some of the best performing states, including Colorado, Utah, Florida, Oklahoma, and Washington, receive comparatively little in state and student resources, while some of the least productive states are those with some of the highest levels of resources.