21st Century Snake Oil?

Texas is the latest state to jump on the higher education “performance funding” bandwagon, joining a group of other states that see this shiny technocratic strategy as the solution to the many problems surrounding higher education.

Despite the title of this article, I’m not opposed to the core idea: it obviously makes more sense to fund universities (the issues for two-year colleges are often quite different, so I’m leaving them out of this discussion) on the basis of course completions than on the traditional one of enrollments. Adding additional focus to outcomes by counting graduates in some way could also make sense, though the inputs and assessments would have to be fairly weighted -- not easy to do.

The problem with performance funding isn’t in the formulas; it’s in the unreasonably exaggerated expectations for results.

The promoters of performance funding — let’s call them the resultari to save space — are good and capable people with laudable goals whose belief that performance funding will effect major change in higher education appears to stem from two basic assumptions: 1) university leaders are inept managers; 2) both university leaders and faculty care too little about quality and success in undergraduate education.

Examining the Assumptions Behind the Push for Performance Funding

At their core, the assumptions of the resultari stem from a belief that universities suffer from internal contradictions, ones that inevitably flow from the self-interest of both faculty and administrators. Their view is that faculty are too focused on their research and that faculty and administrators are too often intent on enhancing institutional prestige through such things as unneeded programs (principally in the graduate and professional areas). These self-interested emphases, the resultari argue, distract from the real business of the university — graduating as many students as possible with the highest levels of learning and at the lowest possible cost.

Specific manifestations of this alleged administrator-faculty malfeasance include: a general unwillingness to make use of data on productivity and learning, innocence of where money is actually spent, lack of accountability for attrition, a resistance to using technology, and a failure to measure student learning. Performance funding, we are told, will change all this because it will force priorities back to where they should be. And then, when the death grip of administrator-faculty self-interest is wrested from the helm, the ship of higher education will steer a new and better course.

A History of Recent Times

If we accept the resultari’s assumptions, then they are certainly right about the profound impact that performance funding will bring. But are their beliefs about the state of the university correct? Let’s look at what’s happened in public higher education in the past few decades.

First, the resultari often cite as an illustration of inept university management the fact that tuition — even after declining state support — is increasing faster than inflation. There are many flaws in this argument, not least the point, recently cited in these pages, that it ignores basic economics: in a knowledge economy, the cost of all services that rely on highly educated individuals has been going up relative to that of the manufactured goods and other fruits of unskilled labor that comprise so much of the CPI market basket. And, compared to many businesses, higher education has less ability to improve productivity through outsourcing and automation.

Second, the problem of misplaced faculty and administrative priorities is mostly in the past, especially the 1970s and to some extent the 1980s. But universities have undergone radical internal changes in the last 30 years. I saw fine scholars turned down for promotion because of ineffective teaching at the university where I worked (Ohio State) as early as the late 1970s. I know from conferences, etc., that we weren’t atypical. The professoriate’s sense of entitlement, largely generated by the faculty shortages of the 1960s, has long since worked its way through the system. Today’s younger group of faculty is highly attuned to the importance of student learning, and even at major research-focused universities, this aspect of the job garners a significant proportion of their energy and creativity.

I’ve worked with several dozen presidents in both Ohio and South Carolina, and haven’t met one in the last 15 years who didn’t have student success as his or her highest priority — this statement most specifically includes leaders at large public research universities. And it isn’t just talk: these presidents have and are following up with incentives, programs, and rigorously balanced promotion and tenure criteria.

While the big and small universities changed early, it’s true that some mid-sized institutions continued to be a problem well into the 1980s, and in a few cases into the 1990s. Far too many of these institutions suffered the reign of what I call Louis XIV presidents (l’université, c’est moi), who focused on building prestige in the form of doctoral and professional programs that were typically of indifferent quality. But my observation is that this plague of academic locusts has passed and now shows up as just the occasional grasshopper.

There are several reasons why the Louis XIV presidents have met the fate of Louis XVI: state systems have woken up to issues of quality and duplication and cracked down — in many cases drawing public attention to the weak success of the doctoral and professional graduates in the job market; and trustees, often benefiting from the good advice of the Association of Governing Boards of Universities and Colleges, have learned to be vigilant. There was a time when the chairman of the English department could look around the table at a faculty meeting and say, “Damn! There are more than 20 of us now! Let’s offer the Ph.D.!” and see the dream quickly and easily realized. No more (though the unfortunate consequences of many of these earlier decisions linger).

Like generals whose philosophy is shaped by the last war, the resultari appear to be busy preparing to do battle with the universities of the 1970s and ‘80s. Unfortunately, once we’ve constructed the computer-derived version of the Maginot Line, with its walls of data and turrets of formulas, we’ll peek over the top and see there’s no one there.

Measuring

Another dimension of the misplaced priorities category, according to many critics, is that universities show their lack of appreciation for undergraduate learning by failing to measure it, particularly in general education.

I certainly agree that colleges and universities have been terribly deficient in assessment. We’ve operated on the “infectious disease” principle — the faculty are critical thinkers, the students are in contact with them, so…

No matter how you view the issue, higher education’s “take our word for it” approach as an answer to questions about student learning is unconscionable. That being said, I strongly believe that standardized testing isn’t a solution but a new problem. It’s an approach that creates powerful contradictions and also flies in the face of experience — notably, we know from the history of quality management that the “inspect the product at the end of the line” approach is certain to fail. I won’t attempt to recap the extensive literature in this area. My bottom line is that rigorous, improvement-focused, campus-level, non-standardized assessment (and reporting) is critical — and sufficient.

Ever More Data?

A core part of the performance funding push, and one that I find especially alarming, is the relentless, almost ritualistic, advocacy for more and better data. It seems that American education bureaucrats generally suffer from OCD (Obsessive Computational Disorder). A part of the problem in the higher education world is that the “more data” argument is being pushed by folks — I call them the datarati — who have spent their lives with numbers and, like the man whose only tool is a hammer and who therefore thinks everything looks like a nail, they naturally see data as a primary solution. But getting human-based data down to the decimal point is not necessarily a good investment, and this leads us to another problem of expectations.

Education datarati purport to draw their inspiration from business. One source is the late Harold Geneen, CEO of ITT, who is famous for saying that it didn’t matter the kind of business, if you knew the numbers inside out, you knew the company inside out. Geneen may have made other contributions to business, but this observation is on its face nonsense.

If you had applied Geneen’s thinking to IBM in the mid-‘80s (as its leaders did) you’d have seen an incredibly strong business numbers-wise but failed to notice that a major shift in technology was soon to bring the company to the brink of bankruptcy. There are lots of other business examples of this data-centric blindness. A modern government illustration is the recent federal Race to the Top competition, where data systems count for 9% of the total score but “turning around the lowest-achieving schools” gets only 10%.

Based on extensive personal experience, I find the allegation that universities don’t use data amazing. In the fat years of the ‘60s and to a certain extent the ‘70s, there were certainly gross inefficiencies resulting from lack of attention. But, after waves of budget cuts in recent decades, my observation is that presidents and senior staff are very knowledgeable about where the money goes and about where efficiencies can be found, and they do care deeply about attrition and do assign responsibility for it.

How about those longitudinal data systems, ones that will allow for the tracking of students from K-12 through college? I agree these are likely to have some value. But I strongly suspect the probable impact of these systems is seriously (albeit unintentionally) oversold. Why?

The longitudinal data approach anticipates a system that will be able to tell us, for example, that Mme. Maron’s 11th-grade advanced algebra class at Hogwarts High is turning out students who are weak at college algebra. So far so good, but I suspect that in most cases when the Datarati Swat Team arrives at the school they’ll find that leaders were already well aware that Mme. Maron was a concern. Thus, the second issue, which is that it will be much easier to find the sources of problems than to resolve them. An automobile manufacturer detecting faulty parts can force suppliers to revise their practices and, if that doesn’t work, go to another firm. But, as we are reminded nearly every day in the papers, it’s not so simple with teachers (quite often because it’s not really their fault).

I also believe we are seriously underestimating the cost of these new data palaces. Projects to connect disparate information systems have an amazing ability to always cost more than initially projected — usually a lot more. And that’s just putting them in place. I’ve not seen evidence that people are thinking carefully about the long-term costs of maintenance and analysis. There’s no denying there can be value in the systems, but balancing that against true costs isn’t being done.

Like the resultari, the datarati are winning. My office is now making its first hires in two years and they will all be for a federally funded longitudinal data system. After years of bleeding vital staff positions, this project is not anywhere near the top of our priority list. Worse, when the federal dollars are gone, the first of any new state monies will likely have to go to maintaining the effort. In short, the new data system will be the devourer of rational priorities; I’m thinking we’ll call it Grendel.

Failure to Use Technology

Another issue to consider is whether more use of technology will sharply lower the cost of instruction and in consequence, as a great many reformers argue, contain costs and make higher education much more affordable. The idea of transformational change through technology is certainly an appealing one, but is it based on solid fact or does it contain an important amount of wishful thinking? Unfortunately, I think it’s the latter. I’ll cite three reasons.

First, instructional technology buffs have a habit of using “new” as an explicit or implicit modifier. But the reality is that the technology that’s around today has been there for quite a while. Faculty were using computers to offload drill from the classroom 35 years ago and such use has been pervasive for at least a decade and a half (remember that the Web and HTML have been around for this long). Given the history, and the fact that today’s technology is better but not dramatically different, we don’t have evidence to support the idea that breakthroughs in transforming learning and productivity through technology are on the horizon.

Carol Twigg, head of the excellent Course Redesign effort at the National Center for Academic Transformation, argues that the potential is there with existing technology, but laments that faculty want to keep technology as an adjunct to instruction and will not take the major steps needed to allow it to lower costs. Twigg knows the topic better than anyone else, so I won’t dispute her conclusion about faculty reluctance. On the other hand, I really don’t buy the idea that their recalcitrance simply reflects a desire to preserve their own jobs. Instead, I think the faculty are generally right in seeing limits on replacing people with machines.

This leads to the second point about technology — the human side. Talk to people who deal with students and they’ll tell you there’s a psychological breaking point — most people like to work directly with other people and don’t want to do everything on the computer. Highly motivated adults are certainly an exception (and I’m helping to develop programs based on that belief). But it’s not the same for undergraduates. For example, a recent study showed that three-quarters of those surveyed believed that “online courses are not as appropriate for traditional-age college students, who they believe are better served by classroom courses.”

A third dimension of the potential of technology can be seen in the evidence about staffing. All of the college and university leaders I’ve talked to say that, with few exceptions, online courses are more expensive than the classroom equivalent because they require more instructor time, therefore cutting section size. This also appears to be the case in the for-profit sector, where institutions typically charge more for online education than for the classroom equivalent — about 30% more, in the case of the University of Phoenix. Leaders at this university told me the same thing as their public peers — in consequence of the large number of one-to-one vs. one-to-many communications that occur online, instructors can handle fewer students per section and costs therefore go up.

If you add these three things — 1) the already existing proliferation of technology in instruction; 2) the natural limits to human-computer interactions (in this vein I encourage people to read E.M. Forster’s powerful short story "The Machine Stops"); and 3) the fact that experience so far does not suggest technology will lower costs (indeed much of the evidence is in the opposite direction) — then it’s hard to be as sanguine as the resultari about more technology leading to sharply lowered costs.

Put another way: Will technology give us further improvements in learning while also helping to reduce costs? Sure. Will the change be transformational and make college notably more affordable for traditional undergraduates? No.

What Do Proprietary Colleges Tell Us?

Critics and supporters of higher education alike tend to forget that public and private colleges and universities aren’t the only game in town. There’s a growing for-profit sector. In considering the management/efficiency issue, what can we learn from them?

Let’s begin with an important question: Why don’t the for-profits compete on price?

The easy answer is that the public has an accepted price of what it is willing to pay for education, so the proprietaries are able to use savings from greater efficiencies to add to their profit rather than to lower their prices. It’s true that in luxury markets “low price” doesn’t compete well, or not totally (BMW and Lexus do advertise discounts). So, if higher education is presumed to be this kind of market, a low-price alternative probably wouldn’t work — even if you could show comparable learning outcomes (“Learn-A-Lot U — Just as Good as Harvard at Half the Price”).

But this argument has several flaws. First, while the for-profit sector has long been doing well for investors (with the usual market hiccups in stock prices), I haven’t seen evidence that they are making the kind of profits that, if applied to cost reduction at public institutions, would equate to significantly lower tuition — even in the face of lower state support. Again, remember this kind of change is what some are saying we would get if only public universities would operate efficiently.

Second, the for-profits compete against each other to a significant extent, and it would be natural for someone to lower prices at least temporarily to grab market share (as the luxury vehicle makers do). At least that would be true if someone did in fact have much lower costs.

Third, you really ought to see price competition in the lower part of the market — technical programs — where luxury effects don’t apply. But it’s not there. Indeed, after several decades of working with the technically focused for-profit institutions, my observation is that a major part of their marketing sell is that they are “hands-on,” and that students will be able to work closely with accomplished practitioners. Emphasizing personal instruction by highly qualified individuals in a knowledge economy is not how you lower costs.

Conclusion

Here are some suggestions for resultari, datarati, et al.

Do go ahead and, over time and with appropriate care for balance, implement formulas that primarily reflect outcomes — especially course completions and, to the extent reasonable and practicable, graduation. This only makes sense; funding formulas should always have been constructed on outcomes. But…

Do not describe the change to outcomes as a major structural improvement in higher education that will bring great benefits to students and the public. If the formulas are carefully weighted to reflect the varying inputs — principally in terms of student preparation and background in education culture — we’ll find that, with few exceptions, differences between institutions will be very small and not statistically significant. Those using crude data to support winner-loser formulas point out that right now some universities are doing better than others in, for example, graduation rates. Of course. In part that’s a function of different inputs. If we have just a few simple measures to compare disparate entities, we’re going to get simpleminded results that won’t convince anyone — garbage in, garbage out.

Also, even after you control for inputs, variation will exist. In any environment with a lot of players it’s always the case that at any given time some are doing better than others. That’s not a reason to conclude, as resultari seem to do, that the ones doing less well are lagging because they really don’t care and aren’t trying hard. The punitive spirit of formulas designed to create winners and losers serves no one well, not least in the gratuitous damage to students at the allegedly “underperforming” institutions.

Do invest to improve performance. Universities are more deprived of spare funds than at any time in recent history, with the result that start-up monies for new faculty-led initiatives, such as rethinking strategies in general education, are almost impossible to find. State-level incentive programs, ideally ones that are peer-reviewed, could be used to further encourage faculty creativity in both improved learning and improved productivity (Lumina’s Tuning USA project appears to be a great example of what should be done). Investments of this kind will have the added benefit of setting a positive tone.

Do not add more data unless state and campus leaders agree they meet two key criteria: 1) Will the new information lead to better decisions in the real world?; 2) What is the opportunity cost — are there other things we could do with the money that would have greater impact? It’s curious that the “culture of evidence” crowd seems to think there’s little requirement to provide evidence for the value of new data before it’s gathered. A better approach for the datarati would be to shift gears, focus only on what can be shown to matter, and help us avoid creating a voracious new consumer of educational resources without any clear evidence of proportional benefit.

Finally, do change the tone. I agree that there are actions we can take to suppress the cost spiral — among the things that appeal to me are dropping uncompetitive doctoral programs, more shared and/or outsourced instructional and operating services, and in some cases even merging of institutions — but I don’t believe the best way to advocate for change is by implying that only the self-interested will disagree with me. We need fewer leaders of numerical lynch mobs and more people who are willing to offer a truly thoughtful challenge along the lines of, “We recognize these are good and capable people leading our universities and our faculties, and that they are going against some really tough problems. But, as with any organization, there’s more we can do and we have some suggestions.”

I can’t quantify the difference this change in tone would make, but I think it would be huge.