Gaming the Rankings

Last week, Clemson University researcher Catherine E. Watt presented the extensive efforts her university had taken in the last few years to raise its U.S. News & World Report ranking at the Association for Institutional Research. Clemson’s clearly stated goal is to make the U.S. News top 20 public research universities, and Watt was unusually frank about the many actions a college can take to increase its numbers for individual indicators. She stirred up the Clemson administration and other higher education worthies with her talk. Reports about her presentation were e-mailed at a great rate — we’re shocked, shocked that gaming is going on here!

It follows as the day does the night that any time substantial rewards are at stake as a result of measuring performance, those being evaluated will do their best to game the system. The greater the payoffs, the stronger the impetus to game.

Colleges and universities are no different from any other organizations competing for rankings developed from performance measurement. In Clemson’s case, the university president, James F. Barker, has made becoming a top-20 public research university a cornerstone of his presidency. His own performance in the form of a “report card” to the board of trustees ties university goals and progress to the U.S. News rankings. Many other university presidents are striving for the same goal. The 2005-2008 employment contract of Arizona State University president Michael M. Crow rewarded him with a $10,000 performance bonus if his school moved up the U.S. News rankings into the top 120. Changes in reporting, accounting, and practice have undoubtedly been undertaken across the country as presidents pursue the same goal of higher rankings.

Any performance measure is ripe to be gamed. The percentage of alumni giving is a measure worth 5 percent of a ranking in U.S. News. A few years ago, Albion College made its own stir in the higher education rankings world when it increased its percentage of alumni making donations with the stroke of a pen. As The Wall Street Journal reported, the college recorded a $30 donation from a graduating senior as a $6 alumnus gift for the next five years. Clemson, in its systematic approach to raising its rank — “no indicator, no method, no process off limits to create improvement,” as Watt stated — solicited alumni donations in such a way as to increase their giving rate: Alumni were encouraged to give as little as $5 annually.

Watt, according to reports, literally drew gasps from her audience when she revealed that when Clemson administrators fill out U.S. News reputational rankings survey, they rate other universities lower than Clemson across the board. Why not? Reputation accounts for fully 25 percent of a school’s ranking score. Watt’s statement that she was confident that other colleges do the same is perfectly plausible.

Inside Higher Ed reported Monday that the University of Southern California inflated the number of National Academy of Engineering members on its full-time, tenure-track faculty. Because the number of NAE faculty is a criterion for U.S. News rankings, USC has good reason to include NAE faculty who are not full-time or tenure-track.

If we step back from higher education, we will see the same dynamic of gaming a performance measurement system in many other spheres. Hospitals receive “report cards” that measure their performance in many areas, including their mortality rates. A little thought reveals the easiest way to improve the mortality rate is to keep terminally ill patients from being admitted to the hospital in the first place or discharge them prior to death. In fact these events do occur. Nonprofit hospitals receive large tax exemptions but are expected to provide charity care to indigent patients in return. Their substantial tax benefits are currently being scrutinized in the courts and in Congress, so hospitals are certainly scrambling to alter their accounting procedures to increase their charity care levels.

No one should be gasping at the notion that colleges and universities respond to the importance of the U.S. News rankings by making changes and manipulating their reporting of data. Some changes — such as decreasing class size or the faculty-student ratio — may be beneficial in the pursuit of improved education, while others only improve the rankings metrics.

The rankings matter to the colleges and universities! Boards of trustees want to see their institutions move up the ranks and put pressure on the president and the administrators to raise the ranking. As long as presidents are themselves being evaluated by their success in raising the rankings, the sorts of systematic gaming of the rankings undertaken at Clemson will continue.

A few colleges and universities have opted out of the U.S. News rankings system. But most will strive to get ahead in a tough economic time in a very competitive industry. Unfortunately colleges and universities will be spending their time and money on ranking gamesmanship instead of trying to improve education and research. Gaming the rankings system is here to stay.