New Ranking Rules

Quacquarelli Symonds, one of the major groups conducting international rankings of universities, has banned universities from recruiting people to participate in the peer review surveys conducted for the evaluations of institutions.

QS accepts academic volunteers to participate in its rankings reviews. Up until now, QS has permitted universities to recruit volunteers, provided that the institutions don't suggest how they should evaluate the universities.

The action by QS, as the company is known, follows the news that the president of University College Cork sent a letter to all faculty members urging them each to ask three people they know at other universities -- people who would understand the university and its need to move up in the rankings -- to participate in the QS process.

Via e-mail, a spokeswoman for QS said that the changes were prompted by the action of University College Cork (known as UCC), but she declined to say that the Irish university had broken any rules.

"Until now, in the interests of growing the representative sample on which the rankings are based, QS has encouraged institutions to promote participation in the survey, but has prohibited soliciting specific responses. Technically, UCC [has] not infringed this existing policy, but their fine interpretation has revealed it to be less explicit than it was intended to be," she said. "For the avoidance of doubt, QS will discard any further sign-ups to the registration link [that] have emerged from the independent actions of participating institutions or any other source."

The QS policy now reads: "It is not permitted to independently promote participation in QS surveys, nor to solicit or coach specific responses from expected respondents to any survey contributing to any QS ranking. Should the QS Intelligence Unit receive evidence of such activity occurring, institutions will receive one written warning, after which responses to that survey on behalf of the subject institution may be excluded altogether for the year in question."

Further, it says that "in cases of recurrent activity of this nature will first apply a score penalty to the survey index in question and may consider disqualifying an institution from the ranking altogether. QS runs sophisticated screening analysis to detect anomalous patterns in response and routinely discards invalid response [and] any attempt to manipulate the results or to solicit responses, may result in the disqualification of all responses for that survey for that year, invalid or otherwise, where the source cannot be verified as entirely independent."

QS has also published a statement on why it believes its rankings "cannot be effectively manipulated."

It is unclear whether the changes by QS will go far enough to satisfy critics of the way it recruits peer review participants.

Philip G. Altbach, director of the Center for International Higher Education at Boston College (and a blogger for Inside Higher Ed), said via e-mail, that "QS has an unfixable problem in its ranking methodology. There is no real control over who can participate in the rankings."

He explained: "QS rankings rely more on reputation than the other two main rankings -- Times Higher Education and the Shanghai-based Academic Rankings of World Universities (which does not use reputation at all). One wonders how the QS Intelligence Unit will actually monitor data? It is just not realistic to expect the even the most experienced academic leaders will have a reliable grasp of which universities are, in fact, the best ones, especially when 'best' is not carefully defined. And who would expect them to be completely objective even if they did have access to information -- most would like their own institutions to be ranked at the top."