Rigging the Rankings?
The president of University College Cork has asked all faculty members and other academic employees at his institution to each recruit three people from other universities -- people who "understand the importance of UCC improving its university world ranking" – to register to vote in the survey of university reputations conducted by Quacquarelli Symonds (QS), one of the major producers of international higher ed rankings.
The letter appears to violate the rules of the QS rankings, and the company is investigating whether it needs to take steps to remove from its peer review voters those who may be registering just to improve the ranking of University College Cork (UCC). At the same time, the incident points to what some see as an essential flaw in the QS rankings – namely that people volunteer to review other universities as opposed to being selected in a way that would prevent attempts to recruit voters sympathetic to a particular university.
Inside Higher Ed obtained a copy of the letter.
Michael B. Murphy, the president, wrote in the letter to faculty members that “the position of UCC in the world university ranking schemes could seriously impact the external perception that funding agencies, government and other partner institutions have of UCC, which in turn can impact on the staff and students of the University. It is essential that we seek to maximise our position in the league tables. To that end, I write to seek your assistance."
Murphy went on to say that while UCC has climbed in some rankings categories, it has had difficulty rising in the QS formula because it has not done well in the “academic peer review” that counts for 40 percent of the ranking. Academics volunteer for that peer review, and then are sent questionnaires about various institutions.
"The greatest opportunity for UCC to maintain its standing and to further improve its world university ranking position in this particular ranking scheme is to improve its academic peer review score," Murphy wrote. "In an attempt to raise UCC’s profile, I am requesting that all academic staff contact three non-UCC international academics and ask them to register with QS to complete the questionnaire…. It is essential that the academics you contact understand the importance of UCC improving its University world ranking."
Murphy included a sample letter to send to academic colleagues. He suggested the following wording, in part: "As you may know, there are a number of university ‘league tables’ which are of import in the perception and standing of universities worldwide. Like all universities, University College Cork is keen to raise its international profile, which we see to be of benefit to our staff and students and also to our international research and teaching partners."
It’s of course natural in political elections for candidates to try to excite their base about an upcoming vote. But the reason the UCC letter upset some of those who received it is that the QS rankings rules specifically bar any attempt to recruit peer reviewers with a request that they consider the needs of any one particular university.
The rules state: "It is not permitted, to solicit or coach specific responses from expected respondents to any survey contributing to any QS ranking. Should the QS Intelligence Unit receive evidence of such activity occurring, institutions will receive one written warning, after which responses to that survey on behalf of the subject institution may be excluded altogether for the year in question."
Ben Sowter, head of the QS Intelligence Unit, which oversees the company's rankings, said that he was sent Murphy's letter last week and is investigating it. He said that if University College Cork is like most institutions, "most of the faculty will likely ignore their president."
But he said that even with the QS system that allows people to volunteer to do peer review surveys, QS has ways to deal with any attempts to influence the voting. "We always make sure there are not swings [in high peer review scores] from one institution to another," and that QS will also view as a red flag a sudden increase in responses from a single country or responses that name only one university (in this case, one in Cork) as worthy.
Sowter said that if QS ends up eliminating some responses, it may well end up killing legitimate comments that also happen to be favorable to Cork.
"Just because a university has been a little bit naughty in responding to a survey doesn't mean it doesn't have quality," Sowter said.
He added that there is nothing wrong with universities encouraging people to sign up for the QS peer review survey, as long as they don't suggest favoring any one institution.
Sowter said that some universities in the past have encouraged people to participate in the survey, and sometimes the requests have raised questions about fairness. But he said that the Cork letter stood out because past letters haven't had "this degree of specificity."
The university released a statement late Friday denying that Murphy's letter was inappropriate.
"University College Cork is committed to quality improvement and one of the ways to achieve this is to benchmark itself against its international partners. The university invited staff to encourage their peers globally to register and participate in rankings surveys," the statement said. "It is the desire of the university that as many of its international partners as possible participate in the rankings surveys. The invitation to academics to participate in the surveys cannot be seen as prescriptive and the university anticipates that all academics who participate in the surveys will exercise their own judgment."
The university has placed considerable emphasis on raising QS scores -- and in a practice that has received scrutiny from both Irish and American journalists, the university is among those that pay QS to undergo a special evaluation that results in a special ranking about which universities can boast.
Phil Baty, editor of the international rankings of Times Higher Education (which used to produce its own rankings with QS, but stopped doing so several years ago, leading QS to become a competitor), said that the Cork letter and the QS system were both troubling. Via e-mail he said: "Global university rankings have become phenomenally influential in recent years – not only helping students to decide where to invest many thousands of dollars in tuition fees, but also in helping university leaders shape strategies and in helping governments to make multimillion-dollar funding decisions in some parts of the world. So it is absolutely essential that they are free from any hint of manipulation, and that they employ rigorous and transparent indicators."
Baty noted that the Times Higher rankings use "invitation-only" surveys for the peer review portion -- so universities can't recruit participants. "It is a fundamental principle that this survey is invitation-only. To ensure that we have a fair and balanced sample we do not accept volunteers to take part in the survey and we do not provide any sign-up facility.”
THE FULL TEXT OF THE E-MAIL SENT BY THE PRESIDENT OF UNIVERSITY COLLEGE CORK:
As you may know, there are a number of university 'league tables' which are of import in the perception and standing of universities worldwide. Last year, for example, over 20 million people viewed the QS World University Ranking with more than 600 newspapers and other media publishing the results.
During a period of financial constraint, the position of UCC in the world university ranking schemes could seriously impact the external perception that funding agencies, government and other partner institutions have of UCC, which in turn can impact on the staff and students of the University. It is essential that we seek to maximise our position in the league tables. To that end, I write to seek your assistance.
In previous years UCC has climbed the league tables; however, we have always been disadvantaged by one element of the assessment methodology, the academic peer review. As part of the assessment process the QS World University Ranking undertakes a peer assessment through a process of consultation with the global academic community. This involves the completion of an electronic questionnaire and accounts for 40% of the overall assessment. Over 46,000 academics participated in the 2012 QS World University Rankings survey. The greatest opportunity for UCC to maintain its standing and to further improve its world university ranking position in this particular ranking scheme is to improve its academic peer review score.
In an attempt to raise UCC’s profile, I am requesting that all academic staff contact three non-UCC international academics and ask them to register with QS to complete the questionnaire. The short registration form can be accessed at the following at the following link - academic sign-up facility. It is essential that the academics you contact understand the importance of UCC improving its University world ranking and that they complete the questionnaire before April 12th 2013.
To assist you in undertaking this task I have included at the end of this email a draft text for emailing to academic colleagues.
For additional information on the QS Global Academic Survey please go to the following link:
iu.qs.com/university rankings/academic survey responses
Thanking you in anticipation of your assistance
Dr Michael B. Murphy
Draft e-mail to academic colleagues.
Subject: University League tables
As you may know, there are a number of university 'league tables' which are of import in the perception and standing of universities worldwide. Like all universities, University College Cork is keen to raise its international profile, which we see to be of benefit to our staff and students and also to our international research and teaching partners. One key element of the assessment process is the academic peer review and one of the more prominent ranking schemes, The QS World University Ranking, places the highest score of the assessment on the peer review.
I would be most grateful if you would consider registering your interest to complete the QS survey at the following link academic sign-up facility. Once registered, QS will then ask you to complete a short survey which should take no longer than 5-10 minutes. The survey must be completed before April 12th 2013.
Search for Jobs