Advertisement

Advertisement

News, Views and Careers for All of Higher Education

Another Rankings Fiasco at ‘U.S. News’

Portland State University’s engineering college has been transformed into “a national and international academic and research institution.” The excellence of the college “illustrates how state investments in higher education can increase programmatic capacity.” The university’s electrical engineering department is so good that it’s in a “top 10″ listing with such institutions as the Massachusetts Institute of Technology, Stanford University and the University of California at Berkeley. “We knew PSU engineering was significantly under-ranked. But Top 10? Wow! It made my day. Go PSU!”

Those quotes are all from a press release that the Oregon university rushed out Friday, upon learning that the 2008 U.S. News & World Report guide to graduate programs had ranked Portland State No. 9 in electrical engineering. You won’t find the press release on the Portland State Web site any more. It turns out that Portland State doesn’t make the Top 10 — or the top 70 for that matter.

The stand-alone guidebook that U.S. News sent to newsstands nationwide last week (the version designed for sale throughout the year and purchased by college counseling centers, not the weekly magazine) turns out to have included Portland State and the University of Texas at Arlington incorrectly in the top 10 for electrical engineering — when their rankings are actually so low that the magazine doesn’t give them out. Portland State’s No. 9 slot really belongs to Carnegie Mellon University. Texas-Arlington’s tie for the No. 10 spot really belongs to another University of Texas campus — the flagship, in Austin.

It was Austin that first notified U.S. News that something might be wrong when officials at the university were confused about how the institution’s top-ranked electrical engineering program could have been overlooked.

How did Portland State and Arlington get their incorrect ranks?

Robert J. Morse, director of data research for U.S. News, blamed technology errors by the company hired by the magazine to conduct its surveys for the graduate rankings. “There was an error in data output,” he said.

Morse stressed that the error was uncovered before the magazine listings of the rankings or the magazine’s Web site went live, so those listings are correct. Next week’s issue of the magazine will contain a correction about the rankings that appear in the stand-alone guide, he said.

The rankings in question are based entirely on a reputational survey conducted of department chairs in the field, who were asked to rank the programs on a scale of 1-5, with 5 being the best score. To be ranked in the top 10 in electrical engineering, a department needed a score of at least 4.1 this year. Morse said he would not reveal the scores of the two universities incorrectly listed in the top 10, but they are below 2.5, since that is the lowest rank published by U.S. News. Eight universities’ electrical engineering departments tied for No. 73 on the list with that score.

One irony of the mistake is that a common criticism of the reputational surveys used by U.S. News is that they tend to favor programs that have existed and done well for a long time. Institutions like Portland State and UT-Arlington — that are respected but don’t have decades of experience as research powerhouses — are very unlikely to see their departments have a rapid rise in a reputational survey.

So how is it that U.S. News didn’t ask questions when Portland State displaced Carnegie Mellon (not to mention Princeton University and the University of California at Los Angeles, among others) and UT-Arlington replaced UT-Austin? Morse said that the magazine relied on the computer printouts it received. “It’s bad that we did this. There’s no excuse,” he said. Morse characterized the mistake as “not a big deal” in a big picture way, but acknowledged that it probably was to the universities involved.

UT-Arlington didn’t issue any announcement about the situation and a spokesman said Tuesday that he was just learning now about its meteoric rise and fall in the electrical engineering rankings.

Portland State had to rush out an e-mail to its press list, advising recipients to ignore the earlier release and to contact Morse at U.S. News to find out why it happened.

Joan Barnes, assistant vice president for communications at Portland State, said that educators there were “disappointed at this unexpected turn of events,” but not discouraged. “We’re redoubling our efforts to serve Oregon with confidence that increased national recognition will follow our success.”

Scott Jaschik

Got something to say?


Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Advertisement

Comments

Typos and Credentialism

What’s the big deal about some typos? They happen all the time.

The difference is the meaning they assume in hyper-competitive, credentialist environments. Without this context, they would just be typos, and no one would be writing news stories about them.

Cynic, at 8:25 am EDT on April 4, 2007

typo or point?

Robert J Morse, Data poobah for USNews said “There was an error in data output.” I first noticed the euphemistic rhetoric in the phrase “data output” because it seems to minimize and objectify the error when the whole point of the publication is numerical ranking. I think we do way too much poorly researched educational ranking in general, and I have never given the USNews ranking much credit for its research methods and validity. By “correcting” a slight numerical error and calling “data output,” USNews may actually be promoting itself. What better way to hide the fallacies of its ranking process than to admit a small error when the erroneous problem is really much larger?

Will Hochman, Associate Professor at SCSU, at 9:50 am EDT on April 4, 2007

Big deal

The big deal is that the folks at U.S. News have gone out of their way to make these rankings a much bigger deal than they ought to be. I have to admit to experiencing some mean-spirited glee when watching them stumble like this.

Anonymous, at 10:00 am EDT on April 4, 2007

What Really Matters

Does Portland generate excellent engineers? Absolutely. Does Carnegie-Mellon generate an occasional dud? Undoubtedly.

People pick schools as a matter of convenience and comfort — why else would “in-state” students dominate virtually every student body — even on championship athletic teams!

The educational difference between 1st and 101st is minuscule. We need to stop obsessing over the rankings to be used by a handful of wealthy kids whose priviledge will ensure their success regardless of the college they choose.

sb, at 10:00 am EDT on April 4, 2007

academic rankings

Newsweek’s academic “rankings” seem arbitrary at best and specious at worst and don’t seem to be taken all that seriously by anyone in academia. Who, besides law firms, really cares where you went to school? It recalls the old joke which goes, “What do you call the person who graduated last in his med. school class? Yeah, Doctor.

dg, at 10:25 am EDT on April 4, 2007

Missing the Real Point

Amusingly, you are all, including the author of the piece, missing the real point: US News rankings would have NO power whatsoever if institutions did not respond the way PSU did. That is the story.

Tod, at 11:00 am EDT on April 4, 2007

data output error

There is probably more than one data output error (not to mention the data input errors!).

This whole ranking system is a prime example of how statistics can be manipulated and misused.

A, at 11:25 am EDT on April 4, 2007

You’re joking?

You mean to tell me that this is a story? How about doing some investigative reporting on how instituition manipulate the data to better position themselves in the ranking. Interesting to see the US News rep repent as if something grave and serious had just slipped by the radar. But...like everything else, the public are no more than sheeple and need this type of thing to lead them!

Bruce, at 12:10 pm EDT on April 4, 2007

Lack of US News Internal Controls

The escape of incorrect information is problematic.

The public is hungry for evaluations other than which school won the latest NCAA sporting event.

Perhaps US News should voluntarily submit to a CPA type audit of their work. In any envent, its internal controls would not pass GAO muster.

Quiizzical, at 2:30 pm EDT on April 4, 2007

The piece on our electrical engineer rankings was fair. But you should talk to whomever writes your headlines. “Another Fiasco” seemed pretty far over the top. I’m not sure what “another” refers to — we haven’t had any recent mistakes. And to suggest that two mistakes in a data set of 10,000 entries is a complete failure does stretch the meaning of the English language.

Brian Kelly, Executive Editor at U.S. News & World Report, at 3:06 pm EDT on April 4, 2007

I like Fiasco

I like Fiasco, and “another.”

I’d also like to suggest that Portland talk to Reed about adding themselves to their 3/2 program for Electrical Engineering. They may find students there are more interested in quality education than ersatz rankings.

John Burnette, Asst Director College Counseling at Savannah Country Day School, at 9:50 pm EDT on April 4, 2007

US News Ranking

What is stunning is the failure of US News to maintain a staff with sufficient commonsense and expertise to review such data and realize something was really “off.” Isn’t there someone responsible to check this year’s list against last year’s? To have some knowledge of the schools who have historically ranked well? To review and question the data? Given the extent to which institutional reputations are at stake—and US News promotes and encourages dependence on their rankings—it seems irresponsible not to carefully vet the information before release.

Andrea Bernat, at 9:31 am EDT on April 5, 2007

Ranking What?

What does the U.S. News really rank when they rank schools? They do not measure output (learning outcomes or role performance of graduates), but only certain measures of input. It is like ranking automobiles based on how much the manufacurer spends per vehicle rather than how the vehicle actually performs on the road. If the U.S. News is interested in measurig the quality of education, they should follow the example of Consumer Reports.

Why is the U.S. News in the business of ranking colleges, any way? What is the connection between their mission and higher education?

Mathew

Mathew, at 10:01 am EDT on April 6, 2007

In response to the previous question: US News is NOT interested in providing clear outcome-based rankings. They are interested in selling magazines. I heard a keynote given by Mr. Kelly (I believe) in which he played the ‘we’re just giving the people what they want’ card and blamed higher ed for not providing a clear bottom line of what benefits education provides. These things are not easily quantifiable (i.e. did the student get a job associated w/their degree or did the student’s interest in that field decrease but the skills and experiences in college provide them another avenue in the job market; how do you quantify a graduate’s ability to interact with people from different backgrounds that is associated with their college experiences?; etc.)

I believe they take the easy way and use such traditional (see: not all that relevant) measures as GPA and test scores and of all things impressions of college presidents! that the rankings are nothing but a good old boys club that will never show the true value of an education one can get from a less historic institution.

I’m anonymous because my institution still plays this game and I wouldn’t want my personal comments to somehow cause retribution to my school. (See this for a related topic: http://www.washingtonpost.com/wp-...cle/2007/03/09/AR2007030901836.html)

Not a fan, at 4:22 pm EDT on April 6, 2007

What a great April Fools joke!

US News fan, at 4:20 am EDT on April 9, 2007

US News

I noticed the same type of errors in the US News Undergraduate rankings which listed several open admission campuses in the West region as highly selective.

Mia, at 10:51 am EDT on April 9, 2007

Advertisement

 Jobs Related to Another Rankings Fiasco at 'U.S. News'

or search for jobs directly.

Faculty, Economics
Lone Star College System

Located just north of Houston, Texas, our five campuses serve 1,400 square miles. Our student enrollment is nearly 50,000 in ... see job

108985 — Instructor: Sociology — FT
Greenville Technical College

Come teach Sociology! see job

Adjunct Faculty, English/World Languages: Composition, Developmental Reading, Developmental Writing
Joliet Junior College

Joliet Junior College is located in the Chicagoland area. JJC offers over 100 degree and certificate programs in the arts and ... see job

Adjunct Instructor, Aviation Management
Lone Star College System

Located just north of Houston, Texas, our five campuses serve 1,400 square miles. Our student enrollment is nearly 50,000 in ... see job

Associate Researcher
University of Wisconsin—La Crosse

The University of Wisconsin – La Crosse invites applications for an Associate Researcher as part of a two-year National ... see job

Instructor (HVAC)
Columbus State Community College

Columbus State Community College invests in employee development by providing numerous resources, partnerships, training and ... see job

Project Manager
Harvard University

Project Manager see job

Communications and Change Management Lead
Harvard University

Financial Administration Systems Solutions

Duties And Responsibilities: For the Harvard ... see job

Coordinator for Academic Intervention
University of California, Riverside

The University of California Riverside invests in your future through employee training and career development, access to ... see job

Assistant Professor of Arabic
Portland State University

Assistant Professor of Arabic, tenure track, beginning September, 2009. Qualifications: PhD in Arabic literature or ... see job