Saturday
August 30, 2008
Advertisement
Advertisement
News, Views and Careers for All of Higher Education
April 9
‘U.S. News’ Adds Surveys That Could Alter Methodology
U.S. News & World Report may be on the verge of significant changes in its methodology.
Related stories
The magazine has sent out surveys to 1,600 high school counseling offices asking them to evaluate colleges, and the results may be used in next year’s rankings. Or not. U.S. News isn’t deciding how to use the high school data until after the results come in. In addition, the magazine is asking a series of new questions of college presidents, having them identify “up and coming” colleges, inviting them to offer suggestions on changing the relative weights the magazine uses in its rankings, and giving them a series of possible additional measures and asking which should be added to the methodology.
Taken together, these changes could result in significant changes to the magazine’s controversial and much debated methodology. And the magazine — following a year in which more college presidents criticized it and fewer participated in its surveys — is reaching out to educators for advice. But already the possible changes are themselves causing more controversy. Some high school counselors are calling for a boycott of the survey, and a leading critic of the rankings says that the possible changes show once again how flawed the rankings are.
“I think these changes reveal that they are acknowledging the lack of precision and the impropriety attached to this instrument,” said Lloyd Thacker, president of the Education Conservancy. He said that at a time that the failings of the rankings have been exposed, U.S. News is seeking to involve more people “to give this the appearance of authority.”
The new survey of high school counselors is going to the college advising offices at the 1,600 high schools that are the winners of another of the magazine’s rankings, “America’s Best High Schools.” Half of the high schools will be asked to rank the 266 institutions that U.S. News considers “national liberal arts colleges.” The other half of counselors will be asked to rank the 262 institutions that the magazine considers “national universities.” The rankings will be on a five-point scale. Counselors will be told to rank only colleges they think they know something about, but the basic format is subjective in the same way the magazine’s presidential survey is subjective, allowing different people to rank in different ways.
Robert Morse, who runs the magazine’s college rankings division, said that high school counselors have over the years asked the magazine why their views were never sought, so the magazine decided it was time to ask them. Morse said that the magazine would publish the results in some way, and might — but might not — add the results to the actual rankings of colleges.
Many critics of U.S. News have charged in the past that when the magazine changes its methodology, it usually does so only in ways that keep certain colleges (Harvard, Yale, Princeton and Stanford Universities; and Amherst, Swarthmore and Williams Colleges) at the top, because many in the public expect them there. Asked if it wouldn’t reinforce that criticism for the magazine to conduct a major national survey and decide whether to use the results from it only after those results are in, Morse reiterated that the magazine was trying to be responsive. He also said that the magazine wanted to be sure it received good response rates from different parts of the country.
On Tuesday, as word of the high school survey reached the discussion list of the National Association for College Admission Counseling, some were pointing out that what many high school counselors have been saying for years isn’t that they want to be asked to rank colleges, but that they view the U.S. News rankings as counterproductive to their efforts to help students make thoughtful choices.
One counselor posted a message making fun of the magazine’s claim that its readers needed the knowledge of counselors. “I suggest that everyone who gets one of these things sends it back with a letter that says we are refusing to participate in this ridiculous exercise. I do have knowledge that is extremely valuable to U.S. News readers — Don’t buy this magazine.”
Another counselor suggested a draft pledge for counselors to sign in which they disavow the new survey and say: “I believe that ranking colleges in the way that U.S. News has done over the past years has contributed greatly to a climate of anxiety that has exploited the very children I have given my professional life to serve, guide, and educate.”
Morse, of U.S. News, said he hoped that counselors wouldn’t be influenced by the views of colleagues he characterized as having “anti-U.S. News attitudes.”
The magazine also has just sent out this year’s survey to college presidents asking them to rank institutions. This year’s survey also includes additional questions:
- Presidents will be asked to name up to 10 colleges they think are “up and coming.” Morse said that this is a response to complaints that the magazine’s reputational survey — among the most controversial parts of the rankings — is slow to respond to improvements by colleges.
- Presidents will be asked for advice on changes the magazine should consider in its methodology.
- Presidents will be given a list of possible additional categories on which colleges could be ranked, and which could be used in the methodology for best colleges. These items include: ethnic diversity, admission yield, economic diversity, total endowment per student, learning outcomes, job placement data, and spending efficiency.
Morse said that the magazine has not decided whether it will release information on what the presidents say.
Why consider all these changes? Morse said that the effort was “partly in response” to the criticism of college presidents, some of whom have pledged not to fill out the magazine’s surveys any more. But more broadly, Morse said that “we want to make what we are doing better.”
Thacker said he was unimpressed by the effort. The rankings play a role that is “educationally meaningless” and that distorts the decisions of applicants and colleges, he said. But by considering all of these methodology changes, Thacker said, the magazine was admitting what a “fake science” its work is. The magazine, if it changes its methodology, will be admitting the lack of validity of rankings it pushed on high school students and their families for years, he said.
“The magazine is just trying to perpetuate what it has as more people ask questions,” he said.
Further, he said that changes in methodology can’t fix a central problem: “The problem with the rankings is that they are rankings. I do not believe you should ordinally rank a college experience.”
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
Comments
Thought You’d Want To Know
Here at the Manley Institute, we are very enthusiastic about this wonderful innovation by U.S. News and World Report ... and we have made a special note of it. The Manley Institute ranks news magazines once each year, mostly on the basis of (1) “reputation,” (2) several measures that are neither valid nor reliable indicators of readers’ knowledge of current world affairs, and (3) a mind-bogglingly naive objective function. This year’s top ten are ...
1. Economist
2. Mother Jones
3. Maclean’s
4. Newsweek
5. Reason
6. Time
7. Insight
8. FrontPage Magazine
9. U.S. News and World Report
10. The National Enquirer
Just thought you’d want to know.
Frizbane Manley, at 9:00 am EDT on April 9, 2008
What High School Counselors (Don’t) Know
High school counselors—or the ones with whom my sons worked—don’t seem to know much about many colleges. They may have the viewbook information on brand-name and state flagship schools, but they know very little about others. And by the way, I don’t know that they push the local community colleges; they seem to think that they look better when they get students into the brand-names. Some don’t know enough about community colleges. I remember an old cartoon of two high school students talking. One was saying “Why would you want to get career advice from somebody who ended up being a high school counselor?”
Adm. Assoc., Admin. Assoc. to VP for Academic Initiatives, at 10:15 am EDT on April 9, 2008
As a parent of a current college student and a rising college student, I feel I must comment on this article. Are the rankings perfect? Hardly. But how do high schoolers rank schools otherwise? By who wins the NCAA championship in their favorite sport? By what big name state school is close enough to home to get laundry done, but far enough away to exert independence? Both of my children used these rankings as part of their selection process. Not to make a final decision, but to think outside the box of the “typical” schools that were sending them offers to apply. I believe Liberal Arts schools are the best return on investment a parent can make. My children have both changed their minds about majors several times, and a LA curriculum helps broaden their world view of possibilities. We tell children to follow their passions, but if they aren’t exposed to a variety, how do they know? The surveys are a great tool to see what is out there, that was perhaps overlooked at first glance. Only tool? Hardly, but taking it out of the box entirely would be a real shame.
JDM, at 10:55 am EDT on April 9, 2008
HS Counselors Evaluating Colleges?
What criteria is being used as an evaluation tool for High School Counselors to recommend/value a college?
It was brought to my attention by a fellow employee that the local high school (guidance counselor’s office) sent a letter home showing just how many of the seniors were accepted to a list of various colleges throughout the state and country. NOT listed were the number of students who were accepted at the local community college. Why? For some reason, guidance counselors look down on the community college academics. So, their prejudices are transmitted to the student body and the CC’s are seen as second rate.
Put fairness in the criteria — but good luck with removing bias.
Doc R, Dr. at Atlantic Cape Community College, at 11:15 am EDT on April 9, 2008
The Ranking Game
Some years ago, I emailed US News & World Report about their high ranking of Trinity College of Vermont. I noted to them that Trinity had closed their doors and the institution was no longer operational. I received back a rather curt response that it was the ranking Trinity deserved and that US News & World Report stood by the ranking.
Ed T, Vice President of Enrollment/Student Services, at 11:25 am EDT on April 9, 2008
Building a Better Mousetrap
We shouldn’t be surprised that US News is willing to hear suggestions regarding its methodology. After all, in each of the 25 years the magazine has published rankings, it has relied on a slightly different formula. Under the guise of perfecting the science of its rankings, it has been able to ensure new outcomes by tweaking its methodology.
The latest tweak comes in the wake of protests and threats by college presidents to pull out of the peer review survey. Now, it seems the approach to the presidents is “give them a greater voice so they’ll stay in the game.” And that they will!
If the real evil is the ranking system itself, presidents should stop pretending to be concerned. Rather, they should follow the lead of a growing number of high school superintendents who are walking away from the entire ranking process. It is duplicitous to revile the rankings and then find opportunities to continue involvement with them.
And what can be gained by surveying the counselors at elite high schools? Will we learn more about their perceptions of all colleges or just those to which their students aspire? Let’s not forget that there are more than 23,000 high schools servicing the majority of college-bound students around the country. Wouldn’t it be interesting to hear their voices , too?
Peter Van Buskirk, Author, Winning the College Admission Game, at 12:05 pm EDT on April 9, 2008
The wrong approach
Asking GCs for input will not help, and perhaps hurt USNWR’s (unreliable) rankings. The best data USNWR has is data on incoming freshmen. (The same data is available for free.) Everyone knows what the incoming class looks like, and I don’t think GCs have much to add. Do GCs follow the senior class though college or do they immediately focus on the junior class? Do GCs get regular feedback from students once they are in college? Do they call their grads and ask, “How’s it working out for you?”
Another problem is GC turnover. My kids attend an elite public HS and the GC turnover is about 50% annually. I doubt USNWR can get the same kind of continuity of opinion from GCs that USNWR is soliciting elsewhere (college presidents) for their rankings.
justaguy, at 12:15 pm EDT on April 9, 2008
Compare to world rankings
My university consistently ranks in the top 20 universities in the world in the Times Higher Education rankings (UK) and the world university rankings out of Shanghai. Meanwhile, US News always ranks it mid- to low- 30s every year. How can it be in the top 20 in the world according to international rankings yet rank so low in the US for US News? My boyfriend’s university never appears in the top 500 international, yet ranks above mine according to US News— for the sole reason that it has a very high rate of alumni giving. Alumni giving means the place has a good fundraising team and really pressures alums, not that a college is among the best in the world. Many of the categories used to rank are misleading at best. The methodology should certainly be re-arranged.
Gloria, at 12:30 pm EDT on April 9, 2008
Ranking colleges is rank
USNWR tries to make their rankings looks so mathematically accurate, as if they are the consumer price index. They are glorified opinions that are statistically analyzed to give the illusion of objective, measurable criteria. No matter how many people they include in their “rank”-ings, they remain in the opinions of folks who have certain beliefs about certain colleges. Kudos to the writers who support the achievements of the local community colleges. There are many high school students for whom the USNWR rankings would be a waste of $3 they could have spent on lunch.
Community college prof, at 2:00 pm EDT on April 9, 2008
In Response to JDM
JDM makes the worn-out point that the U.S. News and World Report rankings may not be perfect, but they serve a useful purpose for a great many high school youngsters in their quest to find just the right college or university.
RWH has commented extensively on this perspective in response to two earlier InsideHigherEd articles ...
http://www.insidehighered.com/news/2007/05/07/usnews
http://www.insidehighered.com/views/2007/04/12/keller
Even so, I can’t resist quoting him on two points ...
“3. There is no information – and I mean zilch – that Maclean’s and USN&WR provide for prospective college students that is not provided much, much, much better by the literature and on-line sites of Barron’s, Peterson’s, etc.
4. Any young person with a short list of, say, fifteen colleges or universities — and I don’t care how the list was assembled — would be well-advised to whittle the list down to, say, five schools by pouring over the web-sites of the fifteen schools (do one or two a day), and then do hir very best to talk to a few graduates of those schools ... and definitely – most definitely — spend a day or so at each school s/he is likely to attend.”
“Don’t get me wrong, there is a place for the USN&WR rankings. Suppose you are an intellectually challenged 17-year-old ... and you’re just not bright enough to understand or even assess more than one variable at a time when you’re making important decisions. Then in that case, pull out those silly rankings ... and, if you want my input, start somewhere around number 75 on their list of Teir 3 schools and work down from there. For the rest of you, look at your public library’s issue of USN&WR (whatever you do, don’t purchase one), check out their rankings, chuckle a bit, and then get on with the difficult task of making one of the most important decisions you’ll ever make ... and do it pretty much in accordance with the guidelines in Steps 2-4 in the list above.
I suppose it is just a matter of time until the NCAA teaches some enterprising organization how to write a computer program to aggregate the USN&WR, the Washington Monthly’s College Rankings, the Fiske Guide to Colleges, the College Prowler, the Princeton Review, etc rankings to get the definitive BCS-like list. When that happens, I trust the annual announcements will be made on four-hour extravaganzas on national tv, hosted, of course, by Simon Cowell, Paula Abdul, and Randy Jackson. God, I love that image.”
Well, RWH, I love that image too.
Frizbane Manley, at 2:00 pm EDT on April 9, 2008
you’re the best!
As a high school counselor who works with literally hundreds of students and their families, I acknowledge my “pro” USN&WR bias/appreciation for the role they play in the work that I do! In nearly every presentation that I give, if it weren’t for their Best Colleges issue, I would not be able to say, “so don’t let USN&WR tell who the best colleges are — because what you’re actually looking for are the best colleges FOR YOU — and only YOU can determine that!!”
Guidedog, Director College and Career Center at North Penn High School, at 3:20 pm EDT on April 9, 2008
JDM asks: “Are the rankings perfect? Hardly. But how do high schoolers rank schools otherwise?”
They should rank them on this basis: which school is the best for me? And the resulting ranking will almost certainly be quite different from USNews’ rankings which claim to identify America’s best colleges.
It’s as absurd as trying to rank “America’s best automobile” (I’ve owned the same model of car for over 20 years, it’s the best for me, but I’m not going to make an absurd USNews-style claim that it’s America’s Best Car). Or “America’s best spouse” — gather data on congeniality of personality, physical attractiveness, “retention rate” (who’s been able to avoid divorce), etc. and voila!: rankings of America’s Best Spouse.
MKT, at 5:45 pm EDT on April 9, 2008
Advertisement
Jobs Related to 'U.S. News' Adds Surveys That Could Alter Methodology
or search for jobs directly.
Admissions Representative
Corinthian Colleges
Everest College, a respected member of the Corinthian Colleges’ network of schools, is dedicated to helping students ... see job
Coordinator Undergraduate Programs
Temple University
In collaboration with the Assistant Vice Provost for Undergraduate Studies, the Coordinator for Transfer Student Programs ... see job
Enrollment Manager
Monterey Institute of International Studies
MONTEREY INSTITUTE OF INTERNATIONAL STUDIES JOB OPPORTUNITY Enrollment Manager Category: Other Department: Recruiting and ... see job
Assoc VP for Planning, Innovation & Assessment/Sr Assoc Dir for Undergraduate Admissions
University of South Carolina
For two centuries, the University of South Carolina’s scholarship, research and outreach efforts have contributed to the ... see job
Admissions Counselor/Recruiter
New Mexico Military Institute
A great opportunity for an entry level admissions/recruiting position starting at $32,000! see job
Admissions Counselor & Enrollment Advisor for Professional Studies
Northwest Christian University
Northwest Christian University, a Christian, private, liberal-arts, comprehensive university is seeking an Admissions ... see job
Associate Director of Freshman Admissions
Roger Williams University
Roger Williams University is one of the top ranked liberal arts universities in the Northeast and is an Equal Opportunity ... see job
University Registrar
University of California, San Diego
University of California, San Diego
University Registrar
The University of California, San Diego seeks an ... see job
Assistant Dean for Admissions and Student Affairs
University of North Carolina, Chapel Hill
The Assistant Dean for Admissions and Student Affairs is the academic officer whose principal duties relate to the DDS ... see job
Admissions Representative
Corinthian Colleges
Everest Institute, a respected member of the Corinthian Colleges’ network of schools, is dedicated to helping students ... see job
Home
| Sign up | Log in
| About us
| Work for Us
| Press Releases
| Rights and permissions
| Contact us
| Advertise
| Partners
| RSS 
| Podcasts
| Links
| Privacy
| My Account
© Copyright 2008 Inside Higher Ed
Bad Idea
Giving high school guidance counselors at selected schools a voice is a bad idea. A lot of counselors end up pushing their students to Big State U or the local community college instead of schools that actually fit the student better. At the best high schools, that may not be the case. We’ll see how this goes.
Robert, PhD Student, at 8:35 am EDT on April 9, 2008