Would U.S. News Make Up Fake Data?

It's not unusual for college presidents to complain about U.S. News rankings (at least out of the earshot of U.S. News editors). But on Sunday, the president of Sarah Lawrence College publicly charged that the magazine is preparing to publish made up, false data about her institution. Meanwhile, Inside Higher Ed has learned that 10 other liberal arts college presidents are preparing a letter to be sent to hundreds of college presidents proposing a new set of policies that might challenge the role of the rankings. The policy options include complete non-cooperation with U.S. News and refusing to fill out the "reputational" survey -- which many educators deride as a "beauty contest" that is particularly lacking in substance, even though it represents 25 percent of the magazine's rankings formula.

Allegation of Making Up Data

In an op-ed in The Washington Post, Michele Tolela Myers, Sarah Lawrence's president, wrote that because her college no longer collects or examines SAT scores, U.S. News officials have said that the magazine will just assume that the average SAT would have been one standard deviation (about 200 points) below the average of Sarah's Lawrence's peers. "In other words, in the absence of real data, they will make up a number," wrote Myers.

In an interview Sunday, the head of the U.S. News college rankings division acknowledged that he had told Myers of the magazine's plan to use the system she described in her article. But Robert Morse said that while he told her that was the plan, he also said that the magazine would "seriously" consider other approaches, which he declined to name. Myers, in an interview Sunday, said flatly that Morse had never said there was any other approach under consideration.

The dispute between Sarah Lawrence and U.S. News highlights more than just the rankings issue. The reason Sarah Lawrence is facing this problem is that the college -- possibly alone among the many colleges that are dropping SAT requirements -- won't even look at SAT scores any more. As more liberal arts colleges like Sarah Lawrence have dropped SAT requirements, the norm has been to go "SAT optional," meaning that students are still welcome to submit the scores. Because a majority of applicants do so, SAT-optional colleges continue to have average SAT scores to report to U.S. News.

And that's where the dispute starts to point to potential problems with both SAT averages and U.S. News. When applicants learn that a college is SAT-optional, it doesn't take an 800 math score to figure out the statistically wise strategy. If your scores are at or above reported averages, submit them. Otherwise, don't. Not surprisingly then, many colleges that go SAT-optional experience both a surge in applications and an increase in their SAT averages ... and their U.S. News rankings go up.

Along comes Sarah Lawrence and it makes a decision that it doesn't believe in the SAT, so it doesn't want to look at scores -- period. As the college explains in its FAQ for applicants: "Our recent decision to remove all standardized testing from the admission process reflects the college’s emphasis on writing rather than testing. That's right: we no longer look at standardized test scores. Our students aren’t 'numbers,' and we’d rather not use standardized test numbers to select them."

Sarah Lawrence has long asked for more writing from applicants than most colleges request and Myers said in the interview that the college didn't drop the SAT until it studied its data extensively and concluded that the SAT offered "nothing" in terms of predicting students' success. Myers said she was aware of the way other colleges drop the SAT but continue to collect scores and data from many applicants, but she questioned whether this was educationally sound. "I'm not going to play that game," she said.

The U.S. News rankings that will appear this fall would be the first in which Sarah Lawrence will no longer have any SAT data to report. With that date looming, Myers and Morse met and that was where he outlined the magazine's plan. He said he could not think of another institution where he had to propose this approach because around two-thirds of applicants at SAT-optional colleges submit scores and U.S. News will count that average as long as the percentage exceeds 50 percent.

On Sunday, Morse said that he was reluctant to talk about Sarah Lawrence's complaints until he could confer with the top editors at U.S. News. But pressed, he said that Myers had left out "a key fact" from her op-ed in not saying that he had told her U.S. News was considering approaches besides just assigning the college a lower SAT average based on no real data.

Morse said he had told Myers that the approach of cutting the SAT score a standard deviation from the peer group average was "what we were going to do," but he said he added that a new approach might also be considered. He said that as of today, no other approach had been determined, and that the plan remains the same, but that the discussions were "a process still going on."

Morse said that he regularly speaks to groups of institutional researchers about the U.S. News methodology, including how the magazine handles situations where colleges do not provide some of the data the magazine seeks. He said that the magazine doesn't always use the approach of just lowering estimates by a standard deviation, and that the approach varies from category to category. "There are various procedures, and we've been very transparent," he said.

Myers said that in her meeting with Morse, he expressed skepticism about the averages he gets from SAT-optional colleges, saying that he believed they were managing to inflate their averages. But on Sunday, Morse denied believing that. He said that colleges dropping the SAT and submitting the average scores of those who continue to send them would experience "a long term decline" because he expected their graduation rates to decline, especially as projected by their SAT scores, which is part of the U.S. News formula. So while Morse said he knew that it is "widely believed" that going SAT-optional helps in the rankings, he said that wasn't the case. (Morse's thinking is based, of course, on the assumption that the students who don't report SAT scores will have lower retention and graduation rates -- something that colleges that have gone SAT-optional say is not the case.)

Bob Schaeffer, public education director of the National Center for Fair & Open Testing, praised Sarah Lawrence and said it was taking a principled stand. "The U.S. News policy of fabricating data for colleges which refuse to report average scores it is both unfair and unethical," he said, adding that the "practice further undermines the credibility of the magazine's already widely criticized rankings."

A Broader Challenge to the 'Ranksteering'

As Myers took her criticism of the rankings public, other presidents are working behind the scenes to challenge U.S. News. In her op-ed, Myers noted that some on her campus wanted to completely disassociate the college from the magazine's rankings, but that she feared doing so would hurt the college more, as more inaccurate data about it might be included. While she said the college was absolutely committed to keeping its policy of ignoring the SAT, she said that ignoring U.S. News was "too big a risk" for the college to take alone. ( Reed College has taken that step for a number of years, although Morse noted that much of the information that goes into the rankings is public and the magazine gathers it for Reed, which does require the SAT or the ACT.)

A new effort is being organized to allow colleges to act in concert so that they fight back against the rankings. Some of the inspiration for this new effort came in January, when Lloyd Thacker, founder of the Education Conservancy, gave a presentation to the Council on Independent Colleges called "Ranksteering: Driving Under the Influence." Thacker's organization argues that the college admission process has become divorced from educational values and "ranksteering" is his term for the impact of U.S. News and other rankings.

The presentation prompted a group of 10 presidents to work on a letter -- currently circulating among them -- outlining strategies that could be used to take on the rankings. The presidents' idea is to send the letter to all Council on Independent Colleges members (more than 570 institutions) to try to build a movement with the clout to be effective. Among the ideas in the letter: refusing to provide any help to U.S. News, refusing to fill out the "reputational" survey, pledging not to advertise any rankings they receive from U.S. News, and/or posting a prominent statement on their Web sites noting the dubious qualities of many ratings.

In an interview Sunday, Thacker confirmed that such a letter from presidents is in the works, but declined to discuss timing or to name the presidents involved, except to say that they are all from liberal arts institutions, that Myers of Sarah Lawrence had not been involved and that her op-ed was not part of this campaign. At the same time, he said that the issues Myers raised were completely consistent with the Education Conservancy's critiques and those of the presidents' letter.

Thacker said that was most striking about Sarah Lawrence's actions and the Myers op-ed was a willingness to stand up to both the College Board (as the sponsor of the SAT) and U.S. News. "Very few colleges feel they can act as singular moral agents," Thacker said. "What she is doing is demonstrating the courage of her convictions."

In the end, Thacker predicted that places like Sarah Lawrence would do well with applications, even if the U.S. News rankings include inaccurate information about them. Reed has made its rejection of U.S. News "its calling card," Thacker said, and Sarah Lawrence may benefit from its ability "to distinguish itself by demonstrating its character in its admissions process."

Morse, of U.S. News, said he didn't know anything about the letter circulating among college presidents. But he noted that there have been complaints about the rankings in the past that did not amount to much, and that law deans have been complaining about the law school rankings for years -- even as the rankings continue to be popular with applicants. He also noted that much of the information in the rankings is also provided to the federal government so the magazine could still get it.

On the reputational survey, Morse said that 70 percent of liberal arts colleges currently fill it out. "It would have to fall a lot to be statistically significant," he said. "If just 10 college presidents don't return the survey, it isn't one iota going to dilute the response rate."

Morse noted that many of the presidents who don't like his ratings also oppose Education Secretary Margaret Spellings' attempts to make comparable data more widely available on various measures of student learning. He said he wasn't particularly worried about the fact that these presidents "are trying to derail us."

Myers, the Sarah Lawrence president, said she feels good about her college's direction, and she noted that some of the strongest endorsements she has received for the SAT stance have come from high school counselors who like the message the college has sent to students.

As for the meaning of the dispute, Myers said: "It says a lot about U.S. News and higher education in general that we are willing to play dead most of the time just so this magazine can sell the one issue a year that makes a lot of money."