Test Prep, to What End?

The extent to which the SAT is coachable has long been central to debates about the ethics of the test. After all, if tutoring programs that cost money help scores, there is an obvious issue of who will be able to afford such an advantage. For years, the College Board insisted that the SAT was not coachable and, more recently, the board has said that gains from test prep services are modest.

Today, the National Association for College Admission Counseling is releasing an analysis on the impact of test preparation services that backs the claims of test prep companies that they do produce gains on the SAT. But the research described suggests that the gains are relatively small -- gains that theoretically shouldn't matter much in admissions decisions. But NACAC also found evidence that at plenty of colleges, these kinds of gains could make a difference.

The NACAC report suggests several actions: Colleges are urged to avoid using the SAT and other tests in ways for which they aren't intended. And test takers are cautioned against expecting too much of an impact from test prep, and so are urged to be skeptical about gains and the value of these services.

Test-prep companies, while largely concurring with the finding that gains are modest on average, maintain that the NACAC study may hide the extent to which some services result in major gains on the SAT, and thus may make the testing system even less equitable than the report suggests.

The NACAC study was prepared by Derek Briggs, chair of the research and evaluation methodology program at the University of Colorado at Boulder. He reviewed all the existing studies of SAT and ACT test preparation services and found that "a consensus position" has emerged over the last 10 years. That view is that coaching has a demonstrable, but minimal impact in improving SAT scores -- about 10-20 points on average in mathematics and 5-10 points in critical reading. Far less work has been done on the ACT and evidence is inconclusive, Briggs writes.

While noting these findings, Briggs also writes that more research is needed. Most of the studies do not consider the most recent changes in the SAT.

Assuming that the findings on coaching gains are accurate, Briggs set out through a survey of college admissions offices to find out whether gains of the kind that the studies documented would have a serious impact on students' odds of getting in. Both the College Board and NACAC have argued for years that small differences in SAT scores should not be e seen as meaning much.

NACAC found, however, that more than one third of colleges said that an increase of 20 points on the math SAT or 10 points on critical reading would "significantly improve a student's likelihood of admission.” The impact of such gains rises as the total SAT score goes up and at more selective colleges.

David Hawkins, director of public policy and research at NACAC, said that these findings suggest that a major goal for the organization should be to "bring practice into compliance with accepted standards," and that the "only way we can do that is to educate, educate, educate."

In September, NACAC issued a landmark report suggesting that many colleges reconsider their policies of requiring standardized testing in admissions, and the report noted the link between testing requirements and the test prep industry. Not only does coaching favor wealthier applicants, the report found, but it can lead high schoolers to focus on the admissions process to the detriment of their educations. The report called for more work on such topics as the test prep industry, noting that much of the existing knowledge comes from the entities that sponsor standardized tests.

Hawkins also said that he hoped the new report would encourage students and families to view test prep services for what they are: businesses looking for "a commercial transaction."

Needless to say, those businesses have a keen interest in how NACAC describes their services. And test prep officials freely admit that they gain customers because of the perception that they are raising scores by far more than 30 or so points. But different kinds of services have different approaches to how they discuss score gain.

The national chains tend not to promise specific gains or to quote average gains. Kaplan and Princeton Review, for example, have an agreement between them that they will quote figures only if they let their competitor examine the methodology and critique it -- and under this agreement it has been years since either company quoted an average score gain. Both companies, however, boast on their Web sites that students will see scores go up and both offer forms of money-back guarantees for those who are not satisfied with score gains.

Seppy Basili, senior vice president at Kaplan, said NACAC is wise to encourage students not to trust promotions about score gains because the averages mean very little. A 150-point gain for someone on the low end of the SAT range is fairly easy to accomplish, Basili said, while a similar gain might be quite challenging for someone already near the top. "We're very careful not to use average score increase," he said.

Basili also said that admissions counselors and the College Board have "a vested interest" in promoting the idea that test prep doesn't have a huge impact. "It's very uncomfortable for them because if test prep works, does everyone have equal access?"

The answer to that question is clear, he said. "The SAT favors anyone who can get test preparation."

Paul Kanarek, senior vice president at Princeton Review, agreed. He said that score claims are all over the place in terms of how they count "before" and "after" scores, who is in the pool, and so forth. At the same time, he suggested that the NACAC report understates the impact of coaching. "If we didn't raise scores in the eyes of the market place, we would not be one of the largest players in the market place," he said.

He said that this reputation comes largely from satisfied test takers. The only notable change in recent years, he said, is that the student describes satisfaction in different ways. "Ten years ago, it was 'I went up 300 points,' but now it's 'I broke 2000.' It's all about the final score."

While the dominant players shy away from explicit score gains, many others do not.

If you call Ivy Bound Test Prep, for example, the recording that greets you says you should choose it "if you are looking to raise your SAT scores 200 points or more." And the Web site says that "last year's students reported a 171.1 point increase (math and critical reading) over a previous SAT or PSAT."

Mark Greenstein, the president and founder, said he too agreed that students should be "skeptical and smart shoppers," but that there was nothing wrong with his company's claims. He said that his average is based on "diligent students," whom he defined as doing at least 90 percent of assignments and calling the help line at least once a week. "If you do that, you are good for 150 points at least," he said. Greenstein said that he uses actual test scores for the "before" score. He also said that he updates the score average annually, so that it reflects the latest SAT.

Those who sign up for Ivy Bound's courses have gains about 10 points lower than the online figure of 171.1.The higher gains are for those who opt for private tutors, and Greenstein said it is clear that paying for such help pays off.

Families pay by the hour and at least 25 hours of tutoring are needed to achieve the score gains (plus another 8 if students want help on the writing test of the SAT, too), he said. For less experienced tutors, families pay $60-$80 an hour, while veterans cost $210 an hour and the top of the line is $350 an hour.

The money is worth it, especially when considering the merit scholarships that students are earning these days, Greenstein said. He echoed the criticism of others in test prep in saying that admissions counselors "are dismissing the economic benefit of test prep." And while NACAC questioned the need for expensive services, Greenstein said that he sees a relationship between what students pay and their diligence and corresponding gains.

"There is an aspect of 'my Mom's paying $3,000 so I better do this right,' " he said.

Hawkins said that in fact admissions counselors and NACAC are "in the middle of competing interests." He said that he doesn't doubt that some test prep raises scores, even if the precise size of the gains isn't clear.

But he pointed to the ethical issues raised by the NACAC report on standardized testing. That report said: "Access to test preparation will always be differentiated based on family income, school setting, and other variables external to the student. As a result, students without the financial resources to gain access to test preparation may, in effect, be penalized for lower test scores in some admission and scholarship scenarios, especially in those that use test scores in a 'cut score' or non-contextualized linear fashion.... [T]he ability of more affluent students to afford both test preparation and multiple test administrations, particularly when they are able to report the highest scores to colleges and universities, puts students of more modest economic means at further risk of being overlooked in the transition process."