Ethical College Admissions: Khan Air

What’s your favorite Steve Buscemi moment? It’s hard to top the wood chipper scene from Fargo, but I’m partial to his definition of irony from Con Air.

As a hijacked plane full of escaped convicts takes off with the convicts singing and dancing to Lynyrd Skynyrd’s “Sweet Home Alabama,” Buscemi’s character wryly observes, “Define irony: a bunch of idiots dancing on a plane to a song made famous by a band that died in a plane crash.”

Recently the College Board issued a statement trumpeting the success of its collaboration with Khan Academy to provide free online practice for the new SAT. According to the statement, students who spend at least 20 hours on Khan Academy’s “Official SAT Practice” have an average score increase of 115 points, nearly double the increase of those who don’t use the Khan Academy program.

Define irony.

The College Board affirming the value of test preparation is akin to Greenpeace suddenly denying the existence of global warming. For nearly 60 years the College Board’s position has been that test prep provides minimal benefits. A College Board document, “Effects of Coaching on SAT Scores,” argues that the estimates of the benefits of test prep reported by test prep vendors are much too high, that the typical gain from coaching is eight points on the verbal side and 18 on the math side.

Is that document, which involves a study of coaching done back in 1996, no longer valid? What’s changed? According to Zach Goldberg, senior director of media relations at the College Board, both the SAT and the approach to test preparation used by Khan Academy are different.

Regarding the test itself, Goldberg responded to an inquiry from Ethical College Admissions, “The new SAT is a different test. It is an achievement test that measures what students are already learning in high school and what they need to know to succeed in college and career. With the new SAT there is no penalty for guessing. Students no longer lose points for wrong answers. Gone are ‘SAT words’ -- words no one has seen before or will likely see again. Only relevant math concepts are tested. The SAT makes it easier for students to show their best work.”

That statement raises as many questions as it answers. So the SAT is now an “achievement” test? SAT was originally an acronym for Scholastic Aptitude Test. It then became the Scholastic Assessment Test, then just the SAT, standing for nothing in particular. If the new test is an achievement test, then why still have the SAT Subject tests, which are much more closely linked to advanced high school work in specific academic subjects, once upon a time known as “Achievement Tests”?

I get the “new and improved” vibe for the new test, and like that vocabulary is now tested in context, but does the statement “Only relevant math concepts are tested” suggest that there were irrelevant math concepts on the old test? I am also unclear about how elimination of the penalty for guessing makes the test a better test or “makes it easier for students to show their best work.” With or without the guessing penalty, SAT scores are scaled scores.

If the test is different, so is the Khan Academy approach to preparation. The College Board is careful not to call it “coaching,” but rather practice, and “official” practice at that. The College Board spokesman told Ethical College Admissions, “Too much of commercial test prep teaches to the test -- looking for shortcuts and tricks to ‘beat’ the test ... The College Board and Khan Academy firmly believe in practice, and particularly practice that is personalized to pinpoint areas where learners need additional help.”

I’m not a fan of the test prep industry, especially the “beat the test” mentality, but what the College Board calls practice sounds a lot like coaching.

Let’s look more carefully at the 115-point increase claim. Whenever I encounter a test prep claim for a score increase larger than the 20-30 point increase that the 2008 National Association for College Admission Counseling Testing Commission concluded was supported by reliable academic research, the first question is what is being compared.

One of the tricks used by some test-prep firms is using an in-house pre-test to set the baseline. Last fall some of my students took a practice ACT through a national test-prep company. When the scores came back they looked low. Some of that was that the students taking the test were first semester juniors, but my staff also suspected another factor. We wondered if the practice test questions were a “hard” version of the test, artificially lowering scores. Why do that? Two reasons. If your pre-test scores are low it provides incentive to take a test prep course, and it increases the score benefit you receive from the course. A representative from the test prep firm confirmed that practice.

The College Board 115-point claim is based on a comparison of scores between the PSAT taken in October of 2015 and the SAT for students with 20 or more hours of Khan Academy practice, including the 4 percent who had no increase in scores. The determination of those using Khan Academy for 20 hours or more comes from Khan Academy usage data, making the figure more reliable than if it were self-reported by students.

So how much of the 115-point increase (for some reason the College Board has no idea what the increase is for each section) should be credited to Khan Academy? Several test-prep companies have chimed in to argue that the students using Khan Academy may not be using it exclusively, that more traditional coaching may also deserve some of the credit. It is also the case that some of the score improvement comes from a year of maturity and intellectual growth.

Perhaps the most interesting nugget in the press release is buried several paragraphs below the headline. The 115-point increase is associated with 20 or more hours of Official SAT Practice, but students who do six to eight hours have an average increase of 90 points. So there is a 10-15 point per hour improvement for the first six to eight hours, but two points per hour after that. Define irony. That raises two questions. Is that kind of diminishing return true with regard to all kinds of preparation? And channeling my students, all males who are by nature strategic thinkers, why would I spend 20 hours practicing when I can get almost as much improvement in six hours?

That leaves two other more important questions. The first is, Why bother to brag about the score increase? The College Board hit a home run in collaborating with Khan Academy, an organization with a good reputation, to provide students with a free resource. Why not leave it at that? Why play the “My test prep is bigger than yours” game?

Finally, what does all this mean? Khan Academy founder Sal Khan seemed to be equating higher SAT scores with increased readiness for college. But is that the case? How significant are SAT scores as a factor predicting readiness for college, and are scores more or less meaningful because they come through coaching/Official SAT Practice? Is a 70-point increase without practice better than a 115-point increase with practice? The College Board is not going to tackle that question, because it cuts at the heart of its business model. Define irony.