The Next Testing Scandal

Almost one year ago, a college admissions cheating conspiracy made headlines, partially because the behavior involved was memorably egregious, and partially because some of the participants were celebrities. It was a great story, but it might have given the impression that this kind of wholesale fraud is more common than it is.

Consider the number of moving parts and risks involved in bribing proctors, coaches and admissions officials. Any one of those links could have brought down the entire operation. Then add the risks of making up extracurricular activities out of whole cloth. Some fact-checking here and there could have uncovered the defendants’ “creative” Photoshopping and then exposed the whole scheme. For these reasons and others, complex conspiracies involving all aspects of the college admissions process are not the biggest threat to the legitimacy of the process. But some threats are real, such as:

The Time Zone Scam

If everyone takes the test at the same time, then it’s impossible for one person to finish the test and give the answers to another person, right? Well, not necessarily, because the same test can be given in different time zones, making it just barely possible to have a good test taker nail the test on the East Coast, record their answers and give the sequence to someone on the West Coast, who would need to smuggle it in somehow. An operation like this came to light about 20 years ago, but there’s no reason that it couldn’t be going on now. Still, this does have a lot of moving parts, and other approaches might be easier, such as …

The Fake ID Scam

Why depend on getting the answers from a better test taker when you can have that person take the test for you instead? This has always been a risk, and there have been isolated cases in which impostors took the test for other people. But this is much harder to pull this off now, because when you register for the SAT or ACT, you need to upload a picture of yourself. It’s still possible that someone could initially upload the picture of the impostor, but that would be a risky thing to do for both parties, because if anyone checks, there’s a pretty undeniable record of the fraud. Someone who would take that risk might try something with less of a paper trail, such as …

Bribing the Proctor/Stealing the Test

If the more sophisticated approaches are unacceptable, there are always the blunt approaches, such as bribing the proctor to look the other way while someone else takes the test or gets access to the test ahead of time. The former approach was one part of the admissions scandal of 2019, and the latter has been known to happen from time to time. A leaked test might be leaked to one person, or for sale generally. But if the test is known to be exposed, the testing agency can cancel the test. Special testing conditions may make this scam easier to pull off. Sometimes, test takers with accommodations for, say, anxiety can take their test privately with a proctor they know from their school. There are legitimate reasons to do this, but allowing such accommodations means that there wouldn’t be much stopping two people from pulling off this kind of fraud, especially because there wouldn’t be lots of extra witnesses.

Countermeasures

As dangerous as these schemes might seem, they remain relatively rare, which is perhaps testimony to both the rule of law in our society and the fact that it’s pretty easy to get caught. The test makers can also take steps to make fraud more difficult. They can make it more difficult for people who aren’t trying to get into colleges to register. Years ago, I took the SAT just to make sure that I understood the details of the testing experience (and, just to be clear, not for any cheating of any kind). Midtest, the proctors asked me if I was “a spy.” I told them that according to their rules, I didn’t need to tell them why I was taking the test. There was nothing they could do. Nowadays, it seems that they’re less tolerant of that kind of thing.

In additional to policing who takes the test, the test makers can also change the order of questions within a test form to make it harder to cheat. For example, on the recent PSAT, different people encountered the same questions in different places. If someone’s answers would have been correct for another version of the test, then that’s pretty solid evidence of cheating, especially for questions in which the students enters a numeric value. No one guesses “31.04” by accident.

The test makers can also look for implausible improvements between tests. I’ve helped some students legitimately earn scores that probably looked suspicious, but I’ve never had a student’s improvement investigated. Still, a huge improvement in a short period of time can look fishy, and sometimes people have their scores held up or canceled for that reason.

Over all, the difficulty and risks associated with blatant cheating on a massive scale make today’s admissions tests relatively secure. There’s always some risk, which in my view is highest for nonstandard test administrations in which only a few people have custody of a test that could be given any time within a fairly large window. Still, in general, any much more widespread scam has a proportionally larger chance of being detected. If an exam is for sale on the internet and lots of test takers know about it, then chances are the test maker knows about it, too.

New Formats, New Vulnerabilities

That being said, if the test format changes, new vulnerabilities could emerge. For example, consider what would happen if the SAT and ACT followed the model of the GRE and GMAT. In that model, tests are administered year-round in permanent test centers. This solves the logistical problem of getting millions of students into test sites on just a few days a year, makes it easier to take the test whenever you want and allows you to get results much more quickly.

However, this approach creates a security problem in that you obviously can’t give the same test to everyone all year-round. Even if you changed the test every day, people who took the test earlier in the day could give information to people who took it later in the day. To solve that problem, test makers often use an adaptive approach in which questions are selected from a large pool so that no two people get the same test. The adaptive algorithm refines its “opinion” of your skill level by giving you easier questions when you get a question wrong and harder questions when you get a question right, zeroing in on an accurate assessment of your skill.

That approach works for most people most of the time, but it has vulnerabilities. The question pool may be more “shallow” than it needs to be, and some questions are more likely than others to be chosen for your test, which together makes it more likely that different people will see the same questions. Also, the algorithm tends to make up its mind about your ability fairly early in the process.

As a result, if you are ready for the, say, 50 questions most likely to be chosen early in your test, you can get your ability estimate (that is, your score) to an extremely high level that won’t be affected much by missing all the questions at the end. This was what Kaplan Test Prep discovered back in the 1990s when the GRE first went adaptive. We (I’m going to say “we” because I worked for Kaplan at the time but was uninvolved in this initiative) suspected that the testing pool was shallow and that the same questions were likely to get picked.

To test the theory, we sent in 25 people, who reconstructed the test with remarkable accuracy. We didn’t tell any examinees about this, of course. Instead, we told ETS, the test maker, about the problem. They then sued us for copyright infringement, among other things. Nice people. The lawsuit was settled, but ETS had a major problem on their hands, because we proved that their test wasn’t secure. And while Kaplan sent in 25 people, I suspect it could have been done with far fewer. When I wrote test prep courses, I took tests in order to monitor trends, and I could reconstruct over 90 percent of what I saw. It wouldn’t take many people like me to bust open even a deep question pool.

The SAT and ACT could face a similar problem if they went to an adaptive model. And remember, with an adaptive model, you don’t need to know all of the questions you might face. It’s enough to know the ones you’ll see early on. If you get them correct, your score estimate will be too high to drag down, even if you tank the end of the test. (Once, when I took a GMAT, I intentionally got the last 25 percent of the test wrong and still got a top score.) Of course, this is all fairly speculative. The SAT and ACT don’t use the model that Kaplan broke open, and even if they did, there are measures they could take to protect the pool and detect when it’s been compromised. Still, you can bet that if there’s an opportunity to beat the system, someone will try.

The Bigger Problems Are in Plain Sight

While the threat from blatant fraud and theft is relatively small and (so far) manageable, there are other, more serious threats to the legitimacy of testing that get less attention, perhaps because they are legal. Consider accommodations. While they are often completely legitimate, they are vulnerable to abuse, and for a price. We wouldn’t tolerate a system in which rich kids get to buy extra time on their tests, but that’s not too far from where we are, because while kids whose families can afford to pay for a diagnosis will get extra time on the SAT and ACT, kids who aren’t even aware that this is an option will run out of time, get bad scores, be stuck in remedial classes, drop out with huge loans and so on.

And while I’ve seen plenty of legitimate cases of disabilities that deserved accommodation, there are plenty of bogus ones, too. Someone will earn a prize in journalism for showing that basically anyone can get extra time if they get the right doctor and can pay. A similar prize awaits the person who shows that millions of low-SES kids are at least as deserving of that extra time but never got it. But right now, all this is legal.

Similarly, it’s revealing to compare the reaction to the admissions scandal to our response to structural forces that we tend to take for granted. We object to blatant cheating, which we should, but we tolerate testing approaches that enforce a compliance-based curriculum that drains the creativity out of learning. And before you counter that it has always been this way, look at recent changes to the tests and decide for yourself whether it’s actually gotten worse.

And even if we were testing the best curriculum, not everyone has access to that curriculum. Just recently, I volunteered to help out a real striver who wanted one last shot at the SAT while juggling school, work and family responsibilities. He did a lot better and will be fine, but most of what he didn’t know was stuff he never saw in the first place. A second student came in with scores that would put her in remedial classes, but just three weeks of advice helped her improve by over 200 points, which changes everything. This result is nice for her, of course, but this case and others point out the tragedy of the kids we’re not reaching, who get punished by an “objective” system that just so happens to magnify inequality.

So let’s keep outraged reactions when celebrities cheat to get ahead. But let’s also be aware of the factors in plain sight that stand in the way of equal opportunity.