Can a $49 English Test Pass Muster?

A new, ultra-low-cost English skills test is crashing the already crowded party of higher ed language assessment, offering prospective international students the ability to demonstrate their English skills from virtually anywhere with an internet connection.

But will colleges take its AI-enabled results seriously?

Developed by the makers of a free, gamelike language-learning smartphone app, the Duolingo English Test costs just $49 -- as little as one-fifth the price of the widely used Test of English as a Foreign Language (TOEFL) and its closest competitor, the International English Language Testing System (IELTS), used by many British, Canadian and Australian universities, among others.

Unlike traditional tests, which often require traveling to testing centers staffed with live proctors, the new test can be administered anytime and anywhere a student can find reliable internet access and a webcam-enabled computer.

For many, that alleviates the need to take an overnight bus, train or plane ride to a major city, said Jen Dewar, Duolingo’s head of strategy.

Dewar spent 17 years working in admissions at Duke University and Washington University in St. Louis, often with students aspiring to attend from abroad. After she saw the new test, she said, she realized “I had never seen such an elegant solution to the challenges that international students face.”

The Duolingo test is adaptive, taking just 45 minutes to administer, far less than the three to four hours of traditional exams -- Duolingo said the test ends when its “grading engine has reached a confidence threshold in the test taker’s score.”

Users can't "fail" in the traditional sense -- universities simply decide upon a cut score. On a 100-point scale, a score of 11 to 30 puts users at "elementary" level, with the ability to deal with "simple, straightforward information and express themselves in familiar contexts." By contrast, a score of 51 to 70, deemed "advanced," predicts that users can "fulfill most communication goals, even on unfamiliar topics," with an ability to interact with native English speakers "fairly painlessly." Advanced scorers can also understand the main ideas of both "concrete and abstract writing."

Dewar said several top universities and colleges, including Columbia University, the University of Arizona and Bowdoin, Colby and Middlebury Colleges, have committed to the test. Duolingo publishes a list of dozens of "partners" that it said may use the test in admissions evaluations, but admissions officials at two of them, the University of Southern California and Duke, don't include information on the test in online admissions materials and didn't immediately respond to queries about it.

Among institutions that post criteria online, requirements vary on how, and under what circumstances, they will accept Duolingo's results. On its admissions website, Colby said it accepts the scores, but only alongside TOEFL or IELTS scores. The new test, Colby said, "may not replace one of these tests." An admissions official at Washington & Lee University said the same. Columbia and Bowdoin both said applicants should score a minimum of 75, putting a user's skill level at the low end of "proficient." Bowdoin also recommends scoring at least a 100 on the TOEFL or 7 on IELTS. 

Wellesley College tells prospective students whose native language is not English that it "strongly" recommends taking one of six proficiency tests, Duolingo among them. An admissions official at Indiana University said it accepts Duolingo scores in tandem with other indicators of English proficiency such as TOEFL or IELTS scores and high school coursework in English. 

Middlebury said it will accept the results of the Duolingo test, along with seven others. Tulane said international applicants may submit to an interview through the Chinese start-up InitialView or through Duolingo's online tool.

“I think they’re recognizing that in some way we’re democratizing access to testing,” Dewar said.

Not so fast, said David Payne, who oversees the Educational Testing Service's graduate and English language programs, including TOEFL.

Payne said the new test is indeed disruptive, but that the classic concept of “disruptive innovation” holds that new companies offer a low-cost alternative that attracts new users -- think Toyota, Honda and Nissan disrupting Detroit’s Big Three automakers in the 1970s -- with cheaper, bare-bones products.

Part of the disruptive appeal, Payne said, is in offering fewer features than the original. That’s what the Duolingo test is, he said -- only the features it leaves out are security, validity and reliability.

“When you are admitting someone to your institution, you are placing a bet that that person will be able to succeed in your college or university.” The new test, he said, won’t tell you that.

For instance, it relies on users identifying real or fake English words. “That’s not something you’d do” in an academic setting, he said. And because the new test isn’t given in a secure setting, there’s no way to know if the test taker is who he or she claims to be -- or if he or she is receiving assistance, he said.

Dewar said security is key. “If we can’t provide a secure test, then the test is worthless,” she said. Each test taker’s ID is verified by comparing his or her government-issued ID photo with another photo of the person and a video of the test session.

Every session is recorded and reviewed by human proctors before a grade is issued, Dewar said. Software tracks test takers’ eye movement, measuring how they respond to prompts. It detects if a test taker is using other devices and prohibits users from wearing headphones, moving outside of the webcam frame, interacting with anyone else or even looking off-screen during testing.

It also locks down users’ web browsers to prevent them from accessing outside materials.

“While no test out there is 100 percent secure, as an ed-tech company, we have the engineering resources and expertise to continue improving our rigorous security protocols,” Dewar said.

For the record, users have tried to game traditional English testing systems, including TOEFL. Last August, three Chinese students pleaded guilty to cheating on the test to obtain U.S. student visas and slots at top universities. In one case, a prospective Arizona State University undergraduate obtained an F-1 visa despite failing three times to earn ASU’s minimum TOEFL score of 61. On the fourth try, however, she scored a 91, but federal agents investigated and found that she had paid $1,000 to another Chinese student attending business school in Massachusetts.

That student eventually confessed to taking tests for three others, for which she charged $7,000 total, the Phoenix New Times reported last year.

In 2015, a federal grand jury in Pittsburgh indicted 15 Chinese nationals on charges of conspiracy, counterfeiting foreign passports, mail fraud and wire fraud after they allegedly hired impostors to take several tests, including TOEFL, the SAT and the Graduate Record Exam.

ETS’s Payne said that while TOEFL is indeed more expensive, the cost is “a drop in the bucket” compared to other costs of attending college. “Given how important the decision is that’s based on (TOEFL) scores, I think it’s a small investment for the individual.”

ETS wouldn’t provide data on its most recent test takers, but it said more than 30 million people worldwide have taken TOEFL since the early 1960s. It is recognized by more than 10,000 colleges, universities and agencies in more than 180 countries.

Likewise, the British Council, which oversees IELTS, said it is accepted by more than 10,000 organizations worldwide, "including schools, universities, employers, immigration authorities and professional bodies." A council spokesman declined to comment on how IELTS compares to the new test.

Dewar said the Duolingo test measures “the whole spectrum of English proficiency” from beginning to native speaker, using the same framework used by other tests. And its video interview “provides a much better view of functional English ability than a test score alone.”

Duolingo maintains that its test is a valid indicator of subjects' English skills -- the company noted in 2017 that the test is calibrated to the Common European Frame of Reference, or CEFR. It is also "aligned with TOEFL and IELTS about as well as they are aligned with each other," Duolingo said, with scores on the new test "substantially correlated" to those of the other tests. The study was based on 1,057 TOEFL scores and 304 IELTS scores. Duolingo said reliability studies are "ongoing."

It also said research conducted in 2016, on an older version of the test, suggests it may actually outperform the others in predicting how college faculty rate the English proficiency and "support needs" of incoming international students. The company said the test "may feel unfamiliar at first glance, but every aspect of the tool’s construction has faced rigorous scrutiny to ensure that it is accurate, reliable, useful and consistent with Duolingo’s mission of making education accessible to all."

While the real-vs.-fake-words skills test "is not obviously related to everyday language tasks," Duolingo said, a research brief last year noted that the task requires the same cognitive processes used in everyday reading, writing and listening.

Kyle Strothmann, Washington University’s senior associate director of recruitment and selection, said TOEFL itself suffers on occasion from a lack of reliability -- he recalled “many, many really heart-wrenching stories” of prospective students who travel far from home only to find that a planned test administration isn’t taking place.

“Oftentimes the exams are in cities that require some sort of travel from home, whether it’s a flight or a train -- so they’re investing money beyond what they’re paying to just take the exam. And students are then making the trip to a major city, if they’re in more of a rural market, or from city to city because the TOEFL exam for them got canceled -- whatever it may be -- only to find that often they arrive at the testing site and the exam has once again been canceled.”

The Duolingo exam, he said, “is really a positive example of disruption in the higher education industry -- it promotes inclusion and it promotes access.”

Strothmann said he likes that the new test shifts questioning in response to users’ answers. As a result, it’s harder to study for -- or to game. In many ways, he said, it’s a more direct indicator of a student’s English skills.

“There are already so many hoops to jump through in the application process,” he said. “Giving [students] the opportunity to just allow this to be something they don’t have to think so long and hard about is, I think, refreshing.”

Wash U. is using the Duolingo test for admission this fall -- Strothmann has been telling prospective students about it and said the university is considering buying $49 “tokens,” each worth one exam session, to offer to prospective students “as a way to gauge interest” in attending.

One of the company's two founders, Guatemalan-born CEO Luis von Ahn, told Slate in 2016 that when he applied to U.S. universities as a prospective student, he had to fly to El Salvador to take the TOEFL. “It cost me $1,000 to certify that I know English,” he said. “That’s like three months’ salary for most people in the developing world.”

Lucas Fink, a senior product manager at the Berkeley, Calif.-based test prep company Magoosh, said it’s not unheard-of for some students to travel to TOEFL testing centers half a dozen times until they get an adequate score, in effect outcompeting those who can’t afford such travel. “That’s not quite fair,” he said.

He said the Duolingo format seems promising. Whereas TOEFL might ask users to complete multiple-choice questions on an academic passage “about the breeding habits of the North American ground squirrel,” the Duolingo test offers shorter readings. Its reliance on identifying real vs. fake English words gives it “a more gamelike feeling,” he said.

The Duolingo test is too new to have attracted any “real demand” for test prep from students, Fink said, but he has played around with it and said it seems to offer “a fairly good approximation of language ability -- is it at the same level as TOEFL? I’m not sure.”