The Danger of Predictions in Education

I am a predictions skeptic.

Who isn’t, though? Our political season has proved that pundits, including supposed “data-driven” ones like Nate Silver – who put Donald Trump’s chances of nomination at less than 5% – have no magical power when it comes to knowing the future.

Sports prognosticators may be even worse. The Cleveland Cavaliers were eulogized after their lopsided losses in the opening games of the NBA championship series. In reviewing the pre-season predictions for the NHL at ESPN.com, not even one of the eleven “experts” correctly predicted either the Eastern Conference (Pittsburgh) or Western Conference (San Jose) champions, let alone the Stanley Cup winning team (Pittsburgh…boo).

Predictions for politics and sports can be fun, entertainment as we make our way towards the eventual outcome. They’re usually harmless, having little to no effect, though the current Brexit crisis in Great Britain suggests that at least some “leave” voters relied on predictions that they’d never win in order to cast an emotionally satisfying protest vote.

When I think about predictions and education, however, I see danger, not merely because predictions can be wrong, particularly when we’re applying aggregated data to individual behavior, but also because those “predictions” – even the “data driven” ones - often tell us more about what happened in the past, than what will happen in the future, and in focusing on the future, we turn a blind eye to remedying the conditions that allow the past to repeat with each successive generation of students.

Take, for example, the famous Stanford Marshmallow Experiment in which children were presented with a treat (like a marshmallow) and told that if they waited back to eat the marshmallow until the researcher came back, they would get double the amount. Those children who resisted the temptation of a single treat for the future promise of more treats are superior at “delayed gratification,” a character trait associated with both better health and increased academic achievement.

Knowing this, we decided that it’s important to teach children self-regulation and an entire industries were spawned: character education, “no excuses” charters, grit, etc…

But those correlations revealed by the Stanford Marshmallow Experiment are not causations. These so-called “tests of character” are more likely to reveal the lived experience of the child prior to the test. In a world where adults don’t keep their promises, denying oneself the marshmallow means you get zero marshmallows, rather than one, and one is better than zero.

Much of the data we look at to improve our futures only explains our pasts, and gives little insight as to what we can do to genuinely impact those futures and tells us nothing about how to create conditions where more children arrive having experienced a world where adults keep their promises about how many marshmallow’s you’re going to get.

So what happens when we start to predict student outcomes before a course even starts?

We have lots of correlations between behaviors and outcomes. For example, a recent report on community college students found that students who “take 15 credits’ worth of classes in their first semester are more likely to graduate than those students who enroll with only 12 credits.”

Another example: at University of Maryland University College, they’ve noticed that when it comes to online education, students who access the course prior to its start are more likely complete it successfully. As reported by Goldie Blumenstyk in CHE, UMUC is now “nudging” students to access materials prior to the start of class in the hopes that this may move the needle on completion.

Blumenstyk’s article explores some of the current conversations regarding the ethics of data collection as educators weigh the rights of students in terms of privacy and transparency. Groups are concerned about ensuring that all of this data collection doesn’t infringe on students’ “open futures.”

(They should be concerned, and I’ll say it again, averages don’t and shouldn’t apply to individuals, at least not if freedom of choice is a fundamental value in education.)

From my perspective, the major difficulty of all of this data is that it provides us reams of numbers about “what” is happening, but almost no insight into “why” what’s happening is happening.

Isn’t it possible that some students aren’t signing up for 15 credit hours in their first semester of community college because they’re working full-time or taking care of a family and have decided that 15 hours simply isn’t doable? In fact, couldn’t some of those people who are taking 12 hours discover that indeed, even 12 hours isn’t doable when weighed against their other obligations?

For those who don’t access online course materials prior to the start of class, perhaps they’re simply too busy, not lacking in preparation or motivation, but time and space.

Those nudges to enroll in more credit hours or get an early jump on a course doesn’t work for people who are already stretched to the limit to even get to 12 hours, or to enroll in an online course. Maybe the proper approach is to tell incoming students to take 15 hours if they can. For others it may be to take fewer than 12 hours.

Of course, under current conditions, perhaps some students are doomed to failure no matter what, and by focusing on questions of “what,” (take 15 hours/access course early) we allow ourselves to keep from confronting the much more important questions of “why.”

Subsequent research surrounding the original Stanford Marshmallow Test has shown how damaging a narrow focus can be. The seductive narrative that says we just need to keep kids from eating the marshmallow keeps us from confronting a world where many kids would be downright foolish if they didn’t eat the marshmallow.

The key to community college completion isn’t taking 15 hours of coursework to start, it’s having the kind of life situation that allows one to be a full-time student. We can’t nudge students into having 30 hours in a day.

Is it better to put students in the academic equivalent of high altitude with occasional puffs of oxygen, or to let them work at sea level?

If we want students to succeed we need to worry less about giving them the right “tools” and instead create conditions that are conducive to thriving, in which case those tools are largely unnecessary.

I’m not against data collection or using technology to help students succeed, but we have a long history of ignoring what’s important for what’s convenient. All this data doesn’t change our obligation to create a culture that allows for access to education and a reasonable path for success.

That’s not a data engineering problem, but a question of whether or not we can live up to the values we claim to hold.