Signals has had a rough few months. Blog posts,articles, and pointed posts on social media have recently taken both the creators and promoters of the tool to task for inflated retention claims and for falling into common statistical traps in making those claims. Some of the wounds are self-inflicted — not responding is rarely received well. Others, however, are misunderstandings of the founding goals and the exciting next phases of related work.
Signals is a technology application originally created by a team at Purdue University that uses a basic rules-based set of predictions and triggers — based on years of educational research and insight from the university's faculty — and combines them with real-time activity of students in a given course. It then uses a “traffic light” interface with student that sends them a familiar kind of message:
Green light: you’re doing well and on the right track.
Yellow light: you’re a little off-track, you might want to think about x,y, or z or talk with someone.
Red light: you’re in trouble. You probably need to reach out to someone for help.
These same data are also shared with faculty and administrators so they can reach out to those who need specific support in overcoming an academic challenge or just extra encouragement. Signals is now part of the services offered by Ellucian, but just about all the major players in education technology offer some version of "early warning" applications. In our insight and analytics work, a number of colleges and universities are piloting related apps; however, the promise and problems of Signals are an important predicate as we move forward with that work.
The Signals app project began with a clear and compelling goal: to allow students, faculty, and advisers access to data that might help them navigate the learning journey. For too long, the key data work in education has been focused on reporting, accreditation, or research that leads to long reports that few people see and are all too often used to make excuses, brag, blame, or shame. More problematic, most of these uses happen after courses are over or worse, after students have already dropped out.
The Signals team was trying to turn that on its head by gathering some useful feedback data we know from research may help students navigate a given course, and to give more information to faculty and administrators dedicated to helping them in the process. The course-level outcomes were strong. More students earned As, fewer got Fs, and the qualitative comments made it clear that many students appreciated the feedback. A welcome “wake-up call,” many called it.
John Campbell, then the associate vice president of information technology at Purdue, was committed to this vision. In numerous presentations he argued that “Signals was an attempt to take what decades of educational research was saying was important — tighter feedback loops — and create a clean, simple way for students, faculty, and advisers to get feedback that would be useful.”
Signals was a vital, high-profile first step in the process of turning the power of educational data work toward getting clean, clear, and useable information to the front lines. It was a pioneer in this work and should be recognized as such. The trouble is the conflation of this work with large-scale retention and student success efforts. Claiming a 21 percent long-term retention lift, as some at Purdue have, is a significant stretch at best. However, Signals has shown itself to be a useful tool to help students navigate specific courses, and for faculty and staff striving to supporting them. And while that will likely be useful in long-term retention, there is still much work to be done to both bring Signals to the next level of utility in courses and to test its impact on larger student success initiatives.
First, as Campbell, now CIO at West Virginia University notes, Signals has to truly leverage analytics. In our recent conversation he posited, “The only way to bring apps like Signals to their full potential, to bring them to scale, to make them sustainable is through analytics.” Front-line tools like Signals have to be powered by analyses that bring better and more personalized insight into individual students based on large-scale, consistently updated, student-level predictive models of pathways through a given institution. Put simply, basing the triggers and tools of these front-line systems on blunt, best-practice rules is not personalized, but generalized. It’s probably useful, but not optimal for that individual student. There needs to be a “next generation” of Signals, as Campbell notes, one that is more sophisticated and personalized.
For example, with a better understanding of the entire student pathway derived from analytics anchored on individual-level course completion, retention, and graduation predictions, a student who was highly likely to struggle in a given course from day one — e.g., a student having consistent difficulty with writing-intensive courses who is trying to take three simultaneously — might be advised away from a combination of courses that could be toxic for him or her. By better balancing the course selection, the problem — which would not necessarily be the challenge of a given course — could be solved before it begins. In addition, an institution may find that for a cluster of students standard “triggers” for intervention are meaningless. We’ve seen institutions that are serving military officers who have stellar completion and grade patterns over multiple semesters; however, because of the challenges of their day jobs, regular attendance patterns are not the norm. A generalized predictive model that pings instructors, advisers, or automated systems to intervene with these students may be simply annoying a highly capable student and/or wasting the time of faculty and advisers who are pushed to intervene.
Second, these tools have to be studied and tuned to better understand and maximize their positive impact on diverse student populations. With large-scale predictive flow models of student progression and propensity-score matching, for example, we can better understand how these tools contribute to long-term student success. Moreover, we can do tighter testing on the impact of user-interface design.
Indeed, we have a lot to learn about how we bring the right data to the right people – students, faculty, and advisers — in the right way. A red traffic light flashing in the face of a first-generation student that says, “You are likely to fail” might be a disaster. It might just reaffirm what he or she feared all along (e.g., “I don’t belong here”) and lead to dropping out. Is there a better way to display the data that would be motivating to that student?
The chief data scientist at our company, David Kil, comes from the world of health care, where they have learned the lessons of the impact of lifespan analysis and rapidly testing interventions. He points out the importance of knowing both when to intervene and how to intervene. Moreover, they learned that sometimes data is best brought right to the patient in an app or even an SMS message, other times the message is better sent through nurses or peer coaches, other times a conversation with a physician is the game changer. Regardless, testing the intervention for interface and impact on unique patient types, and its impact on long-term health, is a must.
The parallel in education is clear: Signals was an important first step to break the data wall and bring more focus to the front lines. However, as Campbell notes, if we want these kinds of tools to become more useful, we need to design them with triggers and tools grounded in truly predictive models and create a large-scale community of practice to test their impact and utility with students, faculty, and advisers – and their long-term contribution to retention and graduation. Moreover, as Mike Caulfield notes, technology-assisted interventions need to be put in the larger context of other interventions and strategies, many of which are deeply personal and/or driven by face-to-face work in instruction, advising, and coaching. Indeed, front-line apps at their best might make the human moments in education more frequent, informed, and meaningful. Because, let’s be clear about it, students don’t get choked up about apps that changed their lives at graduation.
Mark Milliron is Civitas Learning's chief learning officer and co-founder.
A recent article in The Economist, “Learned Luddites,” described liberal arts instructors who refused to adopt MOOCs as “Luddites,” a term made famous in the 19th century by English textile workers who were so paranoid that machinery would replace their jobs that they took to the task of physically destroying the machines they used. To conclude there is a connection between what the Luddites did and the arguments against online learning is reaching, if not absurd, and devalues the discussion happening in academic departments nation wide.
In America, after the launch of Sputnik in 1957 and the creation of the National Defense Education Act of 1958, emphasis was placed on math, science, and foreign language studies, as these three disciplines were deemed crucial to national security. Move forward 10 years and by the late 1960s one out of seven Americans was employed in the defense industry, military spending had risen from 1 percent to 10 percent of the gross domestic product, and corporations were increasingly profiting from an infusion of money from government contracts.
At the same time, high debt from domestic spending combined with outside competition from foreign markets was having an affect, and by the mid-1970s America had slipped into post-industrialism as jobs moved away from manufacturing toward more office based and service type employment opportunities.
The end result of shifting from assembly line to office tech, resulted in a college degree becoming a necessary component to a career, and as universities and community colleges began to accept more and more applicants, higher education began to trend course loads to part-time instructors.
Today, in 2013, a majority of those teaching in academia are working on a contingent basis. Tenure is nearly nonexistent, and liberal arts professors are being made to feel as though they are simply no more than an application, a helpmate, so to speak, that guides the student along as though they were a navigator steering a ship, following a mapped course not set by them, but by some far-off captain who serves as a default programmer for a higher purpose that is kept hush-hush until the time is right, a captain whose job it is to make sure the cargo arrives on time and without any scuffing from the occasional rogue wave.
At worst, more than a few professors feel they are becoming little more than a retention tool, a gimmick or novelty act whose entire future depends on whether or not one can “get with the program” of algorithmic evaluation, spreadsheet printouts, and constant barrage of software programs designed to make keeping track of grades easier, as if a pen and pad were inherently inferior, and all the while the academic is asked to maintain a classroom atmosphere that is not only educational but also so entertaining that even the most mind-numbing of subjects can compete against the fixative trance of the portable handheld device.
Ironically, the analog education one received before the Digital Age, an educational model that emphasized literature and writing, is admired for its fine attention to detail, as detail is considered to be hallmark of success. Yet that style of learning, though suitable for Fitzgerald and Stein, will not work in world where students are groomed as future customers and national security is commingled with corporate wants that drive the areas of study that schools find most lucrative.
It is pathetically sad to think that a classroom could be reduced to a rectangle screen on a distant wall, or thought to be comparable to that of a interior space where a qualified human stands as the moderator before eyes that are watching. A cold, sterile scene from Orwell's 1984 comes to mind in a world where the educator is 20 miles away and the students are considered close.
As a professor, I am not opposed to online teaching, but I do believe we are losing more than we are gaining from a technological hypnosis that has the potential to reclassify the teacher as a network administrator. I am not a lab rat, nor do I want the classroom considered a lab. Our culture is fascinated with language bewitchment and making the obvious appear novel. Yet at the end of the day the MOOC is still no more than a student interacting with a computer regardless how convenient or user friendly the experience has become.
If our embracing and use of technology becomes more important than our mission to teach, to meet in groups for discussion, or to sit one-on-one with a student seeking guidance, then not only should online education be critically evaluated for its unintended affects but also the very system itself that would interpret skepticism as a regress.
Brooks Kohler is an adjunct instructor with an M.A. in history.