Prior learning assessment could be higher education's next big disruptive force, and ACE and CAEL are poised to catch that potential gold rush. But many remain skeptical about academic credit for work experience.
In a rare moment of inattention a couple of years ago, I let myself get talked into becoming the chair of my campus’s Institutional Review Board. Being IRB chair may not be the best way to endear oneself to one’s colleagues, but it does offer an interesting window into how different disciplines conceive of research and the many different ways that scholarly work can be used to produce useful knowledge.
It has also brought home to me how utterly different research and assessment are. I have come to question why anyone with any knowledge of research methods would place any value on the results of typical learning outcomes assessment.
IRB approval is required for any work that involves both research and human subjects. If both conditions are met, the IRB must review it; if only one is present, the IRB can claim no authority. In general, it’s pretty easy to tell when a project involves human subjects, but distinguishing nonresearch from research, as it is defined by the U.S. Department of Health and Human Services, is more complicated. It depends in large part on whether the project will result in generalizable knowledge.
Determining what is research and what is not is interesting from an IRB perspective, but it has also forced me to think more about the differences between research and assessment. Learning outcomes assessment looks superficially like human subjects research, but there are some critical differences. Among other things, assessors routinely ignore practices that are considered essential safeguards for research subjects as well as standard research design principles.
A basic tenet of ethical human subjects research is that the research subjects should consent to participate. That is why obtaining informed consent is a routine part of human subject research. In contrast, students whose courses are being assessed are typically not asked whether they are willing to participate in those assessments. They are simply told that they will be participating. Often there is what an IRB would see as coercion. Whether it’s 20 points of extra credit for doing the posttest or embedding an essay that will be used for assessment in the final exam, assessors go out of their way to compel participation in the study.
Given that assessment involves little physical or psychological risk, the coercion of assessment subjects is not that big of a deal. What is more interesting to me is how assessment plans ignore most of the standard practices of good research. In a typical assessment effort, the assessor first decides what the desired outcomes in his course or program are. Sometimes the next step is to determine what level of knowledge or skill students bring with them when they start the course or program, although that is not always done. The final step is to have some sort of posttest or “artifact” -- assessmentspeak for a student-produced product like a paper rather than, say, a potsherd -- which can be examined (invariably with a rubric) to determine if the course or program outcomes have been met.
On some levels, this looks like research. The pretest gives you a baseline measurement, and then, if students do X percent better on the posttest, you appear to have evidence that they made progress. Even if you don’t establish a baseline, you might still be able to look at a capstone project and say that your students met the declared program-level outcome of being able to write a cogent research paper or design and execute a psychology experiment.
From an IRB perspective, however, this is not research. It does not produce generalizable knowledge, in that the success or, more rarely, failure to meet a particular course or program outcome does not allow us to make inferences about other courses or programs. So what appears to have worked for my students, in my World History course, at my institution, may not provide any guidance about what will work at your institution, with your students, with your approach to teaching.
If assessment does not offer generalizable knowledge, does assessment produce meaningful knowledge about particular courses or programs? I would argue that it does not. Leaving aside arguments about whether the blunt instrument of learning outcomes can capture the complexity of student learning or whether the purpose of an entire degree program can be easily summed up in ways that lend themselves to documentation and measurement, it is hard to see how assessment is giving us meaningful information, even concerning specific courses or programs.
First, the people who devise and administer the assessment have a stake in the outcome. When I assess my own course or program, I have an interest in the outcome of that assessment. If I create the assessment instrument, administer it and assess it, my conscious or even unconscious belief in the awesomeness of my own course or program is certain to influence the results. After all, if my approach did not already seem to be the best possible way of doing things, as a conscientious instructor, I would have changed it long ago.
Even if I were the rare human who is entirely without bias, my assessment results would still be meaningless, because I have no way of knowing what caused any of the changes I have observed. I have never seen a control group used in an assessment plan. We give all the students in the class or program the same course or courses. Then we look at what they can or cannot do at the end and assume that the course work is the cause of any change we have observed. Now, maybe this a valid assumption in a few instances, but if my history students are better writers at the end of the semester than they were at the beginning of the semester, how do I know that my course caused the change?
It could be that they were all in a good composition class at the same time as they took my class, or it could even be the case, especially in a program-level assessment, that they are just older and their brains have matured over the last four years. Without some group that has not been subjected to my course or program to compare them to, there is no compelling reason to assume it’s my course or program that’s causing the changes that are being observed.
If I developed a drug and then tested it myself without a control group, you might be a bit suspicious about my claims that everyone who took it recovered from his head cold after two weeks and thus that my drug is a success. But these are precisely the sorts of claims that we find in assessment.
I suspect that most academics are either consciously aware or at least unconsciously aware of these shortcomings and thus uneasy about the way assessment is done. That no one says anything reflects the sort of empty ritual that assessment is. Faculty members just want to keep the assessment office off their backs, the assessment office wants to keep the accreditors at bay and the accreditors need to appease lawmakers, who in turn want to be able to claim that they are holding higher education accountable.
IRBs are not supposed to critique research design unless it affects the safety of human subjects. However, they are supposed to weigh the balance between the risks posed by the study and the benefits of the research. Above all, you should not waste the time or risk the health of human subjects with research that is so poorly designed that it cannot produce meaningful results.
So, acknowledging that assessment is not research and not governed by IRB rules, it still seems that something silly and wasteful is going on here. Why is it acceptable that we spend more and more time and money -- time and money that have real opportunity costs and could be devoted to our students -- on assessment that is so poorly designed that it does not tell us anything meaningful about our courses or students? Whose interests are really served by this? Not students. Not faculty members.
It’s time to stop this charade. If some people want to do real research on what works in the classroom, more power to them. But making every program and every faculty member engage in nonresearch that yields nothing of value is a colossal, frivolous waste of time and money.
Erik Gilbert is a professor of history at Arkansas State University.
Submitted by Sarah Bray on November 15, 2016 - 3:00am
Is English 101 really just English 101? What about that first lab? Is a B or C in either of those lower-division courses a bellwether of a student’s likelihood to graduate? Until recently, we didn’t think so, but more and more, the data are telling us yes. In fact, insights from our advanced analytics have helped us identify a new segment of at-risk students hiding in plain sight.
It wasn’t until recently that the University of Arizona discovered this problem. As we combed through volumes of academic data and metrics with our partner, Civitas Learning, it became evident that students who seemed poised to graduate were actually leaving at higher rates than we could have foreseen. Why were good students -- students with solid grades in their lower-division foundational courses -- leaving after their first, second or even third year? And what could we do to help them stay and graduate from UA?
There’s a reason it’s hard to identify which students fall into this group; they simply don’t exhibit the traditional warning signs as defined by the retention experts. These students persist into the higher years but never graduate despite the fact that they’re strong students. They persist past their first two years and over 40 percent have GPAs above 3.0 -- so how does one diagnose them as at risk when all metrics indicate that they’re succeeding? Now we’re taking a deeper look at the data from the entire curriculum to find clues about what these students really need and even redefine our notion of what “at risk” really means.
Lower-division foundational courses are a natural starting point for us. These are the courses where basic mastery -- of a skill like writing or the scientific process -- begins, and mastery of these basics increases in necessity over the years. Writing, for instance, becomes more, not less, important over students’ academic careers. A 2015 National Survey of Student Engagement at UA indicated that the number of pages of writing assigned in the academic year to freshmen is 55, compared to 76 pages for seniors. As a freshman or sophomore, falling behind even by a few fractions can hurt you later on.
To wit, when a freshman gets a C in English 101, it doesn’t seem like a big deal -- why would it? She’s not at risk; she still has a 3.0, after all. But this student has unintentionally stepped into an institutional blind spot, because she’s a strong student by all measures. Our data analysis now shows that this student may persist until she hits a wall, usually during her major and upper-division courses, which is oftentimes difficult to overcome.
Let’s fast forward two years, then, when that same freshman is a junior enrolled in demanding upper-level classes. Her problem, a lack of writing command, has compounded into a series of C’s or D’s on research papers. A seemingly strong student is now at risk to persist, and her academic life becomes much less clear. We all thought she was on track to graduate, but now what? From that point, she may change her major, transfer to another institution or even exit college altogether. In the past, we would never have considered wraparound support services for students who earned a C in an intro writing course or a B in an intro lab course, but today we understand that we have to be ready and have to think about a deeper level of academic support across the entire life cycle of an undergrad.
Nationally, institutions like ours have developed many approaches to addressing the classic challenges of student success, developing an infrastructure of broad institutional interventions like centralized tutoring, highly specialized support staff, supplemental classes and more. Likewise, professors and advisers have become more attuned to responding to the one-on-one needs of students who may find themselves in trouble. There’s no doubt that this high/low approach has made an impact and our students have measurably benefited from it. But to assist students caught in the middle, those that by all measurement are already “succeeding,” we have to develop a more comprehensive institutional approach that works at the intersections of curricular innovation and wider student support.
Today, we at UA are adding a new layer to the institutional and one-to-one approaches already in place. In our courses, we are pushing to ensure that mastery matters more than a final grade by developing metrics and models that are vital to student learning. This, we believe, will lead to increases in graduation rates. We are working hand in hand with college faculty members, administrators and curriculum committees, arming those partners with the data necessary to develop revisions and supplementary support for the courses identified as critical to graduation rather than term-over-term persistence. We are modeling new classroom practices through the expansion of student-centered active classrooms and adaptive learning to better meet the diverse needs of our students.
When mastery is what matters most, the customary objections to at-risk student intervention matter less. Grade inflation by the instructor and performance for grade by the student become irrelevant. A foundational course surrounded by the support that a student often finds in lower-division courses is not an additional burden to the student, but an essential experience. Although the approach is added pressure on the faculty and staff, it has to be leavened with the resources that help both the instructor and the students succeed.
This is a true universitywide partnership to help a population of students who have found themselves unintentionally stuck in the middle. We must be data informed, not data driven, in supporting our students, because when our data are mapped with a human touch, we can help students unlock their potential in ways even they couldn’t have imagined.
Angela Baldasare is assistant provost for institutional research. Melissa Vito is senior vice president for student affairs and enrollment management and senior vice provost for academic initiatives and student success. Vincent J. Del Casino Jr. is provost of digital learning and student engagement and associate vice president of student affairs and enrollment management at the University of Arizona.
Inside Higher Edrecently checked up on adoption of badges specifically, and alternative credentialing generally, with a look at early adopter Illinois State University’s rollout of a badge platform. The overarching goal of badging and alternative credentialing initiatives is very valuable: to better communicate the value and variety of people’s skills to employers so that it’s easier to connect with and improve job outcomes. Yet the focus on badges and alternative credentials is like trying to facilitate global trade by inventing Esperanto.
The conception, theory and adoption of badge-based alternative credentialing initiatives starts as far back as 2011, when Mozilla announced the launch of its Open Badge Initiative and HASTAC simultaneously made “Digital Badges for Lifelong Learning” the theme of its fourth Digital Meaning & Learning competition. In the five years since, much has been written and even more time spent developing the theory and practice of alternative credentialing via badges -- from Mozilla and its support by the MacArthur Foundation to Purdue University’s Passport, to BadgeOS and Badge Alliance. Lately, the Lumina Foundation has taken the lead promoting alternative credentialing, most recently participating in a $2.5 million investment in badge platform Credly and a $1.3 million initiative to help university registrars develop a “new transcript.”
The premise behind all of the badge and alternative credential projects is the same: that if only there were a new, unified way to quantify, describe and give evidence of student learning inside the classroom and out, employers would be able to appropriately value those skills and illuminate a path to job outcomes. These kinds of premises often lead to utopian, idealized solutions that imagine transforming society itself. From Lumina’s “Strategy 8” overview:
To maximize our collective potential as a society, we need a revamped system of postsecondary credentials -- a fully integrated system that is learning based, student centered, universally understood and specifically designed to ensure quality at every level.
The problem for Lumina, Mozilla, Credly and the rest is that they’re proposing to replace a rich variety of credential “languages” with a universal one that’s not just unnecessary, but that’s modeled on fundamentally flawed analogies and observations.
I’ll start with the flaws of badges as a credentialing solution. Early on, digital badges often used Boy and Girl Scout badges as an analogy, but the more direct precursor of the current generation of badge solutions is video games. Indeed, attaining badges for completing certain tasks or reaching certain milestones is such a core feature of video game design and experience that the whole practice of rewarding behavior within software is referred to as “gamification.” This approach became widespread (with the launch of Foursquare, Gowalla, GetGlue and dozens more) in the years just preceding the launch of digital badges.
Yet video game badges -- and the badges employed by gamification companies -- are not truly credentials, but behaviorist reward systems designed to keep people on task. As credentials, their only useful meaning was within the systems in which they were earned, specifically within a given video game or bar-hopping app. Scout badges have a similar limitation: whatever their value in motivating attainment toward a worthy skill or outcome, the meaning of those badges is difficult to assess for nonscouts, or those not trained in the visual language of scouting badges.
Badge adherents aim to address the “value” and portability of badges by attaching proof of skills to the badges themselves. This is the same idea behind e-portfolios: that evidence of each skill is not just demonstrable, verifiable and universally understood, but useful to employers. Yet outside of specific fields, portfolios simply don’t matter to employers. As Anthony Carnevale, director of Georgetown University’s Center on Education and the Workforce, told The Chronicle of Higher Education earlier this year about the New Transcript portfolio, “Employers don’t want to take time to go through your portfolio -- they just don’t.” Where evidence of skills is important and useful, solutions already exist: GitHub for software developers; Behance for designers; transcripts, essays and recommendations for graduate school.
The idea of replacing university “dialects” with a new language of skills and outcomes is less metaphorical when think tanks and ed-tech companies talk about alternative credentials as a category. There, advocates propose an entirely new vocabulary: microcredentials, nanodegrees, stackable badges and more, all meant to convey (to employers primarily) the body of skills and knowledge that a student possesses. But they are redefining concepts that already exist, and that exist productively for the marketplace of students, educators and employers.
Consider the stackable badge, the idea that learning competencies should be assessed and verified in a progression that comprises and leads to a certified credential. But stackable credentials already exist in ways that everyone understands. In the undergraduate major, a student completes a series of related and escalating levels of mastery in a given subject area, assessed by experts in that field. Upon completion of those microcredentials -- i.e., classes -- the student is awarded a degree with a focus in that field and with an indication of attainment (honors). The same goes for hundreds of areas of expertise inside and outside higher education: in financial analysis (the extremely demanding and desirable CFA designation), entry-level and advanced manufacturing (the National Association of Manufacturers MSCS system), specific IT areas of focus like ISACA and (ISC)2, bar exams, medical boards, and more.
Credentials, in and of themselves, are a solved problem. I know this because my own company, Merit, launched the biggest, most comprehensive badge experiment that no one has heard of. Between 2011 and 2014 we tested a variation of the scout model -- a badge-based visual language of college milestones and credentials analogous to a military officer’s dress uniform -- that could be quickly read to convey a person’s skills, accomplishments and level of achievement. Nearly 500 colleges granted more than three million students almost 10 million badges that included academic honors, notable cocurriculars, experiential learning, internships and more. We tested interest by employers, educators and students (and continue to). What’s clear is this: it’s far, far more important to simply document existing credentials than to invent new ones, or a new language to describe them. Stakeholders in the high-school-to-college-to-career pipeline understand and value credentials as they exist now, and rarely need or want a new way to understand them. They just want to see them.
Connecting students’ skills and ambitions to the pathways to a career is a big deal, but it doesn’t require a new language that’s based on techno-solutionist fantasies. LinkedIn, the “economic graph” that many hold up as a model, needed more than $100 million of private capital for something as simple as convincing managers and a certain professional class to keep updated résumés online. Doing something similar for every single student is both more valuable and more difficult -- and doesn’t need to reinvent the entire language of credentials to complicate the effort.
My biggest frustration with badges and alternative credentials isn’t that they are an ivory tower solution to a real world problem. It’s that helping students succeed means more than figuring out a new language. Higher education is a demanding, high-stakes endeavor for the vast majority of students. Proposing that they -- and the institutions educating them and the employers who might hire them -- learn a new lingua franca for conveying the value of that learning, every year, over the very short time that they’re mastering the skills and knowledge that they need isn’t just impractical. It’s unfair.
Colin Mathews is founder and president of Merit, a technology company focused on creating and sharing stories about students’ successes.
Technology think tank says standardized testing by outside groups and alternative forms of credentialing could create helpful competitive pressure on higher education and the traditional college degree.