Assessment

Improving Graduation Rates Is Job One at City Colleges of Chicago

Smart Title: 

City Colleges of Chicago have a 7 percent graduation rate. If that number doesn't go up, the system's chancellor, presidents and trustees could lose their jobs.

Essay on flawed assumptions behind digital badging and alternative credentialing

Inside Higher Ed recently checked up on adoption of badges specifically, and alternative credentialing generally, with a look at early adopter Illinois State University’s rollout of a badge platform. The overarching goal of badging and alternative credentialing initiatives is very valuable: to better communicate the value and variety of people’s skills to employers so that it’s easier to connect with and improve job outcomes. Yet the focus on badges and alternative credentials is like trying to facilitate global trade by inventing Esperanto.

The conception, theory and adoption of badge-based alternative credentialing initiatives starts as far back as 2011, when Mozilla announced the launch of its Open Badge Initiative and HASTAC simultaneously made “Digital Badges for Lifelong Learning” the theme of its fourth Digital Meaning & Learning competition. In the five years since, much has been written and even more time spent developing the theory and practice of alternative credentialing via badges -- from Mozilla and its support by the MacArthur Foundation to Purdue University’s Passport, to BadgeOS and Badge Alliance. Lately, the Lumina Foundation has taken the lead promoting alternative credentialing, most recently participating in a $2.5 million investment in badge platform Credly and a $1.3 million initiative to help university registrars develop a “new transcript.”

The premise behind all of the badge and alternative credential projects is the same: that if only there were a new, unified way to quantify, describe and give evidence of student learning inside the classroom and out, employers would be able to appropriately value those skills and illuminate a path to job outcomes. These kinds of premises often lead to utopian, idealized solutions that imagine transforming society itself. From Lumina’s “Strategy 8” overview:

To maximize our collective potential as a society, we need a revamped system of postsecondary credentials -- a fully integrated system that is learning based, student centered, universally understood and specifically designed to ensure quality at every level.

The problem for Lumina, Mozilla, Credly and the rest is that they’re proposing to replace a rich variety of credential “languages” with a universal one that’s not just unnecessary, but that’s modeled on fundamentally flawed analogies and observations.

I’ll start with the flaws of badges as a credentialing solution. Early on, digital badges often used Boy and Girl Scout badges as an analogy, but the more direct precursor of the current generation of badge solutions is video games. Indeed, attaining badges for completing certain tasks or reaching certain milestones is such a core feature of video game design and experience that the whole practice of rewarding behavior within software is referred to as “gamification.” This approach became widespread (with the launch of Foursquare, Gowalla, GetGlue and dozens more) in the years just preceding the launch of digital badges.

Yet video game badges -- and the badges employed by gamification companies -- are not truly credentials, but behaviorist reward systems designed to keep people on task. As credentials, their only useful meaning was within the systems in which they were earned, specifically within a given video game or bar-hopping app. Scout badges have a similar limitation: whatever their value in motivating attainment toward a worthy skill or outcome, the meaning of those badges is difficult to assess for nonscouts, or those not trained in the visual language of scouting badges.

Badge adherents aim to address the “value” and portability of badges by attaching proof of skills to the badges themselves. This is the same idea behind e-portfolios: that evidence of each skill is not just demonstrable, verifiable and universally understood, but useful to employers. Yet outside of specific fields, portfolios simply don’t matter to employers. As Anthony Carnevale, director of Georgetown University’s Center on Education and the Workforce, told The Chronicle of Higher Education earlier this year about the New Transcript portfolio, “Employers don’t want to take time to go through your portfolio -- they just don’t.” Where evidence of skills is important and useful, solutions already exist: GitHub for software developers; Behance for designers; transcripts, essays and recommendations for graduate school.

The idea of replacing university “dialects” with a new language of skills and outcomes is less metaphorical when think tanks and ed-tech companies talk about alternative credentials as a category. There, advocates propose an entirely new vocabulary: microcredentials, nanodegrees, stackable badges and more, all meant to convey (to employers primarily) the body of skills and knowledge that a student possesses. But they are redefining concepts that already exist, and that exist productively for the marketplace of students, educators and employers.

Consider the stackable badge, the idea that learning competencies should be assessed and verified in a progression that comprises and leads to a certified credential. But stackable credentials already exist in ways that everyone understands. In the undergraduate major, a student completes a series of related and escalating levels of mastery in a given subject area, assessed by experts in that field. Upon completion of those microcredentials -- i.e., classes -- the student is awarded a degree with a focus in that field and with an indication of attainment (honors). The same goes for hundreds of areas of expertise inside and outside higher education: in financial analysis (the extremely demanding and desirable CFA designation), entry-level and advanced manufacturing (the National Association of Manufacturers MSCS system), specific IT areas of focus like ISACA and (ISC)2, bar exams, medical boards, and more.

Credentials, in and of themselves, are a solved problem. I know this because my own company, Merit, launched the biggest, most comprehensive badge experiment that no one has heard of. Between 2011 and 2014 we tested a variation of the scout model -- a badge-based visual language of college milestones and credentials analogous to a military officer’s dress uniform -- that could be quickly read to convey a person’s skills, accomplishments and level of achievement. Nearly 500 colleges granted more than three million students almost 10 million badges that included academic honors, notable cocurriculars, experiential learning, internships and more. We tested interest by employers, educators and students (and continue to). What’s clear is this: it’s far, far more important to simply document existing credentials than to invent new ones, or a new language to describe them. Stakeholders in the high-school-to-college-to-career pipeline understand and value credentials as they exist now, and rarely need or want a new way to understand them. They just want to see them.

Connecting students’ skills and ambitions to the pathways to a career is a big deal, but it doesn’t require a new language that’s based on techno-solutionist fantasies. LinkedIn, the “economic graph” that many hold up as a model, needed more than $100 million of private capital for something as simple as convincing managers and a certain professional class to keep updated résumés online. Doing something similar for every single student is both more valuable and more difficult -- and doesn’t need to reinvent the entire language of credentials to complicate the effort.

My biggest frustration with badges and alternative credentials isn’t that they are an ivory tower solution to a real world problem. It’s that helping students succeed means more than figuring out a new language. Higher education is a demanding, high-stakes endeavor for the vast majority of students. Proposing that they -- and the institutions educating them and the employers who might hire them -- learn a new lingua franca for conveying the value of that learning, every year, over the very short time that they’re mastering the skills and knowledge that they need isn’t just impractical. It’s unfair.

Colin Mathews is founder and president of Merit, a technology company focused on creating and sharing stories about students’ successes.

Section: 
Editorial Tags: 
Image Source: 
iStock/DavidGoh

Initiative fatigue, lack of accountability preventing colleges from improving student outcomes

Smart Title: 

Report finds initiative fatigue and a lack of accountability, among other obstacles, are preventing colleges from improving student outcomes.

In England, a push to evaluate teaching quality and learning gains

Smart Title: 

More than 70 institutions are testing different measures of student learning amid new government effort to evaluate universities on teaching quality.

Awards recognize colleges that excel in assessing learning campuswide

Smart Title: 

Ten colleges earn national award for meaningful, cross-campus efforts to track student learning and use results to improve classroom practices.

IT think tank's call for alternative forms of credentialing and measuring competency

Smart Title: 

Technology think tank says standardized testing by outside groups and alternative forms of credentialing could create helpful competitive pressure on higher education and the traditional college degree.

WICHE's interstate passport seeks to help students transfer while preserving colleges' autonomy

Smart Title: 

New interstate network seeks to help students transfer across state lines without losing credits, but also defers to faculty members at each college about how to measure learning.

Defining disciplinary learning outcomes and principles for assessment (essay)

The higher education lore is that faculty members cannot agree on anything. Like other myths, this accepted folk wisdom is far from the truth. 

Indeed, over the course of our careers, we have repeatedly observed faculty members coming together collaboratively to address the challenges faced institutionally or in higher education more broadly. More recently, we have been heartened and inspired in particular by those who spent the last several years grappling with a fundamental question: what should students learn in higher education?

Instead of ignoring external pressures to measure and improve college outcomes, faculty members came together under the auspices of the Social Science Research Council's Measuring College Learning Project, which we have helped lead, to address these pedagogical challenges. Faculty members in six disciplines -- biology, business, communication, economics, history and sociology -- engaged in invigorating discussions, lively debates and difficult conversations. Supported by their disciplinary associations and encouraged by their collaborative spirit, they have articulated frameworks for defining learning outcomes in six disciplines and the principles for assessing learning outcomes in the 21st century, as described in the recently released Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century.

In our work, we have found that faculty members readily agree that higher education is not about efficient acquisition of surface content knowledge and the simple regurgitation of memorized facts.That does not mean that content is unimportant. Content is indeed crucial, but primarily as a building block for more complex forms of thinking. Faculty members are eager to get students to apply, analyze and evaluate from their disciplinary perspectives, to acquire a disciplinary mindset and think like a biologist or an economist.

Faculty members across disciplines in the MCL project rather quickly coalesced around “essential concepts and competencies” for their disciplines, which represent ideas and skills that faculty believe are fundamental to the discipline, valuable to students and worth emphasizing given limited time and resources.There are similarities across disciplines including an emphasis on analytical writing and problem-solving, but these generic skills take form, are defined and are honed within specific fields of study. They are not abstract ideas, but concepts and competencies that faculty members engage, develop and deploy in their work and value in their disciplines.  

Faculty members are also often seen as resisting assessment. But, in fact, what they resist are simplistic assessments of student learning that focus on recollection of knowledge, rely on blunt instruments and are narrow and reductionist. They resist, as would all other professions, externally imposed mandates that fail to reflect the complexity of their jobs or that misrepresent the purpose of higher education. But they also believe that what they are doing makes a difference -- that they are teaching students how to see the world in a new light -- and they would be eager to have the tools to demonstrate their contributions to the development of student cognitive capacities.

Constructive conversations about learning outcomes and assessments require the proper context and frame.That is rarely offered in a world in which we in higher education are on the defensive, trying to argue against externally proposed accountability measures based on distal labor market outcomes, instead of being proactive and making the case on our own terms.There is no shortage of proposals in the public sphere about what higher education should do. But those conversations often lack the voices of faculty members, who are the professionals with responsibility for defining, enabling and assessing what students learn.

The faculty should be at the forefront of the conversations about the purposes of higher education and thus at the center of defining and measuring undergraduate learning outcomes.That is not only a matter of professional duty but also of doing justice to our students.Students from all backgrounds and institutions should have an opportunity to demonstrate their knowledge and skills.

Years of institutionally incentivized grade inflation and proliferation of course titles have all but made transcripts irrelevant. In our research, we found that most employers do not even ask to see them. And while some recent efforts have aimed to add extracurricular activities and other accomplishments to college transcripts, none of those tell us what students actually know or can do.Taking a class is not the same thing as mastering the concepts and competencies presented. Being a member of a club similarly says little about the skills a student has developed.

In addition to placing faculty and student learning at the center of the conversation, the MCL project is committed to recognizing the complexity of what higher education aims to accomplish and ensuring that any measure of learning is part of a larger holistic assessment plan.The project focuses on the disciplines.That does not preclude making sure that students are also civically minded and globally competent. It only means that we need to be clear about which part of the puzzle one hopes to address with a disciplinary focused initiative.

The MCL project is committed to ensuring that institutions use assessment tools on a voluntary basis. We have elaborated elsewhere the pitfalls of externally imposed accountability. Only by willingly looking in the mirror will higher education institutions make progress toward improving student learning outcomes.

While assessment should be voluntary, it need not be a solitary endeavor. Collaborating with other institutions makes us not only realize that we all face challenges and struggle with current circumstances but also offers insight into possible ways forward. Measures of learning outcomes must be of high quality and comparable, so they can allow multiple institutions to use them and share their insights. Governed by the principle of continuous improvement, assessments -- albeit limited and imperfect -- are necessary tools on the road toward reaching our goals.

As we look toward the future, we are excited and energized by the commitment and thoughtfulness of the faculty members who participated in the MCL project.They have put forth a bold and forward-thinking vision for the future of learning and assessment in their disciplines: a set of frameworks that will be subject to ongoing iteration and improvement in the years ahead. Instead of waiting for the storm to subside, these faculty members and their disciplinary associations have tackled the challenge head on. They have paved the way for a more promising future.

Josipa Roksa is associate professor of sociology and education at the University of Virginia. Richard Arum is chair of sociology at New York University and incoming dean of the School of Education at the University of California at Irvine. They are the authors of Academically Adrift: Limited Learning on College Campuses (University of Chicago, 2011).

Editorial Tags: 
Image Source: 
istock

Growing number of community colleges use multiple measures to place students

Smart Title: 

More community colleges are moving away from relying on placement exams alone to figure out whether incoming students need remediation, but establishing a substitute system can be tricky.

Series of studies seeks to gauge higher ed effectiveness, defined broadly

Smart Title: 

New volume of research examines various aspects of higher education performance, going well beyond labor market outcomes to include academic quality and socioeconomic equity.

Pages

Subscribe to RSS - Assessment
Back to Top