Assessment

Developing metrics and models that are vital to student learning and retention (essay)

Is English 101 really just English 101? What about that first lab? Is a B or C in either of those lower-division courses a bellwether of a student’s likelihood to graduate? Until recently, we didn’t think so, but more and more, the data are telling us yes. In fact, insights from our advanced analytics have helped us identify a new segment of at-risk students hiding in plain sight.

It wasn’t until recently that the University of Arizona discovered this problem. As we combed through volumes of academic data and metrics with our partner, Civitas Learning, it became evident that students who seemed poised to graduate were actually leaving at higher rates than we could have foreseen. Why were good students -- students with solid grades in their lower-division foundational courses -- leaving after their first, second or even third year? And what could we do to help them stay and graduate from UA?

There’s a reason it’s hard to identify which students fall into this group; they simply don’t exhibit the traditional warning signs as defined by the retention experts. These students persist into the higher years but never graduate despite the fact that they’re strong students. They persist past their first two years and over 40 percent have GPAs above 3.0 -- so how does one diagnose them as at risk when all metrics indicate that they’re succeeding? Now we’re taking a deeper look at the data from the entire curriculum to find clues about what these students really need and even redefine our notion of what “at risk” really means.

Lower-division foundational courses are a natural starting point for us. These are the courses where basic mastery -- of a skill like writing or the scientific process -- begins, and mastery of these basics increases in necessity over the years. Writing, for instance, becomes more, not less, important over students’ academic careers. A 2015 National Survey of Student Engagement at UA indicated that the number of pages of writing assigned in the academic year to freshmen is 55, compared to 76 pages for seniors. As a freshman or sophomore, falling behind even by a few fractions can hurt you later on.

To wit, when a freshman gets a C in English 101, it doesn’t seem like a big deal -- why would it? She’s not at risk; she still has a 3.0, after all. But this student has unintentionally stepped into an institutional blind spot, because she’s a strong student by all measures. Our data analysis now shows that this student may persist until she hits a wall, usually during her major and upper-division courses, which is oftentimes difficult to overcome.

Let’s fast forward two years, then, when that same freshman is a junior enrolled in demanding upper-level classes. Her problem, a lack of writing command, has compounded into a series of C’s or D’s on research papers. A seemingly strong student is now at risk to persist, and her academic life becomes much less clear. We all thought she was on track to graduate, but now what? From that point, she may change her major, transfer to another institution or even exit college altogether. In the past, we would never have considered wraparound support services for students who earned a C in an intro writing course or a B in an intro lab course, but today we understand that we have to be ready and have to think about a deeper level of academic support across the entire life cycle of an undergrad.

Nationally, institutions like ours have developed many approaches to addressing the classic challenges of student success, developing an infrastructure of broad institutional interventions like centralized tutoring, highly specialized support staff, supplemental classes and more. Likewise, professors and advisers have become more attuned to responding to the one-on-one needs of students who may find themselves in trouble. There’s no doubt that this high/low approach has made an impact and our students have measurably benefited from it. But to assist students caught in the middle, those that by all measurement are already “succeeding,” we have to develop a more comprehensive institutional approach that works at the intersections of curricular innovation and wider student support.

Today, we at UA are adding a new layer to the institutional and one-to-one approaches already in place. In our courses, we are pushing to ensure that mastery matters more than a final grade by developing metrics and models that are vital to student learning. This, we believe, will lead to increases in graduation rates. We are working hand in hand with college faculty members, administrators and curriculum committees, arming those partners with the data necessary to develop revisions and supplementary support for the courses identified as critical to graduation rather than term-over-term persistence. We are modeling new classroom practices through the expansion of student-centered active classrooms and adaptive learning to better meet the diverse needs of our students.

When mastery is what matters most, the customary objections to at-risk student intervention matter less. Grade inflation by the instructor and performance for grade by the student become irrelevant. A foundational course surrounded by the support that a student often finds in lower-division courses is not an additional burden to the student, but an essential experience. Although the approach is added pressure on the faculty and staff, it has to be leavened with the resources that help both the instructor and the students succeed.

This is a true universitywide partnership to help a population of students who have found themselves unintentionally stuck in the middle. We must be data informed, not data driven, in supporting our students, because when our data are mapped with a human touch, we can help students unlock their potential in ways even they couldn’t have imagined.

Angela Baldasare is assistant provost for institutional research. Melissa Vito is senior vice president for student affairs and enrollment management and senior vice provost for academic initiatives and student success. Vincent J. Del Casino Jr. is provost of digital learning and student engagement and associate vice president of student affairs and enrollment management at the University of Arizona.

Image Source: 
iStock/jaker5000
Is this diversity newsletter?: 

Essay on flawed assumptions behind digital badging and alternative credentialing

Inside Higher Ed recently checked up on adoption of badges specifically, and alternative credentialing generally, with a look at early adopter Illinois State University’s rollout of a badge platform. The overarching goal of badging and alternative credentialing initiatives is very valuable: to better communicate the value and variety of people’s skills to employers so that it’s easier to connect with and improve job outcomes. Yet the focus on badges and alternative credentials is like trying to facilitate global trade by inventing Esperanto.

The conception, theory and adoption of badge-based alternative credentialing initiatives starts as far back as 2011, when Mozilla announced the launch of its Open Badge Initiative and HASTAC simultaneously made “Digital Badges for Lifelong Learning” the theme of its fourth Digital Meaning & Learning competition. In the five years since, much has been written and even more time spent developing the theory and practice of alternative credentialing via badges -- from Mozilla and its support by the MacArthur Foundation to Purdue University’s Passport, to BadgeOS and Badge Alliance. Lately, the Lumina Foundation has taken the lead promoting alternative credentialing, most recently participating in a $2.5 million investment in badge platform Credly and a $1.3 million initiative to help university registrars develop a “new transcript.”

The premise behind all of the badge and alternative credential projects is the same: that if only there were a new, unified way to quantify, describe and give evidence of student learning inside the classroom and out, employers would be able to appropriately value those skills and illuminate a path to job outcomes. These kinds of premises often lead to utopian, idealized solutions that imagine transforming society itself. From Lumina’s “Strategy 8” overview:

To maximize our collective potential as a society, we need a revamped system of postsecondary credentials -- a fully integrated system that is learning based, student centered, universally understood and specifically designed to ensure quality at every level.

The problem for Lumina, Mozilla, Credly and the rest is that they’re proposing to replace a rich variety of credential “languages” with a universal one that’s not just unnecessary, but that’s modeled on fundamentally flawed analogies and observations.

I’ll start with the flaws of badges as a credentialing solution. Early on, digital badges often used Boy and Girl Scout badges as an analogy, but the more direct precursor of the current generation of badge solutions is video games. Indeed, attaining badges for completing certain tasks or reaching certain milestones is such a core feature of video game design and experience that the whole practice of rewarding behavior within software is referred to as “gamification.” This approach became widespread (with the launch of Foursquare, Gowalla, GetGlue and dozens more) in the years just preceding the launch of digital badges.

Yet video game badges -- and the badges employed by gamification companies -- are not truly credentials, but behaviorist reward systems designed to keep people on task. As credentials, their only useful meaning was within the systems in which they were earned, specifically within a given video game or bar-hopping app. Scout badges have a similar limitation: whatever their value in motivating attainment toward a worthy skill or outcome, the meaning of those badges is difficult to assess for nonscouts, or those not trained in the visual language of scouting badges.

Badge adherents aim to address the “value” and portability of badges by attaching proof of skills to the badges themselves. This is the same idea behind e-portfolios: that evidence of each skill is not just demonstrable, verifiable and universally understood, but useful to employers. Yet outside of specific fields, portfolios simply don’t matter to employers. As Anthony Carnevale, director of Georgetown University’s Center on Education and the Workforce, told The Chronicle of Higher Education earlier this year about the New Transcript portfolio, “Employers don’t want to take time to go through your portfolio -- they just don’t.” Where evidence of skills is important and useful, solutions already exist: GitHub for software developers; Behance for designers; transcripts, essays and recommendations for graduate school.

The idea of replacing university “dialects” with a new language of skills and outcomes is less metaphorical when think tanks and ed-tech companies talk about alternative credentials as a category. There, advocates propose an entirely new vocabulary: microcredentials, nanodegrees, stackable badges and more, all meant to convey (to employers primarily) the body of skills and knowledge that a student possesses. But they are redefining concepts that already exist, and that exist productively for the marketplace of students, educators and employers.

Consider the stackable badge, the idea that learning competencies should be assessed and verified in a progression that comprises and leads to a certified credential. But stackable credentials already exist in ways that everyone understands. In the undergraduate major, a student completes a series of related and escalating levels of mastery in a given subject area, assessed by experts in that field. Upon completion of those microcredentials -- i.e., classes -- the student is awarded a degree with a focus in that field and with an indication of attainment (honors). The same goes for hundreds of areas of expertise inside and outside higher education: in financial analysis (the extremely demanding and desirable CFA designation), entry-level and advanced manufacturing (the National Association of Manufacturers MSCS system), specific IT areas of focus like ISACA and (ISC)2, bar exams, medical boards, and more.

Credentials, in and of themselves, are a solved problem. I know this because my own company, Merit, launched the biggest, most comprehensive badge experiment that no one has heard of. Between 2011 and 2014 we tested a variation of the scout model -- a badge-based visual language of college milestones and credentials analogous to a military officer’s dress uniform -- that could be quickly read to convey a person’s skills, accomplishments and level of achievement. Nearly 500 colleges granted more than three million students almost 10 million badges that included academic honors, notable cocurriculars, experiential learning, internships and more. We tested interest by employers, educators and students (and continue to). What’s clear is this: it’s far, far more important to simply document existing credentials than to invent new ones, or a new language to describe them. Stakeholders in the high-school-to-college-to-career pipeline understand and value credentials as they exist now, and rarely need or want a new way to understand them. They just want to see them.

Connecting students’ skills and ambitions to the pathways to a career is a big deal, but it doesn’t require a new language that’s based on techno-solutionist fantasies. LinkedIn, the “economic graph” that many hold up as a model, needed more than $100 million of private capital for something as simple as convincing managers and a certain professional class to keep updated résumés online. Doing something similar for every single student is both more valuable and more difficult -- and doesn’t need to reinvent the entire language of credentials to complicate the effort.

My biggest frustration with badges and alternative credentials isn’t that they are an ivory tower solution to a real world problem. It’s that helping students succeed means more than figuring out a new language. Higher education is a demanding, high-stakes endeavor for the vast majority of students. Proposing that they -- and the institutions educating them and the employers who might hire them -- learn a new lingua franca for conveying the value of that learning, every year, over the very short time that they’re mastering the skills and knowledge that they need isn’t just impractical. It’s unfair.

Colin Mathews is founder and president of Merit, a technology company focused on creating and sharing stories about students’ successes.

Section: 
Editorial Tags: 
Image Source: 
iStock/DavidGoh

Initiative fatigue, lack of accountability preventing colleges from improving student outcomes

Report finds initiative fatigue and a lack of accountability, among other obstacles, are preventing colleges from improving student outcomes.

In England, a push to evaluate teaching quality and learning gains

More than 70 institutions are testing different measures of student learning amid new government effort to evaluate universities on teaching quality.

Awards recognize colleges that excel in assessing learning campuswide

Ten colleges earn national award for meaningful, cross-campus efforts to track student learning and use results to improve classroom practices.

IT think tank's call for alternative forms of credentialing and measuring competency

Technology think tank says standardized testing by outside groups and alternative forms of credentialing could create helpful competitive pressure on higher education and the traditional college degree.

WICHE's interstate passport seeks to help students transfer while preserving colleges' autonomy

New interstate network seeks to help students transfer across state lines without losing credits, but also defers to faculty members at each college about how to measure learning.

Defining disciplinary learning outcomes and principles for assessment (essay)

The higher education lore is that faculty members cannot agree on anything. Like other myths, this accepted folk wisdom is far from the truth. 

Indeed, over the course of our careers, we have repeatedly observed faculty members coming together collaboratively to address the challenges faced institutionally or in higher education more broadly. More recently, we have been heartened and inspired in particular by those who spent the last several years grappling with a fundamental question: what should students learn in higher education?

Instead of ignoring external pressures to measure and improve college outcomes, faculty members came together under the auspices of the Social Science Research Council's Measuring College Learning Project, which we have helped lead, to address these pedagogical challenges. Faculty members in six disciplines -- biology, business, communication, economics, history and sociology -- engaged in invigorating discussions, lively debates and difficult conversations. Supported by their disciplinary associations and encouraged by their collaborative spirit, they have articulated frameworks for defining learning outcomes in six disciplines and the principles for assessing learning outcomes in the 21st century, as described in the recently released Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century.

In our work, we have found that faculty members readily agree that higher education is not about efficient acquisition of surface content knowledge and the simple regurgitation of memorized facts.That does not mean that content is unimportant. Content is indeed crucial, but primarily as a building block for more complex forms of thinking. Faculty members are eager to get students to apply, analyze and evaluate from their disciplinary perspectives, to acquire a disciplinary mindset and think like a biologist or an economist.

Faculty members across disciplines in the MCL project rather quickly coalesced around “essential concepts and competencies” for their disciplines, which represent ideas and skills that faculty believe are fundamental to the discipline, valuable to students and worth emphasizing given limited time and resources.There are similarities across disciplines including an emphasis on analytical writing and problem-solving, but these generic skills take form, are defined and are honed within specific fields of study. They are not abstract ideas, but concepts and competencies that faculty members engage, develop and deploy in their work and value in their disciplines.  

Faculty members are also often seen as resisting assessment. But, in fact, what they resist are simplistic assessments of student learning that focus on recollection of knowledge, rely on blunt instruments and are narrow and reductionist. They resist, as would all other professions, externally imposed mandates that fail to reflect the complexity of their jobs or that misrepresent the purpose of higher education. But they also believe that what they are doing makes a difference -- that they are teaching students how to see the world in a new light -- and they would be eager to have the tools to demonstrate their contributions to the development of student cognitive capacities.

Constructive conversations about learning outcomes and assessments require the proper context and frame.That is rarely offered in a world in which we in higher education are on the defensive, trying to argue against externally proposed accountability measures based on distal labor market outcomes, instead of being proactive and making the case on our own terms.There is no shortage of proposals in the public sphere about what higher education should do. But those conversations often lack the voices of faculty members, who are the professionals with responsibility for defining, enabling and assessing what students learn.

The faculty should be at the forefront of the conversations about the purposes of higher education and thus at the center of defining and measuring undergraduate learning outcomes.That is not only a matter of professional duty but also of doing justice to our students.Students from all backgrounds and institutions should have an opportunity to demonstrate their knowledge and skills.

Years of institutionally incentivized grade inflation and proliferation of course titles have all but made transcripts irrelevant. In our research, we found that most employers do not even ask to see them. And while some recent efforts have aimed to add extracurricular activities and other accomplishments to college transcripts, none of those tell us what students actually know or can do.Taking a class is not the same thing as mastering the concepts and competencies presented. Being a member of a club similarly says little about the skills a student has developed.

In addition to placing faculty and student learning at the center of the conversation, the MCL project is committed to recognizing the complexity of what higher education aims to accomplish and ensuring that any measure of learning is part of a larger holistic assessment plan.The project focuses on the disciplines.That does not preclude making sure that students are also civically minded and globally competent. It only means that we need to be clear about which part of the puzzle one hopes to address with a disciplinary focused initiative.

The MCL project is committed to ensuring that institutions use assessment tools on a voluntary basis. We have elaborated elsewhere the pitfalls of externally imposed accountability. Only by willingly looking in the mirror will higher education institutions make progress toward improving student learning outcomes.

While assessment should be voluntary, it need not be a solitary endeavor. Collaborating with other institutions makes us not only realize that we all face challenges and struggle with current circumstances but also offers insight into possible ways forward. Measures of learning outcomes must be of high quality and comparable, so they can allow multiple institutions to use them and share their insights. Governed by the principle of continuous improvement, assessments -- albeit limited and imperfect -- are necessary tools on the road toward reaching our goals.

As we look toward the future, we are excited and energized by the commitment and thoughtfulness of the faculty members who participated in the MCL project.They have put forth a bold and forward-thinking vision for the future of learning and assessment in their disciplines: a set of frameworks that will be subject to ongoing iteration and improvement in the years ahead. Instead of waiting for the storm to subside, these faculty members and their disciplinary associations have tackled the challenge head on. They have paved the way for a more promising future.

Josipa Roksa is associate professor of sociology and education at the University of Virginia. Richard Arum is chair of sociology at New York University and incoming dean of the School of Education at the University of California at Irvine. They are the authors of Academically Adrift: Limited Learning on College Campuses (University of Chicago, 2011).

Editorial Tags: 
Image Source: 
istock

Growing number of community colleges use multiple measures to place students

More community colleges are moving away from relying on placement exams alone to figure out whether incoming students need remediation, but establishing a substitute system can be tricky.

Series of studies seeks to gauge higher ed effectiveness, defined broadly

New volume of research examines various aspects of higher education performance, going well beyond labor market outcomes to include academic quality and socioeconomic equity.

Pages

Subscribe to RSS - Assessment
Back to Top