Assessment

N.C. community college to issue grades, certificates for soft skills

Smart Title: 

Community college in N.C. plans to evaluate "soft skills" and issue workplace readiness certificates for students who get passing marks.

Website recognizes military skills with digital badges

Smart Title: 

New website creates digital badges for veterans, aiming to recognize military skills and training. Will it be the first badging experiment to catch on?

Carnegie Foundation considers a redesign for the credit hour

Smart Title: 

The Carnegie Foundation, which created the credit hour, considers a redesign so the standard could better fit with emerging approaches to higher education.

Saylor Foundation's Free Courses Offer Path to Credit

Smart Title: 

Saylor Foundation's 240 free online courses now offer a pathway to college credit, thanks to new partnerships with Excelsior College and StraighterLine. But will students follow that path?

Prior learning assessment catches on, quietly

Smart Title: 

Prior learning assessment could be higher education's next big disruptive force, and ACE and CAEL are poised to catch that potential gold rush. But many remain skeptical about academic credit for work experience.

Improving Graduation Rates Is Job One at City Colleges of Chicago

Smart Title: 

City Colleges of Chicago have a 7 percent graduation rate. If that number doesn't go up, the system's chancellor, presidents and trustees could lose their jobs.

Nontenured faculty should not be assessed by student evaluations in this politically charged era (essay)

Now that more that 75 percent of the instructors teaching in higher education in the United States do not have tenure, it is important to think about how the current political climate might affect those vulnerable teachers. Although we should pay attention to how all faculty are being threatened, nontenured faculty are in an especially vulnerable position because they often lack any type of academic freedom or shared governance rights. In other words, they are a class without representation, and they usually can be let go at any time for any reason. That type of precarious employment, which is spreading all over the world to all types of occupations, creates a high level of professional insecurity and helps to feed the power of the growing managerial class.

In the case of higher education, we need to recognize that this new faculty majority often relies on getting high student evaluations in order to keep their jobs or earn pay increases. The emphasis on pleasing students not only can result in grade inflation and defensive teaching, but it also places the teacher in an impossible situation when dealing with political issues in a polarized environment. While some students want teachers to talk about political issues, many students will turn against an instructor who does not share their own ideological perspective. Sometimes that type of political disagreement is transformed in student evaluations into vague complaints about the teacher’s attitude or personality.

In this fraught cultural environment, practically everyone feels that they are being censored or silenced or ignored. For example, some of my conservative students have told me that they feel like they are the real minorities on campus, and even though Donald Trump won the U.S. presidency, they still think they cannot express their true opinions. On the other side, some of my self-identified progressive activist students believe that political correctness makes it hard to have an open discussion: from their perspective, since anything can be perceived as a microaggression, people tend to silence themselves. Moreover, the themes of political correctness, safe spaces, trigger warnings and free speech have become contentious issues on both the right and the left.

What I am describing is an educational environment where almost everyone is afraid to speak. The nontenured faculty members are fearful of losing their jobs, the conservative students see themselves as a censored minority and the progressive students are afraid of being called out for their privilege or lack of political correctness. Making matters worse is that students are often socialized by their large lecture classes to simply remain passive and silent.

It appears that we are facing a perfect storm where free speech and real debate are no longer possible. One way of countering this culture is to stop relying on student evaluations to assess nontenured faculty. If we want teachers to promote open dialogue in their classes, they should not have to be afraid that they will lose their jobs for promoting the free exchange of ideas. Therefore, we need to rely more on the peer review of instruction, and we have to stop using the easy way out. In short, we have to change how nontenured faculty members are evaluated.

Non-tenure-track faculty should be empowered to observe and review one another’s courses using established review criteria. It is also helpful to have experienced faculty with expertise in pedagogy involved in the peer-review process of teaching. By examining and discussing effective instructional methods, all faculty members can participate in improving the quality of education.

It is also essential that to protect free speech and open academic dialogue, we should realize that the majority of faculty members no longer have academic freedom or the right to vote in their departments and faculty senates. In order to change this undemocratic situation, tenured professors should understand that it is to their advantage to extend academic freedom and shared governance to all faculty members, regardless of their tenure status. If we do not work together to fight back against the current political climate, we will all suffer together.

Robert Samuels teaches writing at the University of California, Santa Barbara, and is the president of UC-AFT. His forthcoming book is The Politics of Writing Studies.

Editorial Tags: 
Image Source: 
iStock/AndSim
Is this diversity newsletter?: 

Three important questions to ask about credentialing (essay)

Few topics in higher education are getting more attention than credential innovation: making credentials digital, introducing new credential types and communicating more information about learning outcomes.

Credential innovation moves transcripts, certificates and diplomas beyond accounting and verification records for transfer and graduate admissions, or mechanisms for validating completion of a university’s degree program. For many institutions, a clear driver of innovating the form and function of their credentials is a belief that today’s transcripts do not communicate what employers care about.

There are two ways to think about this issue. It’s possible that credentials don’t communicate what employers care about because colleges don’t actually provide what the labor market wants. And plenty of people say that. But I (very emphatically) believe that’s generally not true. In fact, credential innovation is so important because colleges do provide much of what employers are looking for. The problem instead is that they just don’t assess, document and communicate those outcomes. So the information is lost, and students are left to their own devices to effectively represent their collegiate experience. And the impact of colleges and universities is left implicit, not explicit.

If higher education institutions do equip students with much of what employers want, and credentials are the mechanism for communicating those outcomes per student to an employer, then the foundational question animating credential innovation projects should be: What do employers want to know? A vice provost of academic affairs recently asked me this question, and it’s stuck with me ever since.

In fact, I heard the same question, frequently, at a recent Lumina Foundation gathering where community colleges, universities and third-party vendors shared how they are experimenting with a comprehensive student record. It’s a national question for higher education as a whole, but even more so a geographically local and industry-specific question that every institution needs to consider distinctly. Let’s face it, there is no such thing as the unitary actor “employer” any more than there is such a thing as the unitary actor “higher education.”

Surprisingly, too many credential innovation projects lack grounding in clear answers to this simple question. Like the proverbial drunk looking for car keys where the light is brightest, they start with a clear conception of what the college or university can say beyond courses and credits, not a clear conception of what their employer market has asked to understand about their graduates. That makes sense insofar as colleges and universities (should) each have a distinct educational mission and program, and it is the outcome of that program or mission that credentials should communicate. But if the goal is better communication and alignment, leaving the question of employer interest unasked makes it much less likely that the underlying goal of credential innovation will be achieved.

In 2014, Hart Research Associates conducted an online survey among 400 employers on behalf of the Association of American Colleges and Universities. The majority of employers said that possessing both field-specific knowledge and a broad range of knowledge and skills is important for recent college graduates to achieve long-term career success. In fact, 80 percent of employers said it would be very or fairly useful to see an electronic portfolio that summarizes and demonstrates a candidate’s accomplishments in key skill and knowledge areas, including effective communication, knowledge in their field, applied skills, evidence-based reasoning and ethical decision making. It is important to note that, along with course work, those are the very types of capabilities that institutions help students build through activities like co-curricular leadership, study abroad and faculty research collaborations.

Today, community colleges and regional four-year schools often work closely with a particular employer or set of employers that define their local economy. The Kentucky Community and Technical College system, the North Carolina Community College system and many other institutions and systems align programs and assess outcomes in a way that’s highly aligned with what those employers are looking for.

What appears less common is a national dialogue between institutions and employers, particularly regarding white-collar jobs -- the kind of jobs where writing well, speaking well, thinking analytically and being comfortable with numbers are important skills to have. (Those are the types of skills that I describe as desired outcomes for my students as an assistant research professor of sociology at Arizona State University.)

Of course to ground credential innovation in an understanding of what employers want, institutions must begin the credential innovation journey by determining which employers matter to them and their students -- in particular those who actually consume their credentials today -- so they can ask how they use credential documents and what they think about them.

Digital credential platforms like Parchment, where I work, enable registrars to see where their credentials are going, which can then be the pathway for opening up this discussion. For example, we observe in Parchment’s data that Ernst & Young collects academic credentials, as does the Department of Homeland Security. Other national employers that receive transcripts from applicants via Parchment include Boeing, Deloitte, Hewlett-Packard, NASA, the U.S. Army, the U.S. State Department and Wells Fargo, to name a few.

Asking what employers want to know inevitably sparks a second question: Given what employers are looking for, does our educational environment develop that in students? While I think it does (and far more than we’re given credit for), the experiences are not necessarily tracked or assessed and can span multiple information systems. They may not have the rigor and integrity that the recording of course work involves, and by choosing to organize course work into a priori majors and minors, clusters of courses that reflect particular skills or ways of thinking can be lost. While program innovation is no doubt needed to meet employers’ needs, introducing a better and more formal assessment of what we’re already doing becomes low-hanging fruit we can harvest without redesigning programs or creating new ones to meet employers’ needs.

This brings me to a third and final question: How do we document and communicate this information? The answer is providing a credential in a certified and succinct way, in an operationally efficient way, and in a way that has reflective/scaffolding value to the learner so they can maximize their time in college to prepare themselves for the labor market.

For example, Elon University has provided a co-curricular, or experiential, credential for many years as part of its educational experience, which includes undergraduate research, global engagement, leadership, service and internship. Although the experiences were not documented originally with employers in mind (they reflect Elon’s distinctive educational philosophy), the university recently began surveying employers who received their new experiential transcript about what was useful and what was not. They found that 75 percent of employers agreed that the experiential transcript provided useful information for the hiring process, while 44 percent agreed that the experiential transcript increased the chances that an applicant would get an interview.

Communicating credential information effectively means supporting the development of data standards, as today’s national employers increasingly rely on applicant tracking systems with algorithms that use a wide variety of data sources to evaluate prospective employees. According to a 2016 report from Gartner, an information technology research and advisory firm, those algorithms will replace both manual processing of CVs (résumés) by recruiters and automated CV ranking based on word matching. The beauty of data is that you can convey a superset of information and package it into different types of credentials for different audiences (transfer transcript, employer college experiences report), and align with the data-processing practices of employers.

If you are a higher education leader beginning the credential innovation conversation, consider these three questions. What do our employers want to know about our learners? What do we need to do programmatically to accurately and reliably communicate those outcomes? And, recognizing the various audiences and purposes that credentials serve, what form of credential can best communicate it?

I encourage you to look at your credentials with fresh eyes as a currency for opportunity and the key to reaffirming a transformative value of a postsecondary education, especially in a knowledge economy. By taking a new approach to academic credentials, higher education can give employers what they are looking for and help students turn those credentials into opportunities.

Matthew Pittinsky is an assistant research professor at Arizona State University and CEO of Parchment Inc., a digital transcript company in K-12 and higher education.

Image Source: 
iStock/scibak
Is this diversity newsletter?: 

How assessment falls significantly short of valid research (essay)

In a rare moment of inattention a couple of years ago, I let myself get talked into becoming the chair of my campus’s Institutional Review Board. Being IRB chair may not be the best way to endear oneself to one’s colleagues, but it does offer an interesting window into how different disciplines conceive of research and the many different ways that scholarly work can be used to produce useful knowledge.

It has also brought home to me how utterly different research and assessment are. I have come to question why anyone with any knowledge of research methods would place any value on the results of typical learning outcomes assessment.

IRB approval is required for any work that involves both research and human subjects. If both conditions are met, the IRB must review it; if only one is present, the IRB can claim no authority. In general, it’s pretty easy to tell when a project involves human subjects, but distinguishing nonresearch from research, as it is defined by the U.S. Department of Health and Human Services, is more complicated. It depends in large part on whether the project will result in generalizable knowledge.

Determining what is research and what is not is interesting from an IRB perspective, but it has also forced me to think more about the differences between research and assessment. Learning outcomes assessment looks superficially like human subjects research, but there are some critical differences. Among other things, assessors routinely ignore practices that are considered essential safeguards for research subjects as well as standard research design principles.

A basic tenet of ethical human subjects research is that the research subjects should consent to participate. That is why obtaining informed consent is a routine part of human subject research. In contrast, students whose courses are being assessed are typically not asked whether they are willing to participate in those assessments. They are simply told that they will be participating. Often there is what an IRB would see as coercion. Whether it’s 20 points of extra credit for doing the posttest or embedding an essay that will be used for assessment in the final exam, assessors go out of their way to compel participation in the study.

Given that assessment involves little physical or psychological risk, the coercion of assessment subjects is not that big of a deal. What is more interesting to me is how assessment plans ignore most of the standard practices of good research. In a typical assessment effort, the assessor first decides what the desired outcomes in his course or program are. Sometimes the next step is to determine what level of knowledge or skill students bring with them when they start the course or program, although that is not always done. The final step is to have some sort of posttest or “artifact” -- assessmentspeak for a student-produced product like a paper rather than, say, a potsherd -- which can be examined (invariably with a rubric) to determine if the course or program outcomes have been met.

On some levels, this looks like research. The pretest gives you a baseline measurement, and then, if students do X percent better on the posttest, you appear to have evidence that they made progress. Even if you don’t establish a baseline, you might still be able to look at a capstone project and say that your students met the declared program-level outcome of being able to write a cogent research paper or design and execute a psychology experiment.

From an IRB perspective, however, this is not research. It does not produce generalizable knowledge, in that the success or, more rarely, failure to meet a particular course or program outcome does not allow us to make inferences about other courses or programs. So what appears to have worked for my students, in my World History course, at my institution, may not provide any guidance about what will work at your institution, with your students, with your approach to teaching.

If assessment does not offer generalizable knowledge, does assessment produce meaningful knowledge about particular courses or programs? I would argue that it does not. Leaving aside arguments about whether the blunt instrument of learning outcomes can capture the complexity of student learning or whether the purpose of an entire degree program can be easily summed up in ways that lend themselves to documentation and measurement, it is hard to see how assessment is giving us meaningful information, even concerning specific courses or programs.

First, the people who devise and administer the assessment have a stake in the outcome. When I assess my own course or program, I have an interest in the outcome of that assessment. If I create the assessment instrument, administer it and assess it, my conscious or even unconscious belief in the awesomeness of my own course or program is certain to influence the results. After all, if my approach did not already seem to be the best possible way of doing things, as a conscientious instructor, I would have changed it long ago.

Even if I were the rare human who is entirely without bias, my assessment results would still be meaningless, because I have no way of knowing what caused any of the changes I have observed. I have never seen a control group used in an assessment plan. We give all the students in the class or program the same course or courses. Then we look at what they can or cannot do at the end and assume that the course work is the cause of any change we have observed. Now, maybe this a valid assumption in a few instances, but if my history students are better writers at the end of the semester than they were at the beginning of the semester, how do I know that my course caused the change?

It could be that they were all in a good composition class at the same time as they took my class, or it could even be the case, especially in a program-level assessment, that they are just older and their brains have matured over the last four years. Without some group that has not been subjected to my course or program to compare them to, there is no compelling reason to assume it’s my course or program that’s causing the changes that are being observed.

If I developed a drug and then tested it myself without a control group, you might be a bit suspicious about my claims that everyone who took it recovered from his head cold after two weeks and thus that my drug is a success. But these are precisely the sorts of claims that we find in assessment.

I suspect that most academics are either consciously aware or at least unconsciously aware of these shortcomings and thus uneasy about the way assessment is done. That no one says anything reflects the sort of empty ritual that assessment is. Faculty members just want to keep the assessment office off their backs, the assessment office wants to keep the accreditors at bay and the accreditors need to appease lawmakers, who in turn want to be able to claim that they are holding higher education accountable.

IRBs are not supposed to critique research design unless it affects the safety of human subjects. However, they are supposed to weigh the balance between the risks posed by the study and the benefits of the research. Above all, you should not waste the time or risk the health of human subjects with research that is so poorly designed that it cannot produce meaningful results.

So, acknowledging that assessment is not research and not governed by IRB rules, it still seems that something silly and wasteful is going on here. Why is it acceptable that we spend more and more time and money -- time and money that have real opportunity costs and could be devoted to our students -- on assessment that is so poorly designed that it does not tell us anything meaningful about our courses or students? Whose interests are really served by this? Not students. Not faculty members.

It’s time to stop this charade. If some people want to do real research on what works in the classroom, more power to them. But making every program and every faculty member engage in nonresearch that yields nothing of value is a colossal, frivolous waste of time and money.

Erik Gilbert is a professor of history at Arkansas State University.

Editorial Tags: 
Image Source: 
iStock/applesimon
Is this diversity newsletter?: 

Developing metrics and models that are vital to student learning and retention (essay)

Is English 101 really just English 101? What about that first lab? Is a B or C in either of those lower-division courses a bellwether of a student’s likelihood to graduate? Until recently, we didn’t think so, but more and more, the data are telling us yes. In fact, insights from our advanced analytics have helped us identify a new segment of at-risk students hiding in plain sight.

It wasn’t until recently that the University of Arizona discovered this problem. As we combed through volumes of academic data and metrics with our partner, Civitas Learning, it became evident that students who seemed poised to graduate were actually leaving at higher rates than we could have foreseen. Why were good students -- students with solid grades in their lower-division foundational courses -- leaving after their first, second or even third year? And what could we do to help them stay and graduate from UA?

There’s a reason it’s hard to identify which students fall into this group; they simply don’t exhibit the traditional warning signs as defined by the retention experts. These students persist into the higher years but never graduate despite the fact that they’re strong students. They persist past their first two years and over 40 percent have GPAs above 3.0 -- so how does one diagnose them as at risk when all metrics indicate that they’re succeeding? Now we’re taking a deeper look at the data from the entire curriculum to find clues about what these students really need and even redefine our notion of what “at risk” really means.

Lower-division foundational courses are a natural starting point for us. These are the courses where basic mastery -- of a skill like writing or the scientific process -- begins, and mastery of these basics increases in necessity over the years. Writing, for instance, becomes more, not less, important over students’ academic careers. A 2015 National Survey of Student Engagement at UA indicated that the number of pages of writing assigned in the academic year to freshmen is 55, compared to 76 pages for seniors. As a freshman or sophomore, falling behind even by a few fractions can hurt you later on.

To wit, when a freshman gets a C in English 101, it doesn’t seem like a big deal -- why would it? She’s not at risk; she still has a 3.0, after all. But this student has unintentionally stepped into an institutional blind spot, because she’s a strong student by all measures. Our data analysis now shows that this student may persist until she hits a wall, usually during her major and upper-division courses, which is oftentimes difficult to overcome.

Let’s fast forward two years, then, when that same freshman is a junior enrolled in demanding upper-level classes. Her problem, a lack of writing command, has compounded into a series of C’s or D’s on research papers. A seemingly strong student is now at risk to persist, and her academic life becomes much less clear. We all thought she was on track to graduate, but now what? From that point, she may change her major, transfer to another institution or even exit college altogether. In the past, we would never have considered wraparound support services for students who earned a C in an intro writing course or a B in an intro lab course, but today we understand that we have to be ready and have to think about a deeper level of academic support across the entire life cycle of an undergrad.

Nationally, institutions like ours have developed many approaches to addressing the classic challenges of student success, developing an infrastructure of broad institutional interventions like centralized tutoring, highly specialized support staff, supplemental classes and more. Likewise, professors and advisers have become more attuned to responding to the one-on-one needs of students who may find themselves in trouble. There’s no doubt that this high/low approach has made an impact and our students have measurably benefited from it. But to assist students caught in the middle, those that by all measurement are already “succeeding,” we have to develop a more comprehensive institutional approach that works at the intersections of curricular innovation and wider student support.

Today, we at UA are adding a new layer to the institutional and one-to-one approaches already in place. In our courses, we are pushing to ensure that mastery matters more than a final grade by developing metrics and models that are vital to student learning. This, we believe, will lead to increases in graduation rates. We are working hand in hand with college faculty members, administrators and curriculum committees, arming those partners with the data necessary to develop revisions and supplementary support for the courses identified as critical to graduation rather than term-over-term persistence. We are modeling new classroom practices through the expansion of student-centered active classrooms and adaptive learning to better meet the diverse needs of our students.

When mastery is what matters most, the customary objections to at-risk student intervention matter less. Grade inflation by the instructor and performance for grade by the student become irrelevant. A foundational course surrounded by the support that a student often finds in lower-division courses is not an additional burden to the student, but an essential experience. Although the approach is added pressure on the faculty and staff, it has to be leavened with the resources that help both the instructor and the students succeed.

This is a true universitywide partnership to help a population of students who have found themselves unintentionally stuck in the middle. We must be data informed, not data driven, in supporting our students, because when our data are mapped with a human touch, we can help students unlock their potential in ways even they couldn’t have imagined.

Angela Baldasare is assistant provost for institutional research. Melissa Vito is senior vice president for student affairs and enrollment management and senior vice provost for academic initiatives and student success. Vincent J. Del Casino Jr. is provost of digital learning and student engagement and associate vice president of student affairs and enrollment management at the University of Arizona.

Image Source: 
iStock/jaker5000
Is this diversity newsletter?: 

Pages

Subscribe to RSS - Assessment
Back to Top