Assessing Pandemic Learning

Almost all learning experiences, whether inside or outside the classroom, were negatively impacted by the COVID-19 pandemic, according to a new survey from ExamSoft in partnership with the National Institute for Learning Outcomes Assessment.

The one constant during the pandemic was that “change was the norm,” the survey found. Titled “Pandemic Insights to Shape a Better Future,” it analyzed responses from almost 800 individuals representing a variety of roles in higher education—students, administrators, faculty and staff—and a range of institutions. Its aim was not only to provide a high-level analysis of the pandemic learning environment, but also to inform the future of teaching and assessment, said Natasha Jankowski, co-author of the survey and a higher education and assessment consultant.

“I really like to just look forward—like, sure, we did a lot of stuff, there were a lot of changes and that makes really great info fact sheets,” Jankowski said. “But what are we going to do? We’ve had conversations for years and so how are we going to learn from the pandemic and really try to move forward?”

Most changes took place at the classroom level and typically involved a redesign of classroom assignments and assessments, greater flexibility in work submission deadlines, and increased use of proctoring software.

Almost half of the respondents said oral communication was the learning outcome most negatively impacted by the pandemic, followed by teamwork and civic engagement, which were named by more than 40 percent. Jankowski said the lack of oral communication made sense, given that many professors shifted to more written assignments, and teamwork understandably suffered because Zoom made collaboration difficult.

But Jankowski was shocked to see that civic engagement took such a hit during the pandemic. Because so many students participated in national and political movements during that time, which included the 2020 election and the Black Lives Matter protests, she expected it to lean more positively. She speculated that perhaps civic engagement declined because institutions offered fewer volunteer opportunities.

“Civic engagement, for me, was fascinating, because I would have thought that went through the roof,” Jankowski said. “If you think about what was going on nationally, and students being like, ‘We’re gonna go and be involved and get out protest and participate in just social unrest’ … I was not expecting that to be negative.”

In terms of learning experiences, the majority of survey respondents—nearly 60 percent—reported that labs were most negatively impacted, which is not surprising given the difficulty of conducting them remotely. Group work and class discussions also ranked poorly, with almost 40 percent of respondents citing them as most negatively impacted.

Jankowski said it will be interesting to see whether moving labs online during the pandemic will ultimately cause students to transfer out of STEM majors, or if it will slow them down in completing their degrees.

“We did see, particularly within community colleges that had clinicals or some kind of career and technical education, where students had to come in and work in a lab and it was shut down,” Jankowski said. “That’s a huge negative impact. Those students were just gone—they just disappeared—and there’s not a way to get them into those areas.”

Kate Drezek McConnell, vice president for curricular and pedagogical innovation at the American Association of Colleges & Universities, said instruction during the pandemic was all about survival. She said after the spring 2020 semester, many instructors leaned into reconfiguring their courses before they really knew what remote learning could look like.

“Even as I say this, however, I think it is critical to recognize the great variability of experiences across institutions, states and regions,” McConnell said. “So perhaps the biggest change that was shared across institutions was that instructional choices and modalities were no longer just the purview of faculty or even the institutions necessarily, as both public health and political considerations influenced teaching in unprecedented ways.”

Some survey respondents found that the pandemic positively impacted their learning experiences and outcomes. Nearly 40 percent viewed assigned readings as a positive pandemic learning experience, followed by undergraduate research. As for learning outcomes, almost 30 percent of respondents said their involvement in social justice was positively impacted.

However, Jankowski urged caution in reading too much into the positive outcomes, noting that everything about the survey was relative and the positives weren’t resoundingly so.

“I would just add the caveat that we in this survey wanted to say, ‘Here’s negative, positive and not impacted,’” Jankowski said. “That positive is very low.”

Survey respondents were also asked to rank, from most to least effective, the sources of evidence they believe most accurately captured student learning throughout the pandemic. Presentations and video recording ranked at the top, followed by portfolios and capstones. Those considered the least effective in evaluating student learning were games, classroom participation and standardized tests. Jankowski said the findings show respondents understood that exams might not be the best way to assess students, given the stresses of online proctoring.

“What’s really interesting is the awareness of these results during this time, that a standardized test is not a good indicator of learning,” Jankowski said. “There are other sources that are going to be more trusted.”

McConnell said that virtual learning showed students the limitations of traditional testing and lectures. It made them more aware of other forms of assessment, such as portfolios and capstones, and they attributed greater learning gains to those approaches. She said students want intentionality and transparency in the design of assignments and support materials.

“What’s interesting is that the pandemic has helped reveal that the methods many of us have been promoting as more powerful pedagogical approaches since before spring 2020 are indeed what seemed to ensure some continuity of learning despite all the challenges we faced because of COVID-19,” McConnell said. “The weak links in teaching and learning we hear about from faculty and students alike are methods that privilege rote knowledge and comprehension or canned responses.”

According to the survey, 92 percent of respondents agreed that faculty should be given more time to exchange ideas, collaborate, discuss student needs and develop curricula. Ninety-two percent of respondents also agreed that collaborative partnerships among various offices and academic programs—including student services and DEI offices—should be nurtured to help reimagine teaching and learning.

McConnell said successful transitions in learning during the pandemic were due to faculty focusing on their students’ needs in redesigning courses. For the first time, many faculty interacted meaningfully with the other educators on campus who could help them connect with assessment resources, she said.

Ninety-six percent of respondents agreed that faculty should be clear and transparent in regularly communicating their teaching approach and curricular design to students, and 92 percent said that student experiences should inform teaching strategies.

Jankowski said institutions need to keep an ear out and listen to students about which learning practices are working and which aren’t.

“To assume that we know what solutions we need moving forward to best meet their needs without students, I think would be a huge misstep and something that I think we know in principle,” Jankowski said. “But in thinking about what that actually materializes into, or how that becomes a dynamic in our course, that’s something that faculty are still working out.”

McConnell said, looking forward, she’s interested in seeing which learning solutions that emerged during the pandemic persist and which fade into the background.

“I am hopeful that the connections and support structures for teaching, learning and assessment not only continue to exist but are now seen as true collaborators in this space,” McConnell said. “It’s about empowering faculty with the tools and techniques to achieve excellence in teaching, learning and assessment.”

(This story has been updated to correct the name of the company that sponsored the survey; it's ExamSoft.)