Making Engagement Data Meaningful
When the annual figures for the National Survey of Student Engagement come out each year, some colleges issue a press release and move on. Gallaudet University in Washington, however, is making this data central in an effort to revise its undergraduate curriculum and institutional mission.
The annual report -- which provides comparative data on student experiences at four-year institutions nationwide -- contains responses to questions both specific and broad concerning a wide range of topics from academic challenge to student-faculty interaction. While only national aggregate information is provided to the public, individual institutions are given the specifics of their students’ responses.
Gallaudet is paying particular attention to its NSSE results this year, following some significant institutional changes. In the fall of 2007, the university’s Board of Trustees approved a new mission statement, officially declaring it a bilingual institution putting equal focus on instruction in both English and American Sign Language. The general studies curriculum for incoming students was reduced from 60 to 40 credits, and now requires that students take a number of interdisciplinary courses and complete a service project in the community.
These changes came following a period of turmoil during which the university was briefly placed on probation by its accreditor, and high-profile protests about the appointment of a new president brought the campus to a standstill. Now that the university has begun carrying out a new strategic plan to boost its academic credentials, student retention, and graduation rates, its NSSE results are being touted as one of the chief assessment tools it will use to ensure its future success.
“Because we’re taking big risks and making dramatic decisions, we’re just starting to look at NSSE data this year to monitor them,” said Pat Hulsebosch, executive director of academic quality, noting that the institution had participated in the survey twice before but its results generated little interest. “I think the goal is not to prove but to improve our goals: a bilingual mission, an integrative general studies curriculum and a mutual understanding of diverse perspectives.”
NSSE grades institutions on five benchmarks of success: level of academic challenge, active and collaborative learning, student-faculty interaction, enriching education experiences and student campus environment. Since Gallaudet last participated in the survey in 2006, both cohorts surveyed by NSSE -- first-year students and seniors -- have shown improvement in nearly all of these areas.
Between 2006 and 2008, first-year students have only reported a decline in enriching educational experiences. Seniors reported a decline in student-faculty interaction and supportive campus environment -- perhaps a lingering response to the 2006 campus protests. As anticipated, seniors reported higher levels of engagement than first-year students in all five benchmarks.
Though few conclusions can be drawn from one year of data following any institutional change, Hulsebosch said she is optimistic about some of these benchmark improvements. She said she hopes an institution-wide review of this data can lead to changes in and out of the classroom. Beginning in January, the institution will host a number of events to discuss its NSSE results with faculty to promote engaging practices they can learn from the data to increase student retention.
Of some concern to Hulsebosch is the low level of engagement of first-year students in what the survey labels "enriching educational experiences" -- some of these might include participating in community outreach projects or other academic pursuits outside of the classroom. First-year students had a benchmark score of 24.1 percent, compared to the score of 50.4 percent reported by seniors. Hulsebosch said she would like to see this low score discussed among faculty members, noting that decisions would have to be made as to how many outside-the-classroom experiences would be appropriate for first-year students.
Unsure of faculty response, some administrators are finding ways to present the data and make tactical suggestions for instructional changes in a way that does not attack their colleagues’ classroom autonomy.
“You can’t just take numbers and put them in front of faculty and tell them we need to improve them,” said Catherine Andersen, associate provost for enrollment. “That won’t work. You have to look at NSSE numbers in conjunction with what you are doing as best practice to find what is and isn’t working. I’m always drilling down to what we can make of these numbers. I think the NSSE benchmarks provide conversations about hat might be going on. But, the proof is in the pudding with increased retention and graduation rates.”
Andersen said the institution's score on "student faculty interaction" suggested some improvement could be made. Though Gallaudet scored either slightly better than or relatively close to the national average on individual points such as "discussed grades or assignments with an instructor" and "received prompt written or oral feedback from faculty on your academic performance," she said ongoing dialogue with a professor was important for improved student performance.
"If you don't let students know where they are, how are they going to know where they're going?" Andersen asked.
As such, the university is asking for “action plans” from each of its academic departments to ensure that they have some commitment to what Andersen calls “strategic indicators” known to improve student engagement. For example, she said math faculty members, concerned about remedial courses with high rates of failure, have committed to meeting outside of the classroom with any student who has a D or F at midterm, in order to develop a plan to improve their grade. Others have committed to barring students who have failed their most recent test or quiz from taking another until they have either met with a student tutor or an instructor.
The idea of a commitment to such methods, however, does not excite all faculty members.
“There are people who will say this is just one more thing we have to do and don’t understand the institutional commitment to it,” said Leslie Rach, general studies assessment coordinator and English professor, noting that the implementation plan for some of these suggestions are meant to generate discussions about best practice. “These commitments are not going to be a grassroots faculty effort, as their compliance is ultimately totally optional.”
Rach said those faculty members concerned about taking suggestions based on the data at least appreciate the survey’s methodology and statistical analysis. NSSE data will play a significant role in institutional assessment, she noted, because the stated outcomes of the new general studies curriculum now match up with the survey’s fixed benchmarks. The curriculum's focus on inter- and multidisciplinary courses, she said, make looking at the survey's "level of academic challenge" benchmark ideal. For example, NSSE measures this benchmark by asking to what extent students report they are "analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components." She said this data will help the institution determine the effectiveness of its curricular changes.
“We can now approach the data with a set of questions about our [expected] outcomes,” Rach said. “You typically don’t approach data and then formulate questions. It’s usually the other way around.”
Despite the suggestion of the data that increased interaction with students outside the classroom improves their performance, Hulsebosch said even some of these common sense measure cannot truly be required by the institution. On the other hand, Andersen said “clear pathways to success” have to be opened up by faculty, especially now that Gallaudet is serving many more first-generation and other potentially at-risk students.
“Still, it is important that we don’t tell professors what to do,” Andersen said, adding that she does not think suggestions from NSSE data should lead to the homogenization of certain classroom practices. “What you want to happen is to have faculty feel good about what they do to help students every day. That’s best practice.”