News, Views and Careers for All of Higher Education
Nov. 13, 2006
The 2006 National Survey of Student Engagement, released today, for the first time offers a close look at distance education, offering provocative new data suggesting that e-learners report higher levels of engagement, satisfaction and academic challenge than their on-campus peers.
Beyond the numbers, however, what institutions choose to do with the data promises to attract extra attention to this year’s report.
NSSE is one of the few standardized measures of academic effectiveness that most officials across a wide range of higher education institutions agree offers something of value.Yet NSSE does not release institution-specific data, leaving it to colleges to choose whether to publicize their numbers.
Colleges are under mounting pressure, however, to show in concrete, measurable ways that they are successfully educating students, fueled in part by the recent release of the report from the Secretary of Education’s Commission on the Future of Higher Education, which emphasizes the need for the development of comparable measures of student learning. In the commission’s report and in college-led efforts to heed the commission’s call, NSSE has been embraced as one way to do that. In this climate, will a greater number of colleges embrace transparency and release their results?
Anywhere between one-quarter and one-third of the institutions participating in NSSE choose to release some data, said George Kuh, NSSE’s director and a professor of higher education at Indiana University at Bloomington. But that number includes not only those institutions that release all of the data, but also those that pick and choose the statistics they’d like to share.
In the “Looking Ahead” section that concluded the 2006 report, the authors note that NSSE can “contribute to the higher education improvement and accountability agenda,” teaming with institutions to experiment with appropriate ways to publicize their NSSE data and developing common templates for colleges to use. The report cautions that the data released for accountability purposes should be accompanied by other indicators of student success, including persistence and graduation rates, degree/certificate completion rates and measurements of post-college endeavors.
“Has this become a kind of a watershed moment when everybody’s reporting? No. But I think what will happen as a result of the Commission on the Future of Higher Ed, Secretary (Margaret) Spelling’s workgroup, is that there is now more interest in figuring out how to do this,” Kuh said.
Charles Miller, chairman of the Spellings commission, said he understands that NSSE’s pledge not to release institutional data has encouraged colleges to participate — helping the survey, first introduced in 1999, get off the ground and gain wide acceptance. But Miller said he thinks that at this point, any college that chooses to participate in NSSE should make its data public.
“Ultimately, the duty of the colleges that take public funds is to make that kind of data public. It’s not a secret that the people in the academy ought to have. What’s the purpose of it if it’s just for the academy? What about the people who want to get the most for their money?”
Participating public colleges are already obliged to provide the data upon request, but Miller said private institutions, which also rely heavily on public financial aid funds, should share that obligation.
Kuh said that some colleges’ reluctance to publicize the data stems from a number of factors, the primary reason being that they are not satisfied with the results and feel they might reflect poorly on the institution.
In addition, some college officials fear that the information, if publicized, may be misused, even conflated to create a rankings system. Furthermore, sharing the data would represent a shift in the cultural paradigm at some institutions used to keeping sensitive data to themselves, Kuh said.
“The great thing about NSSE and other measures like it is that it comes so close to the core of what colleges and universities are about — teaching and learning. This is some of the most sensitive information that we have about colleges and universities,” Kuh said.
But Miller said the fact that the data get right to the heart of the matter is precisely why it should be publicized. “It measures what students get while they’re at school, right? If it does that, what’s the fear of publishing it?” Miller asked. “If someone would say, ‘It’s too hard to interpret,’ then that’s an insult to the public.” And if colleges are afraid of what their numbers would suggest, they shouldn’t participate in NSSE at all, Miller said.
However, Douglas Bennett, president of Earlham College in Indiana and chair of NSSE’s National Advisory Board, affirmed NSSE’s commitment to opening survey participation to all institutions without imposing any pressure that they should make their institutional results public. “As chair of the NSSE board, we believe strongly that institutions own their own data and what they do with it is up to them. There are a variety of considerations institutions are going to take into account as to whether or not they share their NSSE data,” Bennett said.
However, as president of Earlham, which releases all of its NSSE data and even releases its accreditation reports, Bennett said he thinks colleges, even private institutions, have a professional and moral obligation to demonstrate their effectiveness in response to accountability demands — through NSSE or another means a college might deem appropriate.
This Year’s Survey
The 2006 NSSE survey, which is based on data from 260,000 randomly-selected first-year and senior students at 523 four-year institutions(NSSE’s companion survey, the Community College Survey of Student Engagement, focuses on two-year colleges) looks much more deeply than previous iterations of the survey did into the performance of online students.
Distance learning students outperform or perform on par with on-campus students on measures including level of academic challenge; student-faculty interaction; enriching educational experiences; and higher-order, integrative and reflective learning; and gains in practical competence, personal and social development, and general education. They demonstrate lower levels of engagement when it comes to active and collaborative learning.
Karen Miller, a professor of education at the University of Louisville who studies online learning, said the results showing higher or equal levels of engagement among distance learning students make sense: “If you imagine yourself as an undergraduate in a fairly large class, you can sit in that class and feign engagement. You can nod and make eye contact; your mind can be a million miles away. But when you’re online, you’ve got to respond, you’ve got to key in your comments on the discussion board, you’ve got to take part in the group activities.
Plus, Miller added, typing is a more complex psycho-motor skill than speaking, requiring extra reflection. “You see what you have said, right in front of your eyes, and if you realize it’s kind of half-baked you can go back and correct it before you post it.”
Also, said Kuh, most of the distance learners surveyed were over the age of 25. “Seventy percent of them are adult learners. These folks are more focused; they’re better able to manage their time and so forth,” said Kuh, who added that many of the concerns surrounding distance education focus on traditional-aged students who may not have mastered their time management skills.
Among other results from the 2006 NSSE survey:
Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.
Advertisement
It’s hard to get excited about so-called engagement measures. Why are subjectively constructed remarks by students relevant to the debate on education quality? Why are we asking students what they think they are getting? Why does this account for anything?
Glen, at 8:30 am EST on November 13, 2006
“And if colleges are afraid of what their numbers would suggest, they shouldn’t participate in NSSE at all, Miller said.”
The reason to participate in these kinds of surveys is to identify weaknesses and targets of opportunity. Why assess if you’re only interested in “good” results?
mythbuster, at 8:35 am EST on November 13, 2006
It’s true, Glen, that NSSE doesn’t measure real learning, in the form of performance on learning outcomes; it’s equally true that it’s a survey, and the data is self-reported.
At the same time, it might be appropriate to ask if students’ perceptions about what their institutions think valuable are important—because students will do what they think matters for their success in their institution. A second appropriate question might well be to ask, are the students’ reports about the quality or kind of work asked of them _accurate_? Because if they are, then we can have a sense of the level of expectation institutions ask of their students or the level of performance they’re willing to tolerate. I think those are important issues.
Rich, Dean at Bethel University, at 9:20 am EST on November 13, 2006
Glen asks: “Why are subjectively constructed remarks by students relevant to the debate on education quality? Why are we asking students what they think they are getting? Why does this account for anything?”
In learning, a lot depends on the motivation and attitude of the student. Unlike in physics, where something will move a certain speed based on, say, the laws of gravity, in learning, something will move at a different speed depending on how fast it *wants* to move.
When we measure for student attitudes, we are measuring for this motivation and attitude. This doesn’t tell the whole story about the quality of learning, of course. But it helps us understand *how* an institution is more or less successful.
Some people, for example, may argue that it doesn’t matter how motivated students are, that they should still be successful no matter how they feel. A study like this can show that it *does* matter, that there is a measurable impact on student success (by contrast, it could also show that it doesn’t matter, that there is no measurable impact).
Stephen Downes, at 9:20 am EST on November 13, 2006
Glen questions the relevance of self-report data. George Kuh, the director of NSSE, has written about this: http://nsse.iub.edu/pdf/conceptual_framework_2003.pdf
staffjob, at 9:33 am EST on November 13, 2006
A student self-reports mastery of the material in a course, but two 50-point multiple choice exams (or three or four) show the student doesn’t understand about 30 percent of the course material. A student self-reports little understanding of the course material, but two 50-point multiple choice exams show the student understands 90 percent of it. Which method would you say is a better gauge of that student’s educational accomplishment?
justasking, at 5:25 pm EST on November 13, 2006
Glen asks where’s the beef? Well, I think the reason these self-reported sense of engagement measures are deemed important because clearly the customer is always in the right with a much better ability to know when they receive a quality education than those who have devoted their professional lives to education.
Further, while I don’t doubt that motivation is an important factor, the question is, who is responsible for the motivation? When I was a student, it was all on me. Now that I’m the teacher, it still seems to be all on me, or so some would have it. But I’m not buying. If the student can’t motivate themselves, they need to find something that does motivate them. I’m a teacher, not a miracle worker, damnit! (said as Bones would say to Kirk).
bradley bleck, instructor at Spokane Falls CC, at 4:25 am EST on November 14, 2006
It is amazing after 15 years still people are talking about simply ” online learning is good or bad ” I became the true believer of ONLINE Learning 15 years ago. We are the first in the World applying ONLINE Learning with CDs in the most remote part of Turkey with only 10 computers. But opened up 100 rooms with 10computers in 4 years.Now they are connected to internet. Technology is everything. In every subject. It is the greatest opportunity for the mankind to solve its education problem for the whole world. Sure online learning is 10 times better than oncampus learning. Teacher has limited resources in the class room and very expensive. Some say ” there is face to face communication between teacher and student, it cannot be replaced by computers ” Please tell me “how many of you talked to their teachers and how many times in your 4 years of undergraduate study or in elementary or secondary school for that matter.” ONLINE opens up whole world to you. You can reach 1 billion books of the world with a click. You can simulate rain, snow, atom bombs, car accidents, surgery etc etc better than real life. YOU LEARN BY LIVING IT ON THE SCREEN. You get the best teachers in the world for online learning. You can pay him/her 10 times more than regular teachers. He/she reaches now millions of students. Students can comprehend a subject ONLINE on the screen and interactivity simulation and videos 10 times better than a simple explanation of a teacher whom we can not claim he/she is the best of the world. There is resistance to change always. In free societies like USA it is more so. Everybody has a right to talk. If ONLINE was bad, there would not be 3,000,000 students in the USA universities increasing every year by 20-25 %.
But one important thing is ” PROGRAMMING OF AN ONLINE COURSE EXTREMELY IMPORTANT. WE SHOULD NOT SAVE MONEY TO MAKE A BAD ONLINE COURSE. LET ME GIVE YOU A BENCHMARK. TO DESIGN A COURSE “"EQUVELANT TO 50 MINUTES ONCAMPUS COURSE IN CLASSROOM"” COSTS $ 20,000. Unfortunately some vendors try to sell courses at $ 50,000 for 20 hours course, then we say it is bad. Sure it is bad. We have to spend our times for ” how we can make better online courses ” rather than ” if online is better or worse than oncampus ” Best regards.Muvaffak GOZAYDIN mgozaydin@hotmail.com
muvaffak gozaydin, President at AYDIN Education Technologies Consultants, at 6:00 am EST on November 14, 2006
RE: VALIDITY by justaking ————————————— Student #1 self-reports MASTERY but two 50-point multiple-choice exams => 70% The first example is the student understAnds (learns) what the student understOOd to be the important relative to the payoff. (For many students, a “C” is okay and defines what the student considers important (worth mastering). Not surprisingly, it is not necessarily what the instructor thought was important when designing the test. This “student/customer” is satisfied. Student #2 self-reports LITTLE UNDERSTANDING but two 50-point multiple-choice exams => 90% The second example illustrates the student did not understAnd (learn) but is a good test-taker. Te student echoed back the lesson (using short-term retention techniques) without any meaningful understanding required for making a permanent change in their state of knowledge (also called “deep learning").This student is unsatisfied and probably extremely frustrated.
I am not a fan of exams for the reasons your examples illustrate; exam often measure the ability to take tests not actual learning (permanent change in knowledge). Therefore, I would pick the self-report mastery especially since I cannot assume the student wants or cares about an “A” (supposedly 100% mastery).
If I wanted to evaluate whether self-reported mastery correlates with instructor’s definition of mastery, I would not use exams.
I would use the “defacto,” research-informed triad of questions suggested for instructor self-evaluation of a particular lesson or concept. This 5-minute technique at the end of class provides data about the gap between what the instructor intended to teach and what the student mastery relative to what they expected learn. 1. What did you EXPECT to learn 2. What did you NOT learn3. What DID you learn.
I would say the NSSE survey is measuring student mastery relative to the student understanding (expectation) of what the student was suppose to learn at their particular institution. It is a valid measure.
RymH, at 11:15 pm EST on November 14, 2006
Having both taught DL and taken DL courses myself, what’s missing from the article and all comments above is grading is invariably ‘light’ for these courses because the instructor, at a distance, simply has no idea whether the student is writing his or her own work, and there is simply no way to find out, unless you know the student’s previous in-class track record and writing skills. It is just an excuse, imo, to ‘pass’ many who would not pass if they were actually in class. The only case in which I have noticed a difference is advanced business courses where the students are actually commenting on ‘companies’ they are engaged in, rather than the course itself.
mbkirova, at 12:15 pm EST on November 20, 2006
I believe those of us who are serious about the potential of postsecondary online learning realize what a dreadful mistake it would be to grade “light” simply because we could not see our students face to face. There is no logic there.
In fact, groups such as SACS make it very clear that assessment for the online sections of courses must mirror as closely as possible the face to face sections.
Interestingly enough, I have found myself using digital submissions in my face to face courses more often because, after working with digital submissions from my distant students, I have a good system for managing student work. But the rigor is certainly there.
Karen Hughes Miller, Instructor at University of Louisville, at 11:00 am EST on November 22, 2006
I work for a small private college that puts many of its resources into retention and outcome-based activities. We have both campus-based students and online students. The curriculum is the same for both populations. There are assessments that each student is required to take at certain points throughout the degree. The online students tend to score higher in all areas that we measure. I’m sorry I don’t have the data to report here. Online education is a great tool and is a valid delivery to teach and learn. The difference, in my opinion, in relation to the better scores for online students is the population we attract for online education. Our online students are adult students that have some level of work experience (and greater drive as a whole).
Scott, at 10:20 am EST on December 17, 2006
I agreed with Dr. Downey that the 2006 NSSE conclusion that “Distance learning students outperform or perform on par with on-campus students” is misleading. The NSSE conclusion was based on uncontrolled and subjective answers given by students. There were more authorative studies of this subject before, and their conclusions, as far as I know, were inconclusive.
I taught economics online and on-campus at Consumer Community College in Sacramento in the last five years. I often ask my students about their preferences and their perception of the effectiveness of both. Most of my students prefered online classes even though they thought on-campus classes tend to be more effective. Sorry for my observations are not presented in a more statistically convincing manner.
Distance learning is an important and effective educational tool. However, we need to be aware of its shortcomings such higher dropout rates and possibly unsuitable for first and second-year students. (see Hoang Nguyen, Recent U.S. Experience with Distance Learning in Higher Education, Paper presented at VACETS 2005, San Jose)
Hoang Nguyen, Adjunct Professor of Economics at CRC, at 9:16 pm EDT on March 23, 2007
Advertisement
or search for jobs directly.
Posting Description: University of Colorado Denver Web Developer Academic Technology and Extended Learning ... see job
Southeastern Louisiana University invites applicants for the position of Grants Coordinator/Business Analyst for the Office ... see job
HACC, a leader in education in Central PA, is a comprehensive, multi-campus community college, providing quality instruction ... see job
Eastern Kentucky University, located in Richmond, Madison County, Kentucky near the Heart of the Bluegrass, is a ... see job
Access Services Librarian Information Services – Connecticut College POSITION Information Services at Connecticut College, ... see job
Located thirty miles from Chicago, Wheaton College is an Evangelical Protestant Christian liberal arts college whose emphasis ... see job
Optimized Job Title: Director of the Institute for Survey Research (ISR) Organization Profile: The Institute for Survey ... see job
Eastern Kentucky University, located in Richmond, Madison County, Kentucky near the Heart of the Bluegrass, is a ... see job
Come join our Child Development Center! see job
Eastern Kentucky University, located in Richmond, Madison County, Kentucky near the Heart of the Bluegrass, is a ... see job
Survey results versus outcome data
These results are interesting but I would add a caution that the following paragraph in the article is misleading:
Distance learning students outperform or perform on par with on-campus students on measures including level of academic challenge; student-faculty interaction; enriching educational experiences; and higher-order, integrative and reflective learning; and gains in practical competence, personal and social development, and general education. They demonstrate lower levels of engagement when it comes to active and collaborative learning.
It is important to point out that the NSSE, which is the source of this data, is a student self-report survey. So I believe it is not accurate to conclude that actual performance is superior as the paragraph above implies. Rather, it is most accurate to state that distance learning student report greater engagement than their counterparts enrolled in more traditional classroom instructional formats.
Dr. John Downey, Vice President of Instruction and Student Services at Blue Ridge Community College, at 8:15 am EST on November 13, 2006