Online education has become a hot topic recently, with more and more institutions wanting to expand offerings. And that makes studies of the quality of online education important -- and controversial.

A new paper by the Community College Research Center re-examined and challenged the studies that the Department of Education used in a meta-analysis that stated “on average, students in online learning conditions performed better than those receiving face-to-face instruction” – a conclusion that received much attention and applause among advocates for online education.

But the Education Department's analysis was flawed, according to Shanna Smith Jaggars, lead author of the CCRC paper and a senior research associate at the Columbia University center. Based on its review of the data, the CCRC concluded that online learning in higher education is no more effective than face-to-face learning.

The Education Department study "continues to be cited frequently by people as evidence that online learning can be superior,” Jaggars said. “We just wanted to make sure that we were injecting a note of caution into how people were interpreting what they were seeing.”

The CCRC report states that the Department of Education analysis’ flaws lie in the student populations studied and the conditions of the online courses. The meta-analysis examined the results of 51 published studies that looked at the effects of online versus face-to-face education. Of these, however, only 28 compared face-to-face courses with fully online courses. (The third option, hybrid courses, had most of the students experiencing as much face-to-face class time as they would in a normal course.)

The paper also states that most of these 28 courses studied contained conditions unrepresentative of typical college classes. Most of them looked at short educational programs (as short as 15 minutes) instead of semester-long courses, and some examined online learning in elementary schools or with postgraduate professionals. As a result, Jaggars narrowed the pool to only seven studies that accurately reflected fully-online learning in a college or university setting.

According to the report, these seven online courses “showed no strong advantage or disadvantage in terms of learning outcomes among the samples of students under study.” (Of course that finding may also cheer advocates for distance education, who still face skeptics who insist that the newer form of instruction can't be as good the traditional model.)

Robert Murphy, senior research social scientist at the Center for Technology in Learning and one of the authors of the Department of Education's report, said that Jaggars isn't wrong in her assessment, but that she was looking at very different things. "I'm assuming the center has a great concern for students with poorer academic preparation prior to entering the community college," Murphy said. "But they're taking our study apart for their own purposes: What does our study say about those courses and that group of kids? And they're saying the study doesn't say anything about them, and I would agree with that."

Both Murphy and Jaggars acknowledged the fact that the Department of Education’s conclusions have found support. “Proponents of online learning at a university setting have probably hoped they could erode the resistance at their universities [with this evidence],” Jaggars said. She expressed particular concern about the impact of such thinking at community colleges or institutions with a large number of low-income students.

Very little data in the studies considered by the Department of Education looked at online learning by low-income or academically underprepared students, she said. The CCRC report outlines that full-time employment, transportation costs, or child care may prohibit low-income students from attending face-to-face classes, but that there is no evidence that online learning helps them. In fact, Jaggars suggested that online courses may prove more difficult for such students to complete because they are less likely to have high-speed Internet at home. Other studies have also found that academically underprepared students are more likely to withdraw from an online course than a face-to-face course, and that overall withdrawal rates for online courses are two to three times higher than for face-to-face courses.

“[Online learning] is being seen as a general solution and a lot of people are saying that it should and could help that population” of low-income students, Jaggars said. “It has a lot of potential in helping students, but we need to be aware that it’s not a panacea, and it’s not going to automatically fix things if we just put these courses online.”

The CCRC is currently finishing its own study -- to be released later this summer -- that looks at online learning specifically at community colleges. “Withdrawal rates in that study are a lot higher” than in the studies the Education Department examined, Jaggars said, adding that her response to the Department of Education’s study was a way to lay the groundwork for the results of her upcoming study, “so it’s not so jarring.”

But comparing online to face-to-face learning may not be the best approach to assessing online education, says John Bourne, executive director of the Sloan Consortium, a group of colleges and other organizations that work on online education issues.

“I am exceptionally dubious of studies that tend to compare online education and on-the-ground education without even an attempt to understand the differences in the mechanisms of teaching,” he said, adding that he thinks both reports are flawed, albeit interesting. “The jury is absolutely still out on this, and I don’t believe for a minute that it’s about the delivery mechanism, but what the affordances are of the delivery.”