Filling the Void

"Form triumphs over substance."

If Arthur Levine's 92-page report, "Educating Researchers," could be condensed into a sentence, that would be it. The report, released today, is the third in a series written by Levine, president emeritus of Columbia University's Teachers College and now president of the Woodrow Wilson National Fellowship Foundation, where he has continued his inquiries into the state of teacher education.

Now he has turned his focus to the quality of research at education schools, and the methods and practices passed on to aspiring researchers in education doctoral programs -- programs that, Levine told Inside Higher Ed, are more interested in the "form" of handing out doctorates than the "substance" of good research to back them up.

Levine's previous reports have established him as a frequent critic of the state of American education schools, and his latest will do nothing to dispel that image. But if anything, he said, the report will merely confirm what many people in the educational establishment already believe, providing specific facts and figures to support a growing consensus: "I'm documenting conditions that won't surprise many people. The only thing that may surprise them is degree."

The degree of the problem, the report suggests, is severe: At a time when debates over higher education, and education more generally, are dominated by "ideology," Levine writes, "The right, the left and single-interest groups are locked in a white-hot, self-righteous battle over the directions our [K-12] schools need to take. There has been little rigorous research to produce empirical evidence in support of any position." In part because of the dearth of real, peer-reviewed, high-quality research, the report says, interest groups and think tanks, usually writing from a particular ideological perspective, have stepped in.

"Without evidence to support the strength or weakness of any idea, we allow the ideologies to fill the void," Levine said.

If the ultimate goal of education research is to in some way inform the direction of policy, the report implies that the entire discipline has failed. "For policymakers, the volume of education research is so large as to be inaccessible and incomprehensible, yet so eclectic as to leave gaping holes in coverage," it says. "They criticized education research for differing reasons -- impracticality, bias, self-promotion, inattention to implementation issues, gaps in content, inappropriate and ineffective methods of dissemination, low quality and weak methods, lack of replication and absence of rigor. All [policymakers interviewed] desperately wanted education research and used it to varying degrees, but it was not having the impact on their policymaking it could or should have."

The sense of urgency in the report comes in part because of the No Child Left Behind Act's emphasis on "scientifically based research," a term lawmakers used to refer to work based on random controlled trials as a way to highlight what they felt was a "paucity" of relevant research at the time the law was signed in 2001. Levine noted in the report that there was a backlash among educators who criticized the government intrusion on their institutions and felt that the most expensive methodologies -- those requiring the most rigor, and therefore the most resources -- were being mandated by the law.

Yet the most persistent critics of education research were members of education-school faculties themselves. Only 24 percent rated scholarship produced at education schools as "excellent" or "good," and nearly half of ed-school doctoral recipients felt their programs as a whole weren't rigorous enough. Part of the problem seems to be a lack of agreement on what constitutes good research -- even on basic foundational questions like what methodologies scholars should follow.

One particular concern is the blurry line separating the purpose of the Ph.D. in education from the Ed.D., a distinction that has raised questions recently among scholars. "The degrees graduates receive are fungible," the report says; Levine said in an interview that both degrees produce "research lite."

In the report -- which lists Levine as the sole author but counts the former U.S. News & World Report assistant managing editor Alvin Sanoff, who worked on the magazine's annual college rankings, as project manager for the study -- Levine reserves special criticism for two sets of institutions: the education research community, and think tanks that deal with education issues.

"The largest organization in the field, the American Educational Research Association (AERA) ... is not so much a close-knit research community as a research holding company in which differences among members loom larger than commonalities," the report declares. It goes on to mention the low frequency of citations of articles in education research journals as an example of the field's relatively low impact on practitioners and even other researchers.

"I think the characterization of AERA as a holding company is a caricature and not in any sense a meaningful description," Felice J. Levine, AERA's executive director, said in an e-mail. "One of the hallmarks of education research and of AERA as the national education research association is that the whole is much more than the sum of the parts. As with other large interdisciplinary fields and disciplines (e.g., psychology, economics, sociology), there are certainly tighter networks of communication and collaboration within subfield specialties than among them (thus, for example, scholars of teacher education share more in common than do those in postsecondary education)."

Levine, the AERA director and herself a social psychologist, pointed to the report's admission that it "does not compare research or research preparation in education with that in any other field -- so it would be a mistake to conclude that the quality of education research and research preparation is better or worse than in other fields." She also said the the report's view that education research was "amorphous" and "lacks focus" was a misreading of a broader trend in the sciences. "In all fields of science and scholarship these days, there is present more the quest for integration of knowledge across paradigms and methods and not the search for a single unitary theory whether in the physical or natural sciences or the social and behavioral sciences," she said.

The report's author also attacks from the other side, criticizing institutions that seek to fill the void with research and findings of their own. Of this group, the report says, "the worst offenders have been the growing number of ideological think tanks, overwhelmingly conservative." He said in an interview that "they have been very successful in getting their opinions which are masqueraded as research [disseminated]. That’s very unfortunate because the kids don’t benefit from that."

Rick Hess, the director of education policy studies at the American Enterprise Institute (and a possible target of the report), said he agreed that mostly conservative think tanks were filling that void, but not necessarily because of a preexisting ideological slant. "Certainly in the case of schools of education, that there is a relatively homogeneous body of thinking about what are appropriate ways to improve schools, to professionalize teaching, to manage the business of schooling," he said.

"So I think folks who have been open proponents of choice-based reform, of paying effective teachers more, of encouraging entrepreneurship and innovation, have often felt uncomfortable in conventional education schools." He added, "What we want is an interesting heterogeneous mix. I don’t think many education schools have done a very good job of supporting that kind of research environment."

Chester E. Finn, Jr., president of the Thomas B. Fordham Institute, which supports charter schools, and like Hess a former faculty member at an education school, was more blunt. Despite generally supporting Levine's reports, he said, "There is no more ideological place in the universe than the education schools," implying that criticizing think tanks for being ideological is, for that reason, misplaced.

But at the same time, research on education doesn't come out only of schools of education, education departments and think tanks. Researchers in social sciences such as economics and political science produce scholarship on education that is often influential, although Arthur Levine's report does not focus on it. "We ought to have more economists, political scientists, sociologists ... do research in the area. But there are a series of issues that deal with teaching and learning that don’t deal with any of those areas," Levine said.

Despite its implications for the education research field, the report was not entirely negative. It singles out several schools, such as Stanford University and Boston College, as having programs that prepare good researchers and produce good scholarship (Teachers College, where Levine was president, is excluded from the report), and includes a more in-depth case study of the doctoral program in special education at Vanderbilt University. Among the criteria the report uses to gauge excellence are explicit purpose, curricular coherence, admissions standards, good research, faculty composition and self-assessment standards.

The report concludes with several suggestions that would lead to an improved state of affairs for education research: "high and clearly defined standards" of quality, more connections with other disciplines and policy professionals, enhanced quality control, and allowing inferior programs to close.

"Have some rigor, have some integrity ... and more than that, be able to turn out people who have the capacity to make the kinds of contributions to education research that’s desperately needed right now," Levine said.