Peer Review by Whom?

Science advocates are calling a proposal from Senator Rand Paul a blatant attempt to inject politics into federally funded research.

Paul, a Kentucky Republican, is one of the Senate’s biggest critics of what he sees as wasteful spending by the government. His latest target is federal research he believes has little or no payoff for taxpayers -- a situation Paul would address by altering the peer-review process for evaluating grant applications at all federal agencies, including the National Institutes of Health and the National Science Foundation, as well as smaller agencies such as the National Endowment for the Humanities.

He proposes in legislation introduced in October to add two new members to the peer-review panels that judge applications for federal funding -- one expert in a field unrelated to the research proposed in the grant application in question, who could not have worked at or been affiliated with a college or university for 10 years prior to the grant review. The second addition to peer-review panels would be a “taxpayer advocate” who Paul says would consider the likely returns on the research funding for society.

The bill would also move the responsibilities of the National Science Foundation’s Office of Inspector General to an entirely new government entity charged with determining whether a random sample of research proposals at the agency would “deliver value to the taxpayer.” Those that failed that test would have their funding denied.

Attempts by Republican lawmakers to call out -- and even defund -- research that appears to have limited usefulness to the public is nothing new. Some of those measures have focused on the social sciences in particular. But Paul’s attempt to target the peer-review process is novel.

Research advocates don’t necessarily think the bill goes anywhere. Senator James Lankford, the Oklahoma Republican who chairs the full Senate oversight committee, has also criticized the awarding of federal grants in the past, but Paul’s bill has yet to get a single cosponsor. Science advocates are still concerned about the ideas behind it, though, and the misperceptions about the research funding process it represents.

Sean Gallagher, a senior government relations officer at the American Association for the Advancement of Science, said the kind of legislation Paul is proposing would have “pretty chilling consequences” -- not just for researchers but for the role of science in federal policy.

“It’s pretty much as blatant a political interference into the scientific process as it gets,” he said of the proposal in Paul’s legislation.

Experts in a particular field are best able to assess the value of a research proposal in that area, Gallagher said. He added that to the extent that some proposals may not appear relevant to the general public, federal agencies have been undertaking efforts to better communicate with the public about the value of research.

The peer-review system isn’t perfect, but the remedies in the Paul proposal exacerbate existing problems, said Henry Reichman, chair of the American Association of University Professors’ Committee A on Academic Freedom and Tenure.

“His proposal assumes that phony ‘independence’ trumps legitimate expertise, but what is to prevent his taxpayer advocates from serving the bidding of moneyed special interests?” Reichman said.

In a hearing on the legislation last month, Paul cited studies previously identified by Republican colleagues as “silly science” -- among them, NSF grants to study Ugandan gambling habits and “shrimp on a treadmill.” (The treadmill was a small part of a large study looking at how the immune systems of shrimp react to warming oceans and pollution. And the treadmill only cost $50, one investigator involved in the research has said.)

"How does this happen? More accurately, how does it continue to happen?" Paul asked.

He blamed a "publish or perish" mind-set in academe. He also took issue with policies that in some cases give those proposing grants input into who reviews their proposals. Paul said NSF allows grant applicants to recommend potential reviewers and to name individuals who should be excluded as reviewers. And he said NIH allows grant applicants to recommend a potential study section to review grant proposals as well as to name individuals who should be excluded as reviewers.

"So the people getting the money can recommend who approves giving them the money," he said. "That's right. Researchers get to pick the people who approve their funding. Doesn't sound very objective."

That's not quite how the current peer-review process works, according to the NIH and NSF.

NIH uses a two-tiered peer-review process. The first determines scientific merit and feasibility and requires a specific knowledge base in a particular field. The second peer-review tier, known as the advisory committee review, determines the value of a proposal to the mission of NIH. The advisory committee includes both scientific experts and members of the public.

Richard Nakamura, the director of the NIH Center for Scientific Review, noted that NIH does not allow applicants to request specific reviewers. And he said in an emailed statement that the agency also doesn’t entertain requests to exclude reviewers just because an applicant requests it, although they may note an “infamous public dispute” with a potential reviewer.

Applicants may request a review group, but NIH will only allow it if the panel has the required expertise and the extra application doesn’t overload it. Such allowances don’t confer much advantage, given the intense competition for NIH funding, where only the best of the best get fundable scores.

The approval rate for federal research grant applications is exceedingly low -- a reflection of the intense competition for limited funding. At the National Science Foundation, only 24 percent of grant proposals are ultimately funded.

Nearly every grant proposal at the agency receives an initial evaluation from three independent experts who do not work for the agency or for institutions employing the applicants, a spokeswoman for NSF said. Large, complex project proposals often go through a multistage review process, including site visits. The agency does allow applicants to suggest individual reviewers to include (or not include) on panels, but a decision to use those suggestions is up to the program officer handling the proposal.

“NSF’s merit review process is considered by some to be the gold standard of scientific review,” said Aya Collins, a spokeswoman for the agency. “Perhaps the best evidence of NSF’s success is the repeated replication of its merit-review model for discovery, education and innovation in nations around the globe.”

The Paul legislation was referred to committee but hasn’t advanced further. But Tobin Smith, vice president for policy at the Association of American Universities, said it would be a concern to the group if the bill gained any traction.

“Clearly it is important to have people who understand the science evaluating research proposals,” he said. “And it is concerning to think of people who would not have expertise about very technical subjects and technical grant proposals being part of the review process.”