No Agreement Among Reviewers of Grant Applications

A new study in Proceedings of the National Academy of Sciences finds little to no agreement among reviewers of grant applications to the National Institutes of Health. After replicating the NIH’s peer-review process, researchers analyzed 43 reviewers’ ratings and critiques of the same 25 grant applications. After eliminating weaker proposals, reviewers did not agree on the applications’ quality in either their quantitative or qualitative evaluations. Reviewers also differed in how they translated applications’ strengths and weaknesses into a numeric rating. The study’s authors say that the reviewer ended up having more to do with a grant application’s success than the proposed research. 

“We’re not trying to suggest that peer review is flawed, but that there might be some room to be innovative to improve the process,” co-author Elizabeth Pier, a postdoctoral fellow in educational psychology at the University of Wisconsin at Madison, said in a news release. Among other changes, Pier and her co-authors recommend a modified lottery system, in which weaker proposals are eliminated and the remaining applications are funded at random.