Peer review has been a cornerstone of modern science for nearly 300 years, but in the past decade, many in the scientific community have begun to seriously question the anonymous aspect of conventional peer review. The application of the scientific method to the questions that arise in peer review has been long in coming. Major journals such as Nature, JAMA and BMJ have applied randomized controlled trials to compare the quality of peer review produced by different peer review practices. However these trials have been unable to measure either the accuracy of peer review or social interactions between reviewers and authors.
We developed a model of peer review in the form of an on-line game based in the Amazon cloud and we recruited groups of scientific peers to play anonymous and non-anonymous versions of the game. We compare accuracy and social behavior under open and closed systems. A striking result is that an open peer review system enhances cooperative behavior in our study groups, and that this increased cooperation improves the accuracy of the review process. In addition, reviewers are rewarded for refereeing when the group knows their refereeing behavior. These results provide strong evidence for the potential benefits of increased openness and cooperation in peer review.
