Psychologists’ betting market hints at most reliable research findings

When psychologists are asked which findings in their field will pass a replication attempt, their predictions prove little better than flipping a coin. But if they play a betting market on the same question, the aggregate results give a fairly good prediction of which studies will hold up if repeated, a prediction experiment suggests¹.

The study was run in parallel with a large, crowd-sourced replication effort, which reported² in August that fewer than one-half of 100 findings in the field could be replicated by other teams. Last year, while that effort was underway but before its results were revealed, some forty-odd researchers involved in it filled out a survey indicating — for more than 40 studies — how likely they thought that study’s main finding was to be successfully replicated.

The same researchers also played a betting game. They were given US$100 to buy and trade contracts in the replication efforts that were going on. The contracts each paid $1 if the key finding was eventually replicated, and nothing if it was not. (Researchers were not allowed to bet on studies that they were involved in reproducing).

 

A study published on 9 November in the Proceedings of the National Academy of Sciences¹ reveals the results. The survey predictions were not significantly more accurate than a coin flip. But a model developed from the betting market accurately predicted the results of 71% of the replication efforts, significantly outperforming the simple survey. The market was a little optimistic, however: the average contract price was $0.55, suggesting that players thought 55% of the studies would be replicated.

That suggests that research communities are fairly good judges of which research will stand up to replication, say other scientists. “The results show that a collection of knowledgeable peers do have a good sense of what will replicate and what won’t,” says behavioural economist Colin Camerer at the California Institute of Technology in Pasadena. “This information is in the judgements of peers but has never been collected and quantified until now.”

More here – Nature