Giacomo Bagnara
High level of acceptance for project selection by lot
In our interview, Dr. Dagmar Simon and Dr. Martina Röbbecke from Evaconsult report on initial results from their accompanying research project on the partially randomized procedure applied in the "Experiment!" funding initiative.
The aim of the Volkswagen Foundation’s funding initiative "Experiment!" is to support scientists who explore bold research approaches in the natural, life and technical sciences. Since 2017, the Foundation has been selecting some of the projects funded within the initiative via a partially randomized procedure by lot – in addition to decisions by an independent jury. In their accompanying research project, Dagmar Simon and Martina Röbbecke are investigating the effects of this selection method. Here, an interim report:
What was your initial reaction when you heard that the Volkswagen Foundation would like to select projects by lot?
Dagmar Simon: Brave – that was my first thought; and on reflection, my second one, too. People in science research have been pointing out for decades that peer review, the dominant evaluation system in science, also has its "shortfalls". Above all, the conservative assessment criteria tend to promote mainstream research, failure to reach agreement among reviewers, and gender-specific biases.
Dr. Dagmar Simon
And in terms of science policy, increasing misgivings have been voiced in recent years about the overburdening of the review system – triggered not least by the exponential growth of science and third-party funding that results in such large numbers of applications to be reviewed. In 2017, the German Science Council also recommended that consideration be given to drawing lots as a complementary procedure to peer review. The Volkswagen Foundation not only gave this consideration, but has also decided to act.
Do you think the idea has proven itself?
Dagmar Simon: It’s probably too early to reach any general conclusions. Better to ask what the recipients of funding say. They see the random selection procedure as offering better chances for higher-risk research, for more thematic and methodological diversity, opportunities for subject areas that are poorly represented on the jury, but also – albeit to a lesser extent – the risk of a lower reputation gain in the event of an approval. They appreciate the fact that the Volkswagen Foundation does not publish which projects are funded via peer review or lot, but most of them would have no problems with this. In this respect, we see broad acceptance here.
How did you proceed with your accompanying research project?
Martina Röbbecke: In the accompanying research we concentrate on two thematic areas: The first is higher-risk research and the question of whether the "Experiment!" funding line succeeds in identifying higher-risk and original research questions. This goal was at the heart of the funding initiative in the early years from 2013 to 2016. It is ambitious, if only because the funding of higher-risk research has to take account of major differences between the various subject groups. On the other hand, we examine the effects of the two different selection procedures, since the grantees were selected partly by peer review and partly in a randomized procedure by lot.
Dr. Martina Röbbecke
The usual criteria leading to success, such as publications in respected journals, are only meaningful to a limited extent because the time between the end of the funding and preparation of the publication is often too short. It can also be assumed that "higher-risk" research projects, which may also fail or only indirectly lead to partial results, are important for further research work or scientific careers, but are not readily reflected in publication activities.
In order to get to know the various funding outcomes, we therefore rely mainly on surveys of the funding recipients and their assessments. Evaluation of the online surveys and interviews already allows us to observe important effects of the funding line and also to identify discipline-specific effects.
What surprised you most during your accompanying research?
Dagmar Simon: In the interviews, we asked the recipients whether there was an alternative funding option for their research project from the well-known public research funding organizations. The unanimous answer from the life sciences, the natural sciences and the technical sciences was a resounding no. Without a direct reference to the state of the art in the respective disciplines and proof of their own publications on the project, the overriding opinion was that a submission would have had no chance at all.
A majority of the respondents were also skeptical about implementation through basic funding at universities or non-university research institutions. And we are by no means dealing with inexperienced researchers. For example, 72 percent of the respondents in the 2019 online survey had already successfully acquired third-party funding. Their assessment astonished us in its clarity. After all, new, higher-risk and as yet unproven knowledge is essential for scientific progress. It’s going to be most interesting to discuss the effects of this research funding and its importance for the science and innovation system with policy-makers.
Can you also imagine a randomized procedure for other funding programs?
Martina Röbbecke: The "Experiment!" funding line has not only generated great interest here in Germany, but also in other countries. This is hardly surprising, since the Volkswagen Foundation has thus addressed a number of structural problems within the science system, touching, for example, on how the review system is being overloaded with questions of interdisciplinarity, but also on the career perspectives of young researchers.
Our interim results on "Experiment!" already show that a randomized procedure might well provide an answer to these problems – but only under certain conditions. For instance, the high level of acceptance on the part of funding recipients observed so far is certainly also related to the fact that the funding amount is relatively small and the duration is limited.
Beyond that, other elements of the selection procedure, such as the ease with which applications can be submitted, have certainly contributed to the continuing high level of interest shown by applicants. In the course of our research, we will continue to keep a close eye on the conducive framework conditions for randomized procedures.
Where do we go from here?
Martina Röbbecke: As soon as possible, we want to discuss our initial results with other researchers. For this purpose and for an exchange of experiences with international guests, we are planning a workshop in mid 2021, to which representatives of funding organizations and policy-makers will also be invited. However, most importantly, we will conduct further interviews with the funding recipients – and also take the opportunity to discuss with them what impact the corona pandemic is having on their research work.
