Quality in preclinical research: Networked rather than alone
Neuroscientist Sarah McCann wants to improve the quality of biomedical research and make it more reproducible. The Volkswagen Foundation is funding her research as a pioneering project in the new profile area "Understanding Research – Evaluation and Science Practice".
Sarah McCann from the QUEST Center of Berlin Institute of Health (BIH) at the Charité has an ambitious goal and a firm conviction: Only through the disclosure and networking of biomedical data in a science system where everyone participates as part of a large team, can research be truly successful. Her idea is to establish "Communities for Open Research Synthesis." In the following, Göttingen science journalist Ulrike Gebhardt interviews the pioneer about her COReS project.
Dr. McCann, what role did your scientific background play in formulating your idea?
I come from Australia where I studied neuroscience and physiology at the University of Melbourne. For my doctoral thesis, I focused on the nerve damage that can occur following a stroke. The goal was to develop therapies for stroke patients based on rodent models. There, and during a research stay at the University of Newcastle, a recurring problem of preclinical research showed quite plainly: Unfortunately, experimental therapies that hold promise when applied in animal models prove to be unsuccessful in clinical trials with humans. This problem of translational failure is not only common in stroke research, but in biomedicine in general.
What lies behind this problem?
There are very different reasons for it: The animal models may quite simply be unsuitable for translation into treatment for humans. Or animal experiments may have produced false, seemingly positive results. Or the studies in humans were not conducted under the same conditions as in the animal model. Flawed study design or incomplete reporting of results often overestimate treatment effects in animal studies. More importantly, though, in most cases negative data are not publicly available, leading to mistaken assumptions. When incorrect results become the basis for clinical trials, failure is a foregone conclusion.
Is it possible to express in numbers the magnitude of the damage caused by the translation problem?
In preclinical stroke research, it is estimated that less than half of the apparently promising results published are truly positive. The extent of the translation problem varies. But the majority of therapeutics identified as promising in preclinical research fail in subsequent clinical trials. The magnitude of the problem cannot be expressed solely in terms of losses in time or money. Above all, there are ethical consequences too. People are exposing themselves to therapies that may be useless or even damaging.
What way out do you think there is?
My own experiences in research prompted me to get behind the causes of poor translation. During my time at the University of Edinburgh, I joined the "CAMARADES" group (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies). Colleagues there aim to improve preclinical research by pooling published data in the form of systematic reviews and meta-analyses. For more than three years now, I have been working at the QUEST Center of the BIH, where I have also established a CAMARADES group. The COReS project will help us to establish the concept on a larger scale: The title stands for "Communities for Open Research Synthesis - accelerating translation of biomedical evidence".
Why might systematic reviews be a way out?
A systematic review compiles all the study data available and thus provides an overview of the knowledge in a defined area of research and of the data quality. It makes it possible to discover gaps in knowledge. Such a complete picture of the state of research facilitates the planning of further research. The systematic review not only increases the chances that the most promising therapies, i.e., those with the most robust evidence, will actually advance into clinical research. It also avoids duplicative and unnecessary experiments. This also reduces the number of animal experiments.
Who in the community do you want to target with COReS?
Ideally, we will reach out to everyone involved in biomedical research and motivate them to participate. First of all, the most important people are those working in the laboratory and actually producing the data.
Ideally, we will reach out to everyone involved in biomedical research and motivate them to participate.
How should COReS work in concrete terms?
Until now, there was only a small community that produced systematic reviews – but mostly without direct connection to the researchers who primarily provide the data. In the COReS groups, we promote active exchange between researches, statisticians and information specialists.
Working as an international network, CAMARADES also supports the production of preclinical reviews. The Berlin CAMARADES team now wants to use COReS to provide the basic structure for long-term integration of the systematic review into the research pipeline, initially for medical research institutions in Germany. Building on the existing CAMARADES infrastructure, the COReS project aims to enable such important syntheses of results on a broad basis, by means of a specific collaboration platform and team-building initiatives.
One example of how COReS could work is that doctoral students from research teams additionally join a local or national COReS group, where they participate in creating an overview of the existing literature in their respective research field. The results flow into their own experimental work. The data from this are in turn transferred directly into the review.
Sarah McCann works at the QUEST Center of Berlin Institute of Health (BIH) at the Charité . (Photo: private)
How can this be achieved?
Within the framework of COReS, we plan to initiate one or more such groups at Charité. This model community will be adapted to the needs of the members and then extended to our selected "early adopter" institutes in Germany. The most important thing is that by involving the researchers themselves we can build a community-driven network, which will eventually be self-sustaining.
The most important thing is that by involving the researchers themselves we can build a community-driven network
How does "the system" need to change?
We need to bring about a real cultural shift – away from individual to team-based research. We also need recognition of the benefits of high-quality and reproducible science. This might sound self-evident, but unfortunately the prevailing academic structures present an obstacle to such a perspective: A first or senior authorship of a publication in a "high-impact" journal is still mainly considered as a quality feature per se – and not the quality of a study or even its necessity in the context of the overall state of research.
What steps are needed to bring about such a shift?
In our opinion, university education must play a role here, especially since many researchers are not so familiar with topics such as systematic review or synthesis of results. Ideally, they should already be dealt with in the curriculum. Or at the latest when setting out as an early researcher. We want to raise awareness of the benefits of systematic reviews among doctoral students and postdocs, for example, and give scientists the skills they need to create such reviews themselves and integrate them into their workflow. All of this costs time and money. We are pleased that the Volkswagen Foundation sees the value of what we want to do and is offering support.
We need to bring about a real cultural shift – away from individual to team-based research
For which disciplines outside biomedicine might your project also be of interest?
For most! In any field, it is important to first get an overview of the data situation and the quality of the data. The results of a single study are unlikely to be sufficient to decide which direction research should take in the future. Apart from biomedicine, many other disciplines have already embraced systematic review. But to our knowledge, we are the first to specifically encourage the formation of collaborations with the goal of synthesizing and integrating data. As part of our pilot project at the Charité, we have developed training materials for workshops on the topic and have so far provided them to more than 170 trainees and researchers. So far, we have been able to provide methodological support for 23 projects. We use a free, open-access online platform, "SyRF", with which systematic reviews and meta-analyses can be created. This tool supports the networking of researchers worldwide. More than 850 scientific projects are registered with SyRF, mainly those that also work with animals in the context of biomedical research.
What obstacles have to be overcome?
Researchers are usually very busy and often not so familiar with the methods we propose. We are aware of the pressure of time and resources that most have to struggle with. We therefore focus on raising awareness of the problem and offering the appropriate infrastructure so that the barrier to implementation can be kept as low as possible. For example, we enter at the institutional level, asking where or when we can best incorporate our training offerings into structural operations. Effective communication of the benefits, but also the limitations, of our methods will play an important role in overcoming resistance.
Scientists want their research to be meaningful and effective, and we must communicate how our initiative can help
What do you think are the chances for successful implementation?
We have put a lot of work into planning the implementation and feel the time is right for COReS. Scientists want their research to be meaningful and effective, and we must communicate how our initiative can help. The infrastructure we are developing will be free and open-access to users. Anyone who wants to create a review will eventually be able to do so. We believe we have all the ingredients for success and can create lasting change with the help of the project. I am optimistic.
