Background: There are a number of initiatives to tackle
the reproducibility crisis in biomedicine, such as replicating key landmark
studies before embarking on clinical trials. However, this is both costly and
time consuming. Another approach is to use evidence-based decision making to
verify evidence before embarking on an investment, new direction or clinical
trial.  Initiatives, such as CAMARADES, was
set up to address the risk of bias in animal experiment study design. To
complement these initiatives, we have generated novel tools to critique model and
tissue validity and the likelihood of reproducibility of early-stage biomedical
studies.  

Methods: Using systematic review techniques and
in-house validity tools we provide working examples of 1) reproducibility of
organoids to recapitulate patient response to drug treatment 2) the impact of
tissue validity on genetic mutation prevalence 3) validation of breast tumour
PDX models to identify the most representative model of the patient’s disease
4) validation of (cancer) stem cell models. One reviewer extracted data, a second
reviewer checked the data and discrepancies were resolved by consensus. We use
a traffic light system to clearly guide non-experts to understand the evidence.
The authors conclusions are contrasted to the judgments from the evidence-based
decision-making tools.

Results: 1. In a recent paper published in Science
authors concluded that ‘organoids can recapitulate patient responses in
the clinic’. However, the reproducibility tool highlighted that no sample size
calculation was performed, data was based on three replicates, patient data was
not reported quantitatively, no statistical analysis was performed to prove treatment
effect relative to control or relative to patient. 2. The prevalence of somatic gene mutations in
prostate cancer was based on tissues that were not investigated for tumour
heterogeneity or tumour content. The majority of studies clearly reported
pathology (3/5). A comparison of which studies provide the best evidence will
be presented. 3. Analysis of 5 studies of breast tumour PDX
models found that for some models the tissue of origin could not be verified, the genotype of the patient did not match the PDX and we found discordance on standard of care response.  A comparison of which studies provide the best
evidence will be presented. 4. Analysis of 5 (cancer) stem cell models found that only
one met the criteria of self-renewal and multipotency. A comparison of which
studies provide the best evidence will be presented. 

Conclusion: It is our contention that there is
significant scope for improvement in the design, analysis and reporting of
preclinical experiments. We have demonstrated that our validity tools highlight
which validation experiments have been performed and which have not. This
allows the identification of the best evidence, avoiding opinion and spin; adhering to the principles of open science. These tools will help investors make
evidence-based decisions on the most reliable models and novel technologies/research findings.