by Julia Rucklidge, PhD
There has been a lot of attention on clinical trial registration over the last decade. Essentially, because of some very clever and courageous researchers and clinicians, including Peter Gøtzsche, Ben Goldacre, Irving Kirsch and many many others, the public have increasingly become aware that the literature base to support medications, including psychiatric medications, is tainted and biased at best and fraudulent at worst.
What has been exposed is that medications, that we have been led to believe are evidenced based, are not as good as we thought they were. Negative trials haven’t been published, and researchers have been changing primary outcome measures (POMs; the measures they should have decided on before collecting data) and the time frames in which they intended to look at whether a drug was effective or not. For some medications, the difference between placebo and drug effect is small and often not clinically significant.
To address the bias occurring in the medical literature associated with selective outcome reporting, in 2005, the International Committee of Medical Journal Editors (ICMJE; www.icmje.org) introduced mandatory trial registration guidelines and member journals required prospective registration of trials prior to patient enrolment as a condition of publication. To date, no research has examined whether these guidelines are impacting psychiatry publications.
In February 2013, soon after Ben Goldacre published his bestseller, Bad Pharma, an honours student, Amelia Scott, walked in to my office looking for a project. Inspired by some of Ben’s ideas, we decided to ask a simple question. Do journals stipulating that as a condition of publication a clinical trial must be prospectively registered actually do what they say they do? Now this may not seem like a very important question, but scientists who publish know that there are rules for submission and the assumption is that those rules are followed. Why else would a journal say that a trial must be prospectively registered and then not require it? If this doesn’t happen, then how far have we really come in cleaning up the literature base and how much faith can we have in studies that are currently being published? But if we did find that journals were following the standard they set, then fantastic. We would show that the system was working and that everyone was working to meet these new standards.
So What Did We Do?
We investigated the top 5 journals that mandated prospective registration as part of the conditions of publication. These were the American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and the Journal of Clinical Psychiatry. We based the definition of “top” journal on impact factor which, while not perfect, is a reasonable way to establish influence and reputation.
We identified all clinical trials (as defined by ICMJE) that had been conducted after July 2005 (or 2007 for two journals) and published between January 2009 and July 2013. For each identified trial, where possible, we extracted trial registration information, changes to POMs between publication and registry to assess selective outcome reporting, changes to participant numbers, and funding type.
And What Did We Find?
Out of 3305 articles, 181 studies were identified as clinical trials requiring registration (that is, they should all have been prospectively registered because that’s what the journals that published the trial require).
Twenty-one (11.6%) of the 181 studies were deemed unregistered, 61 (33.7%) were retrospectively registered, 37 (20.4%) had unclear POMs either in the article or the registry and 2 (1.1%) were registered in an inaccessible trial registry. Only 60 (33.1%) studies were prospectively registered with clearly defined POMs.
BUT seventeen of these 60 (28.3%) properly registered trials showed evidence of selective outcome reporting – this means that there had been changes to POMs based on a comparison of the trial registry and the publication. In total, only 26 (14.4%) of the 181 the trials were prospectively registered and did not alter their POMs or the time frames at which they were measured. Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.
Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs. Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.
And How Were These Results Received by the Journals We Investigated?
We submitted this work to two of the five journals we investigated. One journal (JAMA Psychiatry) refused to review it, stating the paper was too focused for their journal, and the American Journal of Psychiatry reviewed it, rejected it, accepted an appeal, allowed it to be re-reviewed and then rejected it again. Clearly AJP didn’t like it. We also tried JAMA (rejected without review), BMJ (rejected without review), the Lancet (rejected without review –they at least said it was interesting but not interesting enough), and Journal of the American Medical Informatics Association (rejected without review). We finally found a home with PLoS One, where it has just been published. All the data have been submitted if someone wants to verify our findings.
So What Does This Mean?
Well it might mean nothing at all. It is possible that all those researchers who didn’t preregister their trials or who changed their outcome measures before publication did so for very benign reasons and that we can trust their results on their word without having a registry to check up on whether they did what they said they were going to do.
Should We Believe This?
History suggests that when left unchecked, researchers have been known to change their data. As demonstrated by case studies on paroxetine (1), burying primary outcome data because it is not statistically significant can have dangerous implications for the individuals who eventually end up using the treatment. Changing time frames can result in a drug appearing more effective as compared with placebo, as evidenced by the reporting of Ballenger et al (2) on Alprazolam. In this study, the researchers focused on the 4 week data despite the fact that the trial was 8 weeks. While at 4 weeks there was clear advantage of the drug over placebo, this advantage disappeared at the 8 week time point. People glorify their positive results and minimize or neglect reporting on negative results. After all that’s why the registry was developed in the first place: to change behaviour.
At worst, our findings mean that the trials published over the last decade cannot be fully trusted. And given that health decisions and funding are based on these published findings, we should be very concerned.
What Can be Done to Improve the Situation?
Looking forward, the situation could be improved if some of the following suggestions were to be adopted:
1) Member journals of the ICMJE should have a dedicated person checking trial registries, trials should simply not be published if they haven’t been prospectively registered as determined by the ICMJE or the journals should state clearly and transparently reasons why studies might be published without adhering to ICMJE guidelines.
2) If authors do change POMs or participant numbers or retrospectively register their trials, the reasons should be clearly outlined in the methods section of the publication.
3) To further improve transparency, authors could upload the full clinical trial protocol, including all amendments, to the registry website and provide the raw data from a clinical trial in a format accessible to the research community.
4) Greater effort needs to be made to ensure authors are aware of the importance of prospectively registering trials, by improving guidelines for submission (3) and when applying for ethical approval.
5) Finally, reviewers should not make decisions about the acceptability of a study for publication based on whether the findings are positive or negative as this may be implicitly encouraging authors to be selective in reporting results.
Indeed, a recent study showed that only one third of peer reviewers examined registered trial information and reported any discrepancies to the journal editors (4). It is clear that while there are guidelines in place to decontaminate the medical literature, authors, reviewers and journal editors need to work together to ensure that these guidelines are upheld.
We are investigating whether psychology is better behaved than psychiatry.
To read our entire article, it is freely available at PloS One:
Scott A, Rucklidge JJ, Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One 10:e0133718, 2015.
- Zarin DA, Tse T, Ide NC. Trial Registration at ClinicalTrials.gov between May and October 2005. N Engl J Med. 2005;353:2779-2787.
- Ballenger JC, Burrows GD, DuPont RL, Jr, et al. Alprazolam in panic disorder and agoraphobia: Results from a multicenter trial: i. efficacy in short-term treatment. Arch Gen Psychiatry. 1988;45:413-422.
- Knuppel H, Metz C, Meerpohl JJ, Strech D. How psychiatry journals support the unbiased translation of clinical research. A cross-sectional study of editorial policies. PLoS One. 2013;8:e75995.
- Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013;8:e59910.
First published in Mad In America