How to share and analyze our MRI data best: COBIDAS Report

SHRUTI GOPAL VIJ
COBIDAS: OHBM Committee on Best Practice in Data Analysis and Sharing 

Neuroimaging researchers study both the structural and functional organization of the brain in health as well as a variety of neuropsychological conditions. Extensive exploration has been conducted over the past few decades using novel experimental and analysis techniques. However, recent evaluations of these techniques have highlighted concerns that published scientific results are less reliable or reproducible primarily due to the lack of transparency in research practices. To address the need for outlining principles of scientific research that will increase transparency and reproducibility in human neuroimaging, the Organization for Human Brain Mapping created the Committee on Best Practices in Data Analysis and Sharing (COBIDAS).

COBIDAS has initiated efforts in distilling best practices for open science in human neuroimaging and has compiled suggestions for specific research practices to support open data and open methodology. These suggestions were composed in a report that was drafted and ratified using community collaboration. While the report itself provides a detailed description of best practices as well as approaches to avoid, a commentary that reviewed the impact of COBIDAS and the challenges ahead was published in February 2017.

The commentary underlines the importance of reproducibility and highlights the different aspects of replicating a study including generalizability over methods, materials and results as shown in Figure 1. One practice that was universally recommended is the transparent and complete reporting of all facets of a study, allowing the critical reader to evaluate the work and fully understand its strengths and limitations. Additionally, thorough reporting will equip the reader/other researchers with a detailed knowledge of how to fully replicate the study.

COBIDAS MRI report presents reporting checklists in Appendix D that researchers can follow while preparing manuscripts. These checklists are highly comprehensive and run the gamut of stages a human neuroimaging study passes through. The report also pinpoints specific bad practices, especially in statistical modeling and inference (Pages 11 & 12), and provides concrete recommendations for avoiding them. COBIDAS also suggest that researchers at all levels be mindful of these considerations while conducting individual studies and compiling manuscripts.

Figure 1: Different conditions of reproducibility and generalizability of MRI studies. For example, results can be reproduced using the same data, same analysis code and same data analysts. Results can also be generalized by using same analysis and experimental methods but with different analysts and experimenters or subjects. This graph summarizes the different dimensions in which experimental results can be reproduced. Image created by the OHBM Committee on Best Practice in Data Analysis and Sharing (COBIDAS).

Another strong recommendation from COBIDAS is data sharing as highlighted in section 7 (Pages 19-24). While sharing of results was strongly agreed upon by the imaging community and COBIDAS, some concerns were articulated by the community. The main concerns of the neuroimaging community outside of COBIDAS included data ownership, maintaining autonomy over results, concern of uncovering errors in analysis, cost of data sharing, and protection of  subject privacy. COBIDAS posits that there are ways to overcome each of these concerns by not only being personally open to the fallible nature of individual research and welcoming collaborations initiated by data sharing, but also by emphasizing the role of the organization and the lab in making these practices widespread. COBIDAS also suggests that a culture of constructive criticism needs to be imbued in the community to allow researchers to work on reproducibility studies. They credit some studies such as Waskom et al, Whitaker et al and Pernet et al as well as the Montreal Neurological Institute that have become the flag bearers of open science by sharing their research data and analysis scripts.

Such recommendations and practices are leading to the development of new data sharing methods and open science tools for neuroimaging. Some of the identified tools for data sharing at various experimental stages include the Brain Imaging Data Structure (BIDS); the CBRAIN web-based analysis service; the COINS service; the LONI pipeline; the Neurovault repository; and the FCP/INDI and ADNI data sharing repositories. The COBIDAS recommends using the many resources available to share data,  analysis scripts, and well-written code, along with unthresholded result maps to allow precise meta-analysis and follow up studies.

Finally, the COBIDAS acknowledges that while many of the practices suggested and recommended require individuals to change the way they approach implementation and reporting of a research study, advancement of these open science initiatives will only bear fruit by institutional support and direction. They posit that universities and research centers, as well as the journals themselves, should require data and code sharing. Foundations and grant agencies should also recognize and fund the explicit costs of data sharing. However, none of these will be able to single-handedly bring about the radical change that open science warrants. COBIDAS seeks a coordinated effort from individual researchers, research centers,universities,grant agencies, and it also invites professional organizations to accelerate the drive towards open science via extensive education and outreach. 

What does this mean for you and me as human brain mappers? We need to be more transparent in what we do. Not just because of the implications this has on clinical applications, but to make human brain mapping a more reliable field. Small changes in our approach to science such as documenting all the analytical methods applied while working through the projects and sharing this document on our personal sites or as supplementary material to manuscripts is a small initiative we can all take. Another simple task would be to share code written for projects on code sharing platforms such as github. At the level of the PI’s, making it a lab requirement to share this information with all the lab members including de-identified participant information and maintaining documentation for all projects: ongoing and completed would be speed up the process. This also makes it easy to follow up on projects once postdocs and graduate students have moved on. 

As they say- “Little drops of water make the mighty ocean”, if we all commit to do our part, we can make open science a reality and very soon.