A Fruitful Rendezvous at SIPS: Neuroimagers Meet Study Preregistration Advocates

By Johannes Algermissen, James Bartlett, Remi Gau, Stephan Heunis, Eduard Klapwijk, Matan Mazor, Mariella Paul, Antonio Schettino, David Mehler 

The neuroimaging field has recently seen a substantial surge in new initiatives that aim to make research practices more robust and transparent. At our annual OHBM meetings you will have likely come across the Open Science room. While many aspects fall under the umbrella term Open Science, for this post we focus on research practices that aim to make science more replicable and reproducible. These include non peer-reviewed study preregistration, peer-reviewed registered reports that reward researchers’ study plan with in-principle acceptance before data collection, but also code and data sharing tools such as NeuroVault and OpenNeuro.

As neuroimagers, we work closely with and learn from other disciplines, including Psychology. One place where a lot of grassroot development has come to fruition in recent years is the annual meeting of the Society for the Improvement of Psychological Science (SIPS). SIPS breaks with the traditional conference format and focuses on practical work, peer projects and solving concrete problems in groups. The SIPS experience can feel a bit like a playground for research practice geeks: participants sit in the driver's seat and can pick from a variety of so-called unconferences where they pitch and debate ideas to reform research practices, hackathons where everyone can contribute their “bits” and thoughts, and workshops where you can catch up on learning to use the latest R packages or Bayesian analysis. In this vibrant setting we embarked as a group of enthusiastic neuroimagers on an expedition to intermingle with other open science crowds. We wanted to find out how study preregistration and registered reports could be tailored more towards neuroimaging studies. Prepared with a list of challenges that we learned about through our informal survey, we felt determined to provide more clarity around adequate statistical power in our field, and strived to ultimately come up with a potential user-friendly template for preregistration of neuroimaging studies. We completed some initial steps at the hackathon and the immediate aftermath with a focus on tools that help researchers preregister their studies. Here, we summarize our group projects and provide you with some (interim) outcomes. 
​

Collection of preregistrations and registered reports in neuroimaging 
Preregistration and registered reports are ways to state in advance what your hypothesis is and how you are planning to run and analyze the study. They are meant as tools to prevent researchers’ own cognitive bias (e.g., hindsight bias or confirmation bias) hijacking their investigation. They are not meant to stifle exploration but to make very explicit what part of a study was confirmatory and what part was exploratory (see http://cos.io/prereg/ and http://cos.io/rr/ for more details). Preregistration protocols have been around for a while for clinical trials but they have only started in the past few years to be on the radar of psychology researchers. The uptake seems to have been much slower in research involving (f)MRI, EEG, or MEG. Apart from the large amount of methodological and analytical detail needed to preregister neuroimaging studies, one reason may be the lack of examples of what a preregistration in those fields could look like. Those M/EEG and fMRI preregistrations and registered reports scattered on the internet are also hard to find. Therefore, during the hackathon, we started a list of all the openly available neuroimaging preregistrations and registered reports. This resulted in a spreadsheet, accompanied by keywords to make it easier to select relevant ones you are interested in. This document is still a work in progress and we welcome contributions to this potentially ever-growing list, especially if we missed one of your own preregistrations! Simply use this form to add an entry. We hope that such an easily accessible list of preregistrations will inspire many more neuroscientists to preregister their studies and will help to establish best practices.  

BrainPower: resources for power analysis in neuroimaging 
Every planning phase of an empirical neuroimaging research project should consider sample size and statistical power: How big is the effect that I am interested in? How likely am I to observe it given the resources (number of participants, number of trials) at my disposal? Power analysis should provide clarity on these questions. It might appear relatively easy for simple designs with one-dimensional behavioural variables, especially with the help of programs such as G*Power and standard effect size measures such as Cohen's d. However, the high-dimensional nature of neuroimaging data and designs (processing three-dimensional data over time with mass univariate and multivariate approaches) requires additional steps, e.g., cluster correction, to prevent false-positive inference. And our understanding of "effects" based on these data and methods is not necessarily as intuitive: how strong should the level of activation be, or how large should the cluster be? 

One important approach to power analysis is simulations: When taking resting-state data and adding an activation of a certain size and extent, can I reliably find the effect? This approach has been facilitated by advances in computational power and new software in recent years, allowing researchers to have full control of the ground-truth. Alternative approaches to estimate effect sizes is relying on past literature (which may provide biased estimates) and re-using existing, or even open data sets. 

For both approaches, experts have created primers, tools, and software. Unfortunately, their use  may not always seem intuitive. Further, researchers might have a hard time recognizing which tools best suits their specific needs. We thus collated a variety of such tools, compared these different approaches, and described their use (“how to”) to empirical researchers. Overall we gather collection that provides:

1. Informative journal articles on the underlying principles of power analysis
2. Software toolboxes for conducting power analysis in fMRI
3. Tools for simulating fMRI data with which to pilot power analyses
4. A list of useful questions to ask or heuristics to consider when planning an experiment.

This list of resources is openly available and still growing in content. The immediate future goal is to expand the resources with tutorials and work examples of conducting power analyses on real and simulated fMRI data. We then plan to formalise these resources into a website. We invite and welcome any and all contributions from the community!

A new way to calibrate the smallest effect size of interest (SESOI) for neuroimaging, using an fMRI example
Adequate sample size planning is crucial to make good use of resources and draw valid inferences from imaging data. One-size-for-all recommended sample sizes are slowly being replaced by power analysis procedures that are based on effect sizes that seem reasonable. In the more common approach, effect sizes are estimated based on available data or previous studies. However, this approach does not account for the ability to necessarily detect a meaningful effect size. An alternative approach is to power studies sufficiently to detect the smallest effect size of interest (SESOI), thereby increasing the chance to find an effect that is meaningful for the research question (e.g., for practical, or theoretical reasons). Also, in the event of a non-significant (i.e., “null”) finding, this approach increases the chances to reject negligible effect sizes, rendering “null findings” more informative. Hence, while this approach is more rigorous, it often requires larger samples, especially when studying higher order cognitive functions where group effect sizes are known to be small. On the other hand, running too many participants also comes with a cost: scanner time is an expensive resource of limited availability. Identifying a procedure that can balance this trade-off would thus be desirable and potentially help researchers to implement a sampling plan that is based on a SESOI. 
​
We thus started with the following thought experiment: in an attempt to optimize sample sizes for specific experiments and statistical tests, one can capitalize on the fact that neuroimaging data is rich and affords numerous statistical tests that are statistically orthogonal. It is safe to assume that some sources of noise are shared between contrasts, within a participant (for example, a participant that moves a lot in the scanner will have more noisy parameter estimates), and that other sources are shared between participants within the same lab (for example, the quality of the scanner). Based on these two points, we envision a dynamic procedure for sample size specification that is sensitive to the noise in the specific sample of participants. Implementing such a procedure seems fairly simple: data acquisition stops exactly when a group-level contrast that is orthogonal to the ones of interest reaches a pre-specified significance level in a pre-specified region of interest.
​

A preregistration template for EEG
Analyzing neuroimaging data involves a myriad of decisions that researchers often consider only after data collection. When preregistering a neuroimaging study, thinking of each detail of the analysis can be challenging, especially because current available preregistration templates are generic and do not ask for the relevant technical details and specifications that are relevant for EEG experiments. For example, preprocessing EEG data involves many decisions - including resampling, filtering, and artefact rejection - that can have a profound impact on the results.

As part of the hackathon, we started to create a preregistration template for EEG studies that highlights such decisions during preprocessing and statistical analysis. For instance, the user is reminded to describe the electrode type and brand, data import, resampling, filtering, epoching, artefact detection/rejection/correction procedures, baseline correction, and averaging. The current version of the template is a text document based on the standard OSF preregistration form where we added specific questions about preprocessing and analysis steps for event-related potentials (ERPs). This EEG preregistration template is an ongoing project. If you have worked with EEG data or preregistrations before, your input would be highly appreciated! Ultimately, we aim to include the finished template on the OSF list of preregistration forms and extend the preregistration template to other analyses of EEG data (e.g., time-frequency analyses). 

To wrap up, SIPS certainly provides a great opportunity for neuroimagers to intermingle with others and contribute to projects related to scientific practices in an open, inclusive, and dynamic environment. Anyone can pitch a session ad-hoc for the next day and the outcome of each project is openly documented on the OSF. This ensures that projects like ours on preregistration and neuroimaging can develop and live a happy after-conference life.