Search the site...

ORGANIZATION FOR HUMAN BRAIN MAPPING
  • BLOG
  • Tutorials
  • Media
  • Contributors
  • OHBM WEBSITE
  • BLOG
  • Tutorials
  • Media
  • Contributors
  • OHBM WEBSITE

 B​RAIN MAPPING

A Fruitful Rendezvous at SIPS: Neuroimagers Meet Study Preregistration Advocates

10/2/2019

0 Comments

 
By Johannes Algermissen, James Bartlett, Remi Gau, Stephan Heunis, Eduard Klapwijk, Matan Mazor, Mariella Paul, Antonio Schettino, David Mehler 
The neuroimaging field has recently seen a substantial surge in new initiatives that aim to make research practices more robust and transparent. At our annual OHBM meetings you will have likely come across the Open Science room. While many aspects fall under the umbrella term Open Science, for this post we focus on research practices that aim to make science more replicable and reproducible. These include non peer-reviewed study preregistration, peer-reviewed registered reports that reward researchers’ study plan with in-principle acceptance before data collection, but also code and data sharing tools such as NeuroVault and OpenNeuro.
As neuroimagers, we work closely with and learn from other disciplines, including Psychology. One place where a lot of grassroot development has come to fruition in recent years is the annual meeting of the Society for the Improvement of Psychological Science (SIPS). SIPS breaks with the traditional conference format and focuses on practical work, peer projects and solving concrete problems in groups. The SIPS experience can feel a bit like a playground for research practice geeks: participants sit in the driver's seat and can pick from a variety of so-called unconferences where they pitch and debate ideas to reform research practices, hackathons where everyone can contribute their “bits” and thoughts, and workshops where you can catch up on learning to use the latest R packages or Bayesian analysis. In this vibrant setting we embarked as a group of enthusiastic neuroimagers on an expedition to intermingle with other open science crowds. We wanted to find out how study preregistration and registered reports could be tailored more towards neuroimaging studies. Prepared with a list of challenges that we learned about through our informal survey, we felt determined to provide more clarity around adequate statistical power in our field, and strived to ultimately come up with a potential user-friendly template for preregistration of neuroimaging studies. We completed some initial steps at the hackathon and the immediate aftermath with a focus on tools that help researchers preregister their studies. Here, we summarize our group projects and provide you with some (interim) outcomes. 
Picture

Collection of preregistrations and registered reports in neuroimaging 
Preregistration and registered reports are ways to state in advance what your hypothesis is and how you are planning to run and analyze the study. They are meant as tools to prevent researchers’ own cognitive bias (e.g., hindsight bias or confirmation bias) hijacking their investigation. They are not meant to stifle exploration but to make very explicit what part of a study was confirmatory and what part was exploratory (see http://cos.io/prereg/ and http://cos.io/rr/ for more details). Preregistration protocols have been around for a while for clinical trials but they have only started in the past few years to be on the radar of psychology researchers. The uptake seems to have been much slower in research involving (f)MRI, EEG, or MEG. Apart from the large amount of methodological and analytical detail needed to preregister neuroimaging studies, one reason may be the lack of examples of what a preregistration in those fields could look like. Those M/EEG and fMRI preregistrations and registered reports scattered on the internet are also hard to find. Therefore, during the hackathon, we started a list of all the openly available neuroimaging preregistrations and registered reports. This resulted in a spreadsheet, accompanied by keywords to make it easier to select relevant ones you are interested in. This document is still a work in progress and we welcome contributions to this potentially ever-growing list, especially if we missed one of your own preregistrations! Simply use this form to add an entry. We hope that such an easily accessible list of preregistrations will inspire many more neuroscientists to preregister their studies and will help to establish best practices.  

BrainPower: resources for power analysis in neuroimaging 
Every planning phase of an empirical neuroimaging research project should consider sample size and statistical power: How big is the effect that I am interested in? How likely am I to observe it given the resources (number of participants, number of trials) at my disposal? Power analysis should provide clarity on these questions. It might appear relatively easy for simple designs with one-dimensional behavioural variables, especially with the help of programs such as G*Power and standard effect size measures such as Cohen's d. However, the high-dimensional nature of neuroimaging data and designs (processing three-dimensional data over time with mass univariate and multivariate approaches) requires additional steps, e.g., cluster correction, to prevent false-positive inference. And our understanding of "effects" based on these data and methods is not necessarily as intuitive: how strong should the level of activation be, or how large should the cluster be? 

One important approach to power analysis is simulations: When taking resting-state data and adding an activation of a certain size and extent, can I reliably find the effect? This approach has been facilitated by advances in computational power and new software in recent years, allowing researchers to have full control of the ground-truth. Alternative approaches to estimate effect sizes is relying on past literature (which may provide biased estimates) and re-using existing, or even open data sets. 

For both approaches, experts have created primers, tools, and software. Unfortunately, their use  may not always seem intuitive. Further, researchers might have a hard time recognizing which tools best suits their specific needs. We thus collated a variety of such tools, compared these different approaches, and described their use (“how to”) to empirical researchers. Overall we gather collection that provides:

  1. Informative journal articles on the underlying principles of power analysis
  2. Software toolboxes for conducting power analysis in fMRI
  3. Tools for simulating fMRI data with which to pilot power analyses
  4. A list of useful questions to ask or heuristics to consider when planning an experiment.
Picture

This list of resources is openly available and still growing in content. The immediate future goal is to expand the resources with tutorials and work examples of conducting power analyses on real and simulated fMRI data. We then plan to formalise these resources into a website. We invite and welcome any and all contributions from the community!

A new way to calibrate the smallest effect size of interest (SESOI) for neuroimaging, using an fMRI example

Adequate sample size planning is crucial to make good use of resources and draw valid inferences from imaging data. One-size-for-all recommended sample sizes are slowly being replaced by power analysis procedures that are based on effect sizes that seem reasonable. In the more common approach, effect sizes are estimated based on available data or previous studies. However, this approach does not account for the ability to necessarily detect a meaningful effect size. An alternative approach is to power studies sufficiently to detect the smallest effect size of interest (SESOI), thereby increasing the chance to find an effect that is meaningful for the research question (e.g., for practical, or theoretical reasons). Also, in the event of a non-significant (i.e., “null”) finding, this approach increases the chances to reject negligible effect sizes, rendering “null findings” more informative. Hence, while this approach is more rigorous, it often requires larger samples, especially when studying higher order cognitive functions where group effect sizes are known to be small. On the other hand, running too many participants also comes with a cost: scanner time is an expensive resource of limited availability. Identifying a procedure that can balance this trade-off would thus be desirable and potentially help researchers to implement a sampling plan that is based on a SESOI. 
​

We thus started with the following thought experiment: in an attempt to optimize sample sizes for specific experiments and statistical tests, one can capitalize on the fact that neuroimaging data is rich and affords numerous statistical tests that are statistically orthogonal. It is safe to assume that some sources of noise are shared between contrasts, within a participant (for example, a participant that moves a lot in the scanner will have more noisy parameter estimates), and that other sources are shared between participants within the same lab (for example, the quality of the scanner). Based on these two points, we envision a dynamic procedure for sample size specification that is sensitive to the noise in the specific sample of participants. Implementing such a procedure seems fairly simple: data acquisition stops exactly when a group-level contrast that is orthogonal to the ones of interest reaches a pre-specified significance level in a pre-specified region of interest.
​
Picture

A preregistration template for EEG
Analyzing neuroimaging data involves a myriad of decisions that researchers often consider only after data collection. When preregistering a neuroimaging study, thinking of each detail of the analysis can be challenging, especially because current available preregistration templates are generic and do not ask for the relevant technical details and specifications that are relevant for EEG experiments. For example, preprocessing EEG data involves many decisions - including resampling, filtering, and artefact rejection - that can have a profound impact on the results.

As part of the hackathon, we started to create a preregistration template for EEG studies that highlights such decisions during preprocessing and statistical analysis. For instance, the user is reminded to describe the electrode type and brand, data import, resampling, filtering, epoching, artefact detection/rejection/correction procedures, baseline correction, and averaging. The current version of the template is a text document based on the standard OSF preregistration form where we added specific questions about preprocessing and analysis steps for event-related potentials (ERPs). This EEG preregistration template is an ongoing project. If you have worked with EEG data or preregistrations before, your input would be highly appreciated! Ultimately, we aim to include the finished template on the OSF list of preregistration forms and extend the preregistration template to other analyses of EEG data (e.g., time-frequency analyses). 

To wrap up, SIPS certainly provides a great opportunity for neuroimagers to intermingle with others and contribute to projects related to scientific practices in an open, inclusive, and dynamic environment. Anyone can pitch a session ad-hoc for the next day and the outcome of each project is openly documented on the OSF. This ensures that projects like ours on preregistration and neuroimaging can develop and live a happy after-conference life. 
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    BLOG HOME

    ​TUTORIALS

    ​MEDIA

    ​contributors

    ​OHBM WEBSITE

    ​

    OHBM OnDemand 
    ​Education Platform


    RSS Feed

    Archives

    October 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    January 2022
    December 2021
    November 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017
    April 2017
    March 2017
    February 2017
    January 2017
    December 2016
    November 2016
    October 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016

stay connected with ohbm!


become a member

Telephone

952-646-2029

Email ohbm

EMAIL BLOG TEAM
Header image created by Thiebaut de Schotten & Batrancourt  
www.brainconnectivitybehaviour.eu