Search the site...

ORGANIZATION FOR HUMAN BRAIN MAPPING
  • BLOG
  • Tutorials
  • Media
  • Contributors
  • OHBM WEBSITE
  • BLOG
  • Tutorials
  • Media
  • Contributors
  • OHBM WEBSITE

Q&A with OHBM 2018 Replication Award Recipient: Benedikt Sundermann

1/4/2019

0 Comments

 
By Chris Gorgolewski & Ekaterina Dobryakova

Reproducibility and transparency are core to all branches of science. Two years ago, OHBM established the Reproducibility Award. The purpose of this award is to honor researchers who conducted a ‘successful’ or ‘unsuccessful’ replication study, while adhering to rigorous standards of study design, data collection and analysis. The second recipient of the Reproducibility Award is Benedikt Sundermann, who received the award during OHBM 2018 in Singapore for his study that was published in the Journal of Neural Transmission. Chris Gorgolewski, one of the initiators of the OHBM Reproducibility Award, interviewed Benedikt about his experience related to this replication study.

PictureBenedikt Sundermann (right) receiving the 2018 OHBM Replication Award from Professor Bernard Mazoyer (2018 OHBM Chair)
Chris Gorgolewski (CG): I am joined here today by Benedikt Sundermann, the recipient of 2018 OHBM Replication Award. Benedikt, thank you for joining us and congratulations on the award.

Benedikt Sundermann (BS): Thank you.

CG: The first question I want to ask you is how would you describe the study if you met a stranger a bar?

BS: In previous studies, people have tried to apply artificial intelligence technologies that are frequently used in face recognition to functional brain imaging data in order to try and diagnose people, for example, with depression. In our study, we wanted to see whether this replicates in a larger and more clinically realistic sample, featuring various comorbidities, heterogeneous age, sex etc. Surprisingly, most of the previously reported results did not replicate in this larger, more heterogeneous and clinically realistic sample. Only when we looked at a subgroup of people could we replicate some models but, still, at a diagnostic accuracy that would not be clinically useful.

CG: I see. So that is quite a controversial statement, especially considering how much we discuss about the application of neuroimaging methods to the clinic. Did you experience challenges in publishing this work?

BS: Yes, definitely. If I remember correctly, it was the fourth or fifth submission of this work that we finally managed to publish. Generally, when you try to publish a replication study with a negative result, you should be prepared to be interrogated much more critically by the reviewers, about the sample, about the methods and so on. But there was also criticism that our work was not scientifically solid. For example, there were comments like “We know from previous studies that this works, the fact that you couldn’t replicate it means that you must have made a mistake.”

CG: I see, but you must have, in a way, anticipated that it’s going to be somehow challenging. I want to understand a bit more what motivated you to do this wonderful work to begin with.

BS: I have a clinical background in radiology and I am mostly interested in actual diagnostic tools and to expand the spectrum of diagnostic tools that we can use in the clinic. In major disorders, we are usually limited to the exclusion of larger structural lesions but we cannot really get the diagnostic information about the actual disease mechanisms and disease correlates in these people. So my main motivation was to work on these technologies to improve these diagnostic imaging techniques based on functional imaging and machine learning.

CG: So after having completed the study - you know they say ‘hindsight is 20/20’ - if you were to take the trip back to the past in a time machine, what would you do differently? What advice would you give to people who are planning to do a replication study?

BS: First, you need to expect frustration. We did not expect that much frustration that we experienced during the study and when we tried to publish it. So just be prepared and then it will probably be easier, I think. The other thing is, in a replication setting you might be interrogated more strictly about your sample and about your analyses. Having a good structure of your analysis and your data will make it a lot easier. This is also one of the points that you are also working on in your initiative, so I think this is pretty important and we should focus more on that.

CG: Great. So what’s next for you? What are you excited about? What are you working on right now?

BS: One thing that I am working on right now is more clinical than scientific. I will be working on project optimisation/project standardization from a clinical point of view. Scientifically, I am currently interested in multi-modal integration of functional and structural imaging data in diagnostic models.

CG: So do I understand correctly that the main implication of your finding is that resting state fMRI is not ready for clinical use?

BS: I think it is currently not ready for clinical use. But I think it is important to realize that this is not an endpoint. Our findings do not suggest that there is zero information about depression in these data. We looked at it with machine learning techniques that were available two to three years ago and there has been a lot of improvement in this field since. Also, the current classification system of patients into major disorders has room for improvement. For example, just saying whether somebody has unipolar depression or not, may not be sufficient to capture the wide spectrum and large heterogeneity of these patients.

CG: So you think that the main reason for the lack of diagnostic ability are noisy labels and poor definitions of the phenomena that we are trying to predict?

BS: I think that it is at least one major determinant.

CG: So as a clinician how do you improve classification of methods?

BS: Currently, the main classifications are based on clinical interviews. You have to ask standardized questions, and you have a checklist to see whether certain criteria are fulfilled or not. But maybe we can use biological information itself to further sub-categorize patients’ mental disorders, and to see some commonalities or links between different mental disorders.

CG: That sounds very exciting! The OHBM replication award will run next year so I encourage everyone to submit their replication studies and see you next year.

Please note, the submission deadline to be considered for the Replication Award is January 11th, 2019.
0 Comments

Your comment will be posted after it is approved.


Leave a Reply.

    BLOG HOME

    ​TUTORIALS

    ​MEDIA

    ​contributors

    ​OHBM WEBSITE

    ​

    OHBM OnDemand 
    ​Education Platform


    RSS Feed

    Archives

    January 2024
    December 2023
    November 2023
    October 2023
    September 2023
    August 2023
    July 2023
    June 2023
    May 2023
    April 2023
    March 2023
    January 2023
    December 2022
    October 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    January 2022
    December 2021
    November 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    May 2017
    April 2017
    March 2017
    February 2017
    January 2017
    December 2016
    November 2016
    October 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016

stay connected with ohbm!


become a member

Telephone

952-646-2029

Email ohbm

EMAIL BLOG TEAM
Header image created by Thiebaut de Schotten & Batrancourt  
www.brainconnectivitybehaviour.eu