Q&A with OHBM 2018 Replication Award Recipient: Benedikt Sundermann

By Chris Gorgolewski & Ekaterina Dobryakova

Reproducibility and transparency are core to all branches of science. Two years ago, OHBM established the Reproducibility Award. The purpose of this award is to honor researchers who conducted a ‘successful’ or ‘unsuccessful’ replication study, while adhering to rigorous standards of study design, data collection and analysis. The second recipient of the Reproducibility Award is Benedikt Sundermann, who received the award during OHBM 2018 in Singapore for his study that was published in the Journal of Neural Transmission. Chris Gorgolewski, one of the initiators of the OHBM Reproducibility Award, interviewed Benedikt about his experience related to this replication study.

Benedikt Sundermann (right) receiving the 2018 OHBM Replication Award from Professor Bernard Mazoyer (2018 OHBM Chair)

Chris Gorgolewski (CG): I am joined here today by Benedikt Sundermann, the recipient of 2018 OHBM Replication Award. Benedikt, thank you for joining us and congratulations on the award.

Benedikt Sundermann (BS): Thank you.

CG: The first question I want to ask you is how would you describe the study if you met a stranger a bar?

BS: In previous studies, people have tried to apply artificial intelligence technologies that are frequently used in face recognition to functional brain imaging data in order to try and diagnose people, for example, with depression. In our study, we wanted to see whether this replicates in a larger and more clinically realistic sample, featuring various comorbidities, heterogeneous age, sex etc. Surprisingly, most of the previously reported results did not replicate in this larger, more heterogeneous and clinically realistic sample. Only when we looked at a subgroup of people could we replicate some models but, still, at a diagnostic accuracy that would not be clinically useful.

---

CG: I see. So that is quite a controversial statement, especially considering how much we discuss about the application of neuroimaging methods to the clinic. Did you experience challenges in publishing this work?

BS: Yes, definitely. If I remember correctly, it was the fourth or fifth submission of this work that we finally managed to publish. Generally, when you try to publish a replication study with a negative result, you should be prepared to be interrogated much more critically by the reviewers, about the sample, about the methods and so on. But there was also criticism that our work was not scientifically solid. For example, there were comments like “We know from previous studies that this works, the fact that you couldn’t replicate it means that you must have made a mistake.”

CG: I see, but you must have, in a way, anticipated that it’s going to be somehow challenging. I want to understand a bit more what motivated you to do this wonderful work to begin with.

BS: I have a clinical background in radiology and I am mostly interested in actual diagnostic tools and to expand the spectrum of diagnostic tools that we can use in the clinic. In major disorders, we are usually limited to the exclusion of larger structural lesions but we cannot really get the diagnostic information about the actual disease mechanisms and disease correlates in these people. So my main motivation was to work on these technologies to improve these diagnostic imaging techniques based on functional imaging and machine learning.

CG: So after having completed the study - you know they say ‘hindsight is 20/20’ - if you were to take the trip back to the past in a time machine, what would you do differently? What advice would you give to people who are planning to do a replication study?

BS: First, you need to expect frustration. We did not expect that much frustration that we experienced during the study and when we tried to publish it. So just be prepared and then it will probably be easier, I think. The other thing is, in a replication setting you might be interrogated more strictly about your sample and about your analyses. Having a good structure of your analysis and your data will make it a lot easier. This is also one of the points that you are also working on in your initiative, so I think this is pretty important and we should focus more on that.

CG: Great. So what’s next for you? What are you excited about? What are you working on right now?

BS: One thing that I am working on right now is more clinical than scientific. I will be working on project optimisation/project standardization from a clinical point of view. Scientifically, I am currently interested in multi-modal integration of functional and structural imaging data in diagnostic models.

CG: So do I understand correctly that the main implication of your finding is that resting state fMRI is not ready for clinical use?

BS: I think it is currently not ready for clinical use. But I think it is important to realize that this is not an endpoint. Our findings do not suggest that there is zero information about depression in these data. We looked at it with machine learning techniques that were available two to three years ago and there has been a lot of improvement in this field since. Also, the current classification system of patients into major disorders has room for improvement. For example, just saying whether somebody has unipolar depression or not, may not be sufficient to capture the wide spectrum and large heterogeneity of these patients.

CG: So you think that the main reason for the lack of diagnostic ability are noisy labels and poor definitions of the phenomena that we are trying to predict?

BS: I think that it is at least one major determinant.

CG: So as a clinician how do you improve classification of methods?

BS: Currently, the main classifications are based on clinical interviews. You have to ask standardized questions, and you have a checklist to see whether certain criteria are fulfilled or not. But maybe we can use biological information itself to further sub-categorize patients’ mental disorders, and to see some commonalities or links between different mental disorders.

CG: That sounds very exciting! The OHBM replication award will run next year so I encourage everyone to submit their replication studies and see you next year.

Please note, the submission deadline to be considered for the Replication Award is January 11th, 2019.