In conversation with PLOS Biology experimental psychology authors Jacqueline Scholl and Nils Kolling

We caught up with the authors of an exciting new study, “The effect of apathy and compulsivity on planning and stopping in sequential decision-making” published this week in PLOS Biology to discuss their hypotheses and results, next steps for the research, and their decision to preregister.

---

Interviewer: You and your coauthors have recently published a research article in PLOS Biology, in which you use job search behavior to explore how sequential decision-making is affected by apathy and compulsion. Can you tell us at a high level what your investigation entailed, and what inspired you to pursue this line of inquiry?

JS: In the past psychology experiments aimed to be as simple as possible. However, this often misses out on many complexities that people actually need to deal with in daily life. We hypothesized that these more “real life” situations are also when psychiatric traits – like apathy –show up more clearly. Our work here is in line with a more ‘ecological approach’ which takes inspiration from real-life situations that need to be solved and uses computational modeling to capture the complexities.

The specific complexity that we were interested in here is how people plan ahead when they need to think about making a series of decisions – like searching for a job. This often entails going through a sequence of getting offers and deciding whether to accept or reject them. While, we had previously investigated the brain mechanisms of the general decision making and planning processes, in this study we identified several clinical dimensions which might manifest in sequential decision making changes. For example, we hypothesized that apathy might lead people to keep searching for too long because they get ‘stuck in a rut’ (‘decision inertia’). And that compulsivity might also lead to searching for too long, but for a different reason, namely being insensitive to the costs searching entails.

NK: Another key inspiration for our study was related to introspection. Generally, decision making studies do not ask participants about their own reasoning for their behaviour but fit models to their behaviour instead. They do so in the belief that “actions speak louder than words”. However, when it comes to clinical changes some of those changes are about how we “feel” and think about our behaviour, not just what we do. This is why it’s so exciting to look at BOTH what people do and what they think together and compare them. For example, only by doing taking both measures we could uncover that apathy leads to decision inertia but no insight into this change, while compulsivity led to self-reported avoidance of situations where people feared they might search for too long, but less actual behavioural change in our task.

Interviewer: And why did you choose to preregister this study? What, if anything, can journals do to encourage preregistration and/or make it easier to do?

NK: When you want to combine a complex task with clinical and real-life dimensions as well as self-reflective report measures about their own behaviour, you need a new analysis approach to match your ambitions. Specifically, the sheer amount of potential associations when it comes to a sophisticated cognitive model that can measure multiple cognitive features and multiple clinical dimensions means you need a way to narrow down the hypothesis space. For example, while previously clinicians might have observed changes in perseverance with both compulsivity and apathy, in our model those could be due to changes in any of multiple parameters. Specifically, the concept of ‘searching too long” could be linked to distinct cognitive features and mathematical parameters in our model.  In short, even when you have clear conceptual hypotheses when formalizing them into a mathematical model for a complex task its specificity can sometimes be greater than our prior knowledge.

JS: In such a situation, it’s incredibly powerful to collect two samples as we did here with a ‘discovery sample’ (roughly 400 participants) / ‘confirmation sample’ (750 participants). The discovery sample can clarify which parameter exactly is linked to which clinical dimension (for example we found that apathy was linked to searching for too long in terms of repeating previous actions, but not in terms of insensitivity to the cost of searching). The confirmation sample gives statistical rigour by confirming (or disconfirming) these links. After all, a subset of the discovery sample results might be a fluke, but we do not know which of them.

A major advantage of this approach is that we could be guided by the data in generating precise hypotheses about clinical dimensions (in the discovery sample). For example, while we intuited that apathy would be linked to decision inertia, this was quite in contrast to previous work that had used different measurement tools. Yet at the same time, we could avoid any potential statistical pitfalls to do with explorations (e.g. p-hacking) through pre-registration before collecting the confirmation sample. The type of work – large-scale internet-based studies – is particularly suited to this approach as data collection is relatively fast (in contrast to clinical or brain imaging studies).

NK: Beyond the scientific rationales for pre-reg, there are some practical motivators that could help. Already, offering pre-registrations in well-respected journals, like PLOS Biology, is a major step forward for visibility of pre-reg studies. We hope that our study can play a role in showing other researchers how pre-reg and discovery/confirmation approaches can empower to answer new questions with new approaches. E.g. they solve problems with complex and high dimensional data.

Interviewer: For readers who may not know, in preregistered research, authors begin by first articulating a hypothesis, study design and analytical approach, which is then submitted for peer review. In your case you actually tested three hypotheses! What was it like to receive comments so early in the process, and how do you think that early peer review may have influenced the study you conducted?

JS: For our specific case, the review process was quite similar to standard peer review because we had a sizeable ‘discovery sample’ on which it was based. In fact, in the past, the discovery sample would have been for most researchers the only sample they had in the paper. Our huge advantage now was that we could boost our confidence in the validity of the findings through the confirmation sample.

One major advantage compared to standard peer-review was that when reviewers made sensible suggestions about additional control variables to collect, we could actually implement this; rather than just thinking ‘we wished we had done that’.

NK: For other types of pre-reg studies, that are earlier in the process, a lot more feedback can go into initial design, analysis and planning, showing what a large spectrum of pre-reg reports are possible!

Interviewer: And when it came time to submit the completed manuscript for publication, did the review process differ at all from your experience with classic (non-preregistered) peer review and publication? If so, in what way?

JS: Analysis and write-up of the confirmation sample were much smoother than expect, as we could just stick to our a priori plan. At the same time, we didn’t feel constrained by the process as, when we had a new idea for analyses based on publications that had emerged since the pre-registration, we could include it in a new ‘exploratory’ section. In fact, it was great to have a mix of pre-registered results and a completely new cutting-edge way of looking at the data, complementing the original plan.

NK: The post collection review itself was very pleasant and straightforward as the reviewers already had familiarity with the paper and the goals already agreed with the plan at the pre-registration stage. Nevertheless, they could give some nice feedback on our final data and suggest some further exploratory elements which can inspire and fuel future research in the field!

Interviewer: Do you think that preregistration has the potential to improve reproducibility in the field? To increase trust in scientific results among readers and the public?

NK: Definitely. We were actually surprised how pre-registration can be part of a larger cultural shift of writing more transparent and well-curated code. This is because you need to look at your own work twice and apply it to two independent data sets. For example, in our study, it allowed us to catch an indexing error we had previously missed. There are many such honest mistakes that can get caught by trying to replicate your own results. Also, by having transparent and good code, many more researchers will be willing to put their code online. In our experience what prevents people putting things online isn’t bad intentions but lack of time and embarrassment about badly curated code. If they already have the good code established, putting it online is super easy. And this also allows for more easily establishing new collaborations – e.g. with clinicians who can use our task in patient samples. Their analyses will subsequentially also be more closely aligned with our own (by re-using the same code) and more reproducible with the need for them to re-invent the wheel.

JS: For our particular example of pre-registering with complete analysis code, one very useful aspect is that it removes any ‘wriggle room’/ ‘degrees of freedom’ (in the confirmation sample) in how to translate concepts into the actual statistical tests that could still be there in a pre-registration that formulates hypotheses only in words, rather than in code. However, this would of course only apply to areas where collecting two samples is feasible.

NK: Pre-registration particularly in combination with discovery and confirmation samples have the added appeal that they are easy to explain to the public. You can simply tell them, “we found this surprising thing so we checked again and found it a second time” without having to explain more complex concepts.

Interviewer: Turning back to the research itself now, you explored two traits associated with a broad range of illnesses: apathy, which can be linked to depression, brain injury, neurodegenerative disorders, and other factors; and compulsivity, which is a core feature of OCD. On the surface to someone who’s not an expert in the field, apathy and compulsivity may appear to be opposites. Why does it make sense to study them in tandem?

JS: Being able to look at different psychiatrically traits at the same time is a key advantage of the kind of large online studies we have done here. The reason that this is important is that many psychiatric traits are correlated. For example, in a sample of patients with OCD [obsessive compulsive disorder], many would also have other diagnoses. This makes controlling for related but distinct clinical dimensions key. If one instead recruits patients with only OCD, these patients would be quite unrepresentative of the overall patient population.

Taking advantage of the ease of data collection over the internet, we could get hundreds of participants to allow us to statistically dissociate the different psychiatric traits using a ‘transdiagnostic approach’. This means, that instead of splitting participants into groups according to diagnostic categories, we measured how much of each trait (e.g. apathy or compulsivity) each person has. And then in turn we could link these traits uniquely to differences in behaviour (controlling for other traits).

NK: Importantly, our clinical dimensions were established empirically from questionnaire measures that covered a wide ranges of clinical questions/features. Thus, we can say that apathy and compulsivity are not the exact opposite from each other, as they otherwise would have been absorbed into one clinical dimension. In our study, both dimensions also appeared to lead to quite different changes. Apathy led to quite specific behavioral change in decision inertia but no corresponding change in insight. On the other hand, Compulsivity, came with a series of meta-cognitive or self-reported changes, but relatively little behavioural effects in our task.

Interviewer: Your team hypothesized that these two traits would be associated with similar outcomes, but for different reasons:

1. Apathy will be linked to decision inertia, resulting in an extended job search with a high cost to the searcher
2. Compulsivity will be linked to over-chasing, resulting in an extended job search with a high cost to the searcher
3. A belief of compulsive over-chasing would lead to a belief of pre-emptive avoidance of high-cost job-search

To what extent were your hypotheses born out? Were you surprised by the results?

NK: For the most part, our hypotheses were confirmed, which might have to do with the fact that collected a large discovery sample giving us a lot of prior information. However, we were surprised by the fact that compulsivity seemed to link only unreliable, if at all, to behavioural changes in our task but had very strong impact on metacognitive beliefs about behaviours.

How do you think your findings could potentially impact the field? Are there any implications for patient care at this stage?

JS: For apathy, our finding opens up new questions about why this decision inertia comes about. We’re at the moment planning work to understand this better and whether it’s for example related to what in psychotherapy is called an ability to ‘decenter’, i.e. take a step back from what one is currently engaged with and see the ‘larger picture’. This is a particular skill that is thought to be increased by mindfulness training. In the future, interesting to see behavioural evidence for this and based on this develop preventative interventions in healthy people

NK: Additionally, we know a lot about how the brain in general solves the task based on a previous study we’ve done. Thus, new brain-based treatments (like ultrasound stimulation) might have potential in helping people in the long term by combining stimulation with behavioural training, which is something that has shown great promise in motor recovery.

Interviewer: Anything else we should know about the work?

We hope this work will inspire more research into understanding individual differences including clinical changes related more complex and real-life cognitions. It hopefully also exemplifies how self-report or introspective measures about a specific behaviour, do not need to be at odds with mechanistic cognitive modelling of the same behaviour, but instead can complement each other. That way we can gain a deeper understanding into what mechanisms appear to be driving behaviour as well as what people belief does. So far, we have only scratched the surface of what might be possible if both types of measures are combined into more overarching models of decision making. Many of the problems that emerge when using such complex task design and adding self-report measures can be overcome with large data sets combined with a discovery and confirmation sample approach. Pre-registration serves as a great vehicle for such work as it formalizes the approach in a way that allows the researcher to get valuable feedback from the community when it still matters for the collection and analysis plans!

Lastly, we want to say that this approach can beautifully complement smaller sampled studies aimed at the neural mechanisms underpinning the general behaviour as the same cognitive models can be used to predict neural processes.

Interviewer: Thank you so much for your time, and this fascinating discussion!