27th September 2022

 

Welcome to the RAPPORT project blog!

 

RAPPORT stands for Developing a Rapid AI-based Policy Probing and Observational Research Tool

 

The project is funded by Wellcome via their Mental Health Data Prize

https://wellcome.org/grant-funding/schemes/wellcome-mental-health-data-prize

I am really excited about this project. It is an amazing opportunity to apply and publicise new, powerful data science methods for mental health research. Thank you Wellcome! 😀

I am Paul Tiffin, the principal investigator (PI) for the project. Before I let the other team members introduce themselves let me give you an overview of the project:

It’s important that treatments and health policies actually improve well-being and health. Sometimes well-intentioned healthcare interventions either don’t work for some groups of individuals, or can even cause unintended harms. 

Scientific research is vital if we are to have evidence over whether such interventions are likely to work or not, and for which groups of people. Traditionally such evidence has been often provided by ‘randomised controlled trials’ (RCTs). RCTs are experiments where usually one group of people have a particular treatment, and another group are offered an alternative treatment, or no treatment as such (e.g. a ‘placebo’, or sugar pill). Who gets which treatment is decided by chance, hence the term ‘Randomised’ controlled trial. In this way the actual causal effect of the experimental treatment, compared to any alternative or placebo, can be worked out. This is because any other characteristics of the people involved in the trial, that may be associated with the health outcome of interest, should be similar between the two groups, due to the randomisation.

However, RCTs can take a long time to set up and report their findings. They are also very expensive to run, often costing several millions of pounds. 

We now have lots of information on patients, in the form of data from everyday practice and from research studies. However, unlike in RCTs, in the real world patients aren’t given treatments according to chance. This can make working out whether a particular health policy or treatment has actually caused an improvement (or even worsening) of health very difficult.

However, using new statistical methods we can use routine health data, and those from scientific studies (even those that don’t involve randomised trials) to understand if new policies, practices or treatments cause improvements in health. Moreover, we can now use mathematical approaches to work out which interventions are likely to work for whom? 

Machine learning is part of Artificial Intelligence (AI)- where decisions can be made automatically without a human being involved. In machine learning computers can learn to recognise patterns in data in order to make predictions. 

In the RAPPORT project we will use machine learning in two main ways. Firstly, we will use traditional statistical methods used to understand population health (‘epidemiology’) and combine these with machine learning. This approach will enable us to better understand the causal effects of a mental health intervention, not just whether it is associated with better outcomes. This approach is known as ‘targeted learning’ and seems to be more effective than existing statistical methods for this purpose. Targeted learning has started to be applied to the understanding of physical health issues but there are very few examples of it being used in mental health research.

The second way we will be using machine learning is to understand how different groups of people respond differently to an intervention or treatment. These methods are known as ‘causal forests’. They work by learning the rules that can predict who is most likely to respond most positively to an intervention. This means we can identify the groups of people most likely to benefit from a certain policy or treatment. 

The initial Discovery phase of the project will apply these new approaches to the Millennium Cohort Study data to assess the impact of childhood physical activity (an ‘active ingredient’) on depression. The Millennium Cohort (also known as ‘Child of the New Century’) is following the lives of around 19,000 young people born in the UK in 2000-02 (Millenium Cohort Study). It contains a lot of information on health and lifestyle.

For the ‘Prototyping’ phase, if funded, we plan to develop some proof of concept tools, that will show how we can help other mental health researchers access these new, powerful techniques in easily available user-friendly software.

Our work will involve and engage experts with relevant lived experience from the start. Their knowledge and insights will help inform our approach to data analysis, as well as how we make sense of, report and communicate our findings.

Science in general, and machine learning studies in particular, has been criticised recently for the lack of transparency and reproducibility. By this, people mean that researchers do not always give all the details required for other scientists to be able to replicate their results, thus showing they are likely to be true and accurate.  Working with experts by experience and other stakeholders we will establish a framework for the transparent and replicable implementation of this approach. This will involve ensuring we report all the relevant details of our methods when we give our findings. Also, it will involve clearly labelling all the computer code we have used and making that publicly available also. This will help other scientists in the field understand what we did and how we got the results we report. More generally it will set high standards for reporting these kinds of studies, encouraging and supporting better practice in machine learning-based research.

Overall our project is ambitious, but feasible. We aim to both specifically understand the causal impact of ‘key ingredients ‘ related to young people’s mental health. More generally, we intend to make new, accessible, digital tools available to the research community and set a new standard for transparency and reproducibility in reporting machine learning based studies. 


Comments

Post a Comment

Popular posts from this blog

Co-producing reporting guidelines for targeted learning studies

24th February 2023 There has been well publicised problems with reproducible and transparent reporting from observational data based research studies. This is a particular issue once machine learning gets involved, come due to the complexity. Many of the studies that are published would be difficult to reproduce, even by the research team themselves. Therefore today we had a workshop with two lived experience expert researchers, an additional lived experience expert and myself. Together we went through the draft reporting guidance we have been creating for targeted learning studies. This was a very useful exercise and helped harness lay researcher and lived experience into the guidance. This was important not just for making plans of analysis and protocols available publicly before analysis starts. It was also important in thinking about how results are reported and disseminated and the role of lived experience experts in this process. These guidelines were intended to build on s
Read more
13 January 2023 Whole team event The whole team, including the lay researchers, met again to discuss a preliminary Directed Acyclic Graph (DAG). This is a diagram that describes how different variables are thought to be causally related to each. It was co-produced by the researchers and experts by experience. The DAG was intended to capture important variables that had been raised previously by the lay researchers and a young people’s advisory group. The meeting was very useful. In addition to discussing the DAG, the data scientist on the project ran through some preliminary results of the machine learning analysis to see if the findings resonated with the experts by experience. This led to some debate and the PPI lead is going to follow up with the lay researchers individually following the meeting. Finally, I delivered some training on the concept of machine learning in order to de-mystify this approach. All team members contributed during the meeting and it was great to hear a varie
Read more

The 'Table 2 fallacy'

 21st December 2022 I was asked a question the other day that got me reflecting on some issues in causal inference. The question was this: ‘…if a multivariable model includes all the relevant potential confounders then what is the advantage of using causal modelling methods ?’ Having reflected a bit on my own training in causal inference and after a quick Google I found this paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3626058/ The ‘Table 2 Fallacy’ refers to the assumption that all the effects of covariates estimated in a multivariable model are independent effect sizes. Indeed, the main exposure of interest is treated like any other covariate within a multivariable model.  It’s called the ‘Table 2 Fallacy’ because observational studies often use Table 1 to present the descriptives of the data used and Table 2 the results of a multivariable analysis. In my case it should probably be called the ‘Table 3 Fallacy’ as I usually use Table 2 to present my univariable results.  Interp
Read more