Predicting A&E Attendance - Retrospective Thoughts on an AI & ML Course

23 Feb

Introduction

In a fit of madness 6 months ago I decided to enrol in an AI and ML course from Imperial College Business School (Professional Certificate in Machine Learning & Artificial Intelligence). The course is very fast paced and took out 10-20 hours a week of my already busy schedule. However it was incredibly useful in taking a step into an area everyone is talking about and sated the FOMO ghost which had been haunting me until this point. I came back realising that there is so much more to AI than just LLMs (Large Language Model). In fact we didn’t do LLMs at all stopping on Convoluted Neural Networks. As a medic, I was also fascinated by how, now and in the future, the more advanced topics look to mimic neural pathways of the brain and visual cortex trying to emulate the complex journey of converting an image into a recognisable entity or reasoning with pure maths and code. I also realised how useful domain knowledge is on solving problems with AI and ML and hope to put this to good use in the future.

Pre-Requisites

The course required knowledge of maths in the form of linear algebra and calculus and in the end I brushed up on this with the help of my son who is a university student. The other requirement was coding, which I felt was more important. You really need to understand at least the fundamentals of coding, and it’s all in Python and Jupyter Notebooks using Numpy and Pandas at its core. I had to take a C# to Python course before, but Python has a simpler syntax. I recommend https://www.youtube.com/@3blue1brown as as primer for the maths side. The mathematic notations, although simple, really got me, but once you got your head around them, they seemed ok (at times!). The beauty of the AI course is that they really helped you with the maths and made the concepts really clear, a lot of which was from first principles, which was a great relief.

Topics Covered with brief explanations

The scope of the course covered both supervised and unsupervised learning with statistical analysis at its core. You could call an aspect of ML “Stats on Steroids”. The core to a lot of basic concepts is understanding the relationship between inputs and output via a function with the addition of a noise or unknown entity. You then split your dataset into training data, testing data and possibly validation data to see if your model fits. The following topics were covered.

Bias-Variance Trade-off - Balancing accuracy and overfitting in predictive models.
Nearest Neighbour Methods - Predictions based on closest data points.
Decision Trees - Uses branching questions to classify or predict with eg Random Forests. Gradient Boosting to refine weak models to reduce errors each round sequentially. Viewing classification problems with Confusion Matrices
Naïve Bayes - Treats each feature separately and uses probabilities
Bayesian Optimization - Efficiently searches for optimal model parameters via a “Black Box” analogy.
Logistic Regression - Estimates probability for yes/no outcomes.
Support Vector Machines - Draws an optimal boundary separating data classes.
Principal Component Analysis - Reduces data dimensions by identifying key directions.
Deep Learning and Neural Networks - Layered "brain-like" systems for complex patterns.
Reinforcement Learning - Learns actions through trial-and-error rewards.

When you add classification predictors (yes/no) vs regression (continuous values), it gets complex to which type of “tool” best fits which type of problem. I get the impression that it will come with experience, trial and error and trying different methods for the same problem for you to get the best fit.

Is this Course for You?

You probably know by now if this course is for you or not. For me, it gave a good introduction to each aspect, explaining concepts and principles clearly and to the point. The material we went over was extensive and laid the ground to help open up aspects of my analysis that was not possible before. You still have to develop and learn more in these areas, but I have example code and principles under my belt. It treated you like an adult and appreciated we all have busy lives yet also pushed you. If any of the above interests you, I’d very much recommend it.

Anyway, over to my project. We had to do a project independently at the end, preferably using our own domain knowledge. I chose A&E attendance (not admissions), which has always been a topical issue within the health ecosystem and represents the interface between primary and secondary care.

My profile project - Predicting A&E Attendance from Primary Care Data

This is a link to the project with the source code. I was unable to upload patient data even though it was anonymised as it is from my surgery only.

This is a link to the jupyter notebook used for the project if you’d like to go over the code base

This project investigates factors predicting A&E (Accident & Emergency) attendance in the UK using a dataset of 4,495 patients from a single practice. It includes demographic, health, and deprivation data alongside A&E attendance records over three years. The aim is to identify links between primary care factors (e.g., long-term conditions, deprivation index) and A&E visits with the aim of improving resource allocation and preventive care. If we know the risk of a patient in front of us in the clinic, we can manage them more effectively. Mental health and access to primary care are highlighted for future exploration. The dataset is anonymised to comply with data protection laws, focusing on avoiding stigmatisation or over-generalisation in its application.

The dataset was created

by South West London BI service, who securely emailed attendances to A&E over the last 3 years of patients from my surgery only
A data analysis tool I use in my surgery to risk stratify and link population groups to deprivation and population health metrics.

From these two files, I had to cleanse the data, manage anomalies and outliers, and ensure the information was mapped. I also anonymised the data source here. This resulted in a CSV with the following (inputs followed by outputs). I partially used pandas to help.

Inputs (data required to decide on attendance)

Age, Is BAME, Conditions, Condition Count, Longterm Conditions, Longterm Condition Count, AF, ASTHMA, BP, CANCER, CHD, CHOLESTEROL, CKD, COPD, DEMENTIA, DM, HF, MH, NDH, PAD, PALLIATIVE, PLD, STROKE, THYROID, WEIGHTMX, Deprivation, Index, Health Index, PHM Level, All RST Cond. + PHM Level, All RST Cond. Level, Is Housebound

Outputs (if that patient went to A+E and how often did they attend)

A+E Frequency A+E Attendance

After importing the cleaned dataset, the project was divided into three sections.
1. Exploratory Data Analysis
2. Logistical Regression to visualise relationships for binary classification and see which features influence A&E Attendance
3. Random Forests to establish if there is a non-linear relationship between input and outputs to see how much each factor contributes to A&E Visits

I settled on Random Forest with Oversampling. This gave me the best outcome on the most important aspect, which is predicting A+E Attendance.

Exploratory Data Analysis

The first task is to determine if there is any correlation between predictors/inputs and A&E attendance; otherwise, you wouldn’t get a good model if you used all the values in your predictions. I used a heatmap for this initially

It looks like patients with more long-term conditions (0.12) and a low deprivation index (-0.08) tend to visit A&E more, but this is weak and might be diluted by the list size.

Let’s see which categories have the strongest relationship with A&E attendance, that is, whether they attend or not.

So we are in an exploratory phase here, slicing and dicing the data until we end up with something which makes sense. This is why domain expert knowledge is important when analysing the data. For example, in the above table, why should patients who have high cholesterol be a predictor for A&E Attendance? I put it down to a spurious reading as it doesn’t make sense, but it might be something to explore further, especially if we find the same if we analyse other surgeries.

The most interesting analysis of this phase was the following

It looks like being housebound and having co-morbidities are the top two predictors of the frequency of A&E admissions.Also, Mental Health is high for frequency, which goes in tandem with High Intensity Users projects nationally.

Based on the above, the following predictors were chosen to run a Logistic Regression on as these values from the data make sense to be areas to focus on and are backed up with evidence elsewhere too

Housebound
Mental Health
AF
COPD
HF
Health Index
Number of Long-Term Conditions
Age

After a bit of further analysis, overfitting, threshold adjustment to 0.3, and trial for SVG Boost, which didn’t work out, I settled on the following confusion matrix. See this link for more details.

So what does this all mean? Well, if I have a spreadsheet of patients with the following columns as yes/no (Housebound, Mental Health, AF, COPD, HF, Deprivation Health Index) with their age and number of long-term conditions, we can push them through the model, then can predict with 69% accuracy if they are at risk of attending A&E. However there is also a 40% chance that the model will pick a patient who it thinks are at risk of attending when they are not (False Positives or worried well)

From my point of view, it’s a start, but there is so much more to determining if patients attend A&E rather than get admitted to the hospital. Factors such as access to primary care and perception and trust of the patient with their own family doctor are important, too. So, to improve on this figure, we’ll need data from other surgeries and more data around access and trust to see if we can improve on true positives and reduce the false positives further.

What use would this have in the real world? Currently, we tend to triage each patient the same way when they come through the front door. If we knew their risk of attending A+E, we could prioritise them to avoid attendance, which might ultimately lead to admission.

The AI Bandwagon

The new shiny object not only in health care but in all areas of software development is AI. As usual, health care in the UK is behind the game. Sometimes, I feel we are joining the AI Bandwagon to head west to find gold in "them their hills” when we need to see where it best fits. Outside the NHS we are talking about AI Agents which are autonomous bots who self reason to get the job done and in the NHS we are still thinking about the risks adopting basic LLM integration in the form of AI Scribes being held back by clinical safety and IG. It is also more due to the reluctance to uptake the ubiquitous digital space in our world. I love AI Scribes, and they are a good example of a use case that showcases how AI works well. However, we are also in a position where there are so many AI scribes out there, all doing similar things. Ironically, what will make them sink or swim isn’t the product; it’s the marketing, pitching, and selling.

Just like when we went to telephone consultations from face to face only, everyone was in an uproar around how unsafe it was, and now we are in a situation where we are also conversing with our patients via text or emails, and it’s the same with AI. We all have a reluctance to adopt new technologies and need it to be forced on us or to develop trust that the new system will work, and this will take time, as it always does with the adoption of new digital technologies to the end user in health care. Ironically, the main accelerator of digital adoption in the last few years has been COVID-19, which has nothing to do with digital perceptions.

There is always a perception that you slap the word .ai at the end, and it will sell. As you can appreciate from this blog, AI and ML are not just self-driving cars and AI scribes. There is a lot more to how we can use this technology, and it’s also been around for longer than you think with established best practices. As always, it’s important to be pragmatic and use the right technology for the right job, whether it’s patient-facing or data analysis within confined projects with outcomes which actually make sense.

Summary

I hope this blog gives some indication of the power and usefulness of AI and ML as a tool and my experience from the Professional Certification Course I attended. The question you should ask, and I do frequently, is, why bother doing this? There are bigger fish in the sea who know more and, more importantly, can spend more time than we ever could as doctors creating better models with more experience, and that is true. If you are a medic and have read this far, you have to ask yourself if you want to stick to your domain expertise and let others do a better job or dip your toes into their world to help you interface better with experts in the field. I have no idea where this certification will take me, but I do know I’ll enjoy whatever I do, and that’s the most important thing!

Raza Toosy