Natural language processing diagnosed behavioural disturbance phenotypes in the intensive care unit: characteristics, prevalence, trajectory, treatment, and outcomes

We performed a non-interventional, retrospective study of a cohort of critically ill adult patients (≥ 18 years old) admitted to the three ICUs of a university affiliated hospital in Melbourne, Australia between 1 May 2019 and 31 December 2020. The study was approved by the Austin Hospital Human Research Ethics Committee (LNR/19/Austin/38), which waived the requirement for informed consent. For patients with multiple admissions, only the first was included. No other exclusion criteria were applied. During the study period, all patients received care designed to reduce the risk of developing delirium including family visits, dimmed lights at night, minimal interaction to facilitate night-time sleep, and the use of visual and auditory aids as required.

We obtained the patient clinical progress notes entered into the electronic health record (EHR) by doctors, nurses, physiotherapists, and other allied health professionals. We analysed these notes using NLP techniques. As previously described [8, 15], each progress note was converted into sentence vectors, tokenised and searched for the presence of words indicative of agitated or non-agitated behaviour (Natural Language Toolkit; NLTK 3.5) [16], a process equivalent to the first step of Large Language Model Generative Pre-trained Transformer strategies recently popularised by OpenAI.

The words used in our NLP model were derived from the results of a previously published survey of clinical staff who were asked to identify words, terms or expressions that they would associate with disturbed behaviour and possible delirium [17].

In this study, we further categorised each of these words as being suggestive of agitated or a non-agitated behavioural disturbance state, while the presence of negation or resolution words was also determined (Table 1). Accordingly, all notes for the same patient were aggregated and categorised into four groups of NLP diagnosed behavioural disturbance (NLP-Ex-BD): (1) agitated, when only agitated words were present in available notes; (2) non-agitated, when only non-agitated words were present in available notes; (3) combined, when both agitated and non-agitated words were present in available notes; and (4) no disturbance, when no agitated or non-agitated words were present in any note available.

We use the term NLP-Dx-BD instead of delirium for the sake of accuracy. Importantly, we wish to emphasise that the words in Table 1 are intuitively associated with cognitive deterioration, which is recognised with delirium as well. However, we are also aware that we do not have sufficient data, for example, to exclude the fact that agitated behaviour was not, in fact, due (in a very few individuals) to underlying severe pain and/or dementia.

Baseline and outcome data were obtained from the Australian and New Zealand Intensive Care Society Adult ICU Patient Database run by the Centre for Outcome and Resource Evaluation [18]. Data detailing the use of antipsychotic medications were obtained from the hospital’s electronic medication management system.

The primary exposure of this study for a given patient was the occurrence at any time of having a word/s indicating agitated or non-agitated behaviour, or combined behavioural disturbance, within the notes recorded during their ICU stay.

The primary outcome of this study was the use of antipsychotic medications.

Although such treatment is controversial [19‐22], antipsychotic medication use was chosen as the primary outcome measure because we were studying behaviour and considered it likely that such medications would be used differently according to the presence or absence of an agitated behavioural phenotype [23, 24].

Secondary outcomes included ICU and hospital mortality and 28-day mortality censored at hospital discharge as well as ICU and hospital length of stay and duration of mechanical ventilation.

All continuous data are reported as median (quartile 25%–quartile 75%) and categorical data as numbers and percentage. Baseline, clinical characteristics and outcomes of the patients were compared among the groups using the Fisher exact test and Kruskal–Wallis test.

Multivariable logistic regression models were used to assess the impact of the exposure on the use of antipsychotic medications and hospital clinical outcomes, according to each phenotype. The model was adjusted by age, type of admission, and by the Australian and New Zealand (ANZ) Risk of Death (ANZROD) after log transformation [25]. ANZROD is a local recalibration of the APACHE III score which adjusts for the persistent lower than expected mortality in ANZ and contains variables such as admission diagnosis and pre-existing comorbidities [25]. As previously demonstrated, ANZROD is an accurate outcome predictor and explains most of the mortality found in ICUs in Australia and New Zealand [26]. Effect estimates were reported as odds ratio (OR) with its 95% confidence interval (CI). To account for immortal time bias, we conducted a time-dependent Cox proportional hazard model for the primary outcome and hospital mortality that considered all measurements available in each note. For the primary outcome, only exposure variables happening before the first outcome (first time the patient received an antipsychotic) were included in the model to avoid exposures measured after the medication had been given. Due to pairwise comparisons, the significance level was adjusted using a Bonferroni method and a p < 0.01 was considered statistically significant. All analyses were case complete analyses and were conducted in R v.4.0.2 (R Foundation, Vienna, Austria) [27]

We studied 2931 patients, 79,807 progress notes, and 17,110,747 words. Among the study patients, 1436 (49.0%) were Natural Language Processing Diagnosed Behavioural Disturbance (NLP-Dx-BD) positive. Of the positive patients, 225 (15.7% of positive and 7.7% of all patients) were categorised into the agitated phenotype; 544 (38.0% of positive and 18.6% of all patients) into the non-agitated phenotype and 667 (46.4% of positive and 22.7% of all patients) into the combined phenotype (Additional file 1: Fig. S1). Patients categorised into the combined phenotype displayed the characteristics of the agitated and non-agitated phenotypes at the same or different times during their admission.

The words ‘agitated’, ‘agitation’ and ‘restraint’ were the most common words detected from the NLP-Dx-BD agitated word list (Additional file 1: Fig. S2a) and ‘delirium’, ‘confused’ and ‘fluctuating’ the most common words from the non-agitated word list (Additional file 1: Fig. S2b).

The baseline (first 24 h of ICU admission) characteristics of the study patients are summarised in Table 2, which shows multiple significant differences according to the presence of NLP-Dx-BD and of its three different phenotypes. Compared to patients with the agitated phenotype, those with the non-agitated phenotype were older, had higher APACHE III scores, different admission diagnoses, and higher serum creatinine levels. However, they were also less likely to receive invasive ventilation and vasopressor drugs.

Compared with agitated phenotype patients, those with a combined phenotype were older, had higher APACHE III scores, a different source of admission, and were more likely to receive vasopressor drugs and renal replacement therapy. Finally, compared with the non-agitated phenotype those with the combined phenotype were younger, had higher APACHE III scores, different types of admission, were more likely to have acute kidney injury, had different admission diagnoses and source of admission, and were more likely to receive mechanical ventilation, vasopressor drugs and renal replacement therapy. A pairwise comparison of these three different phenotypes is presented in Additional file 1: Table S1, which confirmed the findings of Table 2 and identified the relationship between phenotypes.

Most episodes of NLP-Dx-BD occurred in the first 2 days (Additional file 1: Fig. S3). Thus, we studied the trajectory of NLP-Dx-BD phenotypes within these first 48 h among patients who, on day one or day two, had developed NLP-Dx-BD (Fig. 1, Additional file 1: Table S6). We found that behavioural phenotypes within this population changed dynamically.

More than half (59.55%) of the patients diagnosed with the combined phenotype of NLP-Dx-BD on the day of admission were negative (resolution) for NLP-Dx-BD by the second day. Approximately a third (31.46%) continued to display the combined phenotype and a small number transitioned to either the agitated (4.49%) or non-agitated (4.49%) phenotype. Patients with the non-agitated phenotype on the day of admission were similarly likely to achieve resolution of their NLP-Dx-BD (70.21%). However, approximately one third (17.02%) remained in a non-agitated phenotype state with a smaller number (12.77%) transitioning to the combined phenotype and no patients in the non-agitated phenotype transitioned to an agitated phenotype on day two. Finally, almost a half (45.72%) of the patients with the agitated phenotype on admission achieved resolution of NLP-Dx-BD, approximately one in two (51.42%) transitioned to the combined phenotype or remained agitated and only one (2.86%) became non-agitated.

The raw numbers for each antipsychotic drug used in each phenotype are presented in Table 3, which shows significant differences in their use according to phenotype. In the unadjusted pairwise comparison (Additional file 1: Table S2), antipsychotic drug use was significantly different only when the combined phenotype was the comparator. Thus, their use was significantly greater in the combined vs both other phenotypes (p < 0.001). This was also true for each individual agent (p < 0.001).

When analysing the use of antipsychotic drugs as outcome and considering all patients and groups as time-dependent variables (Additional file 1: Table S3), the univariable model showed that the odds ratio for receiving such drugs was greatest for the agitated phenotype. This was confirmed on multivariable modelling considering each phenotype as time dependent variable (Table 4) and was true for invasively ventilated and non-ventilated patients. Moreover, compared with patients without disturbance, the impact of NLP-Dx-BD with an agitated component (agitated or combined) on the prescription of anti-psychotic medications was strong (Additional file 1: Table S4).

The raw data for the secondary outcomes of the study patients are shown in Additional file 1: Table S4 and the unadjusted pairwise comparison is presented in Additional file 1: Table S5. The duration of ventilation, ICU length of stay and hospital length of stay were significantly greater for the combined phenotype vs. the agitated and non-agitated phenotype groups (p < 0.001). The unadjusted ICU mortality, hospital mortality and 28-day mortality rates were not significantly different across the phenotypes. However, once multivariable models were applied with hospital mortality as outcome and groups as time-dependent variables (Table 5), patients with the agitated phenotype were more likely to die than patients with other phenotypes (OR 1.57, 1.10–2.25, p = 0.012 vs. non-agitated phenotype; OR 4.61, 2.14–9.90, p < 0.001 vs. combined phenotype). This association was strongest in those patients receiving mechanical ventilation when compared with the combined phenotype (OR 7.03, 2.07–23.79, p = 0.002). A similar increased risk was also seen for patients with the non-agitated phenotype compared with the combined phenotype (OR 6.10, 1.80–20.64, p = 0.004).

We applied Natural Language Processing (NLP) to evaluate the characteristics, prevalence, trajectory, treatment, and outcomes of behavioural disturbance phenotypes in critically ill patients. We identified three major phenotypes: agitated, non-agitated, and combined. We found that each of these three phenotypes was associated with different patient characteristics. Of these, the combined phenotype was the most common, followed by the agitated phenotype. However, the trajectory of patients was such that movement from one phenotype to another in the first 48 h was common (65.66%). Moreover, during this time, it was uncommon for non-agitated patients to develop an agitated phenotype, many agitated patients achieved resolution of their NLP-Dx-BD, and both non-agitated and agitated phenotype patients typically transitioned to a combined phenotype. We also found significant differences in the use of antipsychotic medication. Thus, in time-variant multivariable models and after adjustment, patients with an agitated component to their NLP-Dx-BD were significantly more likely to receive antipsychotic medications overall compared with those patients with non-agitated NLP-Dx-BD. Finally, we found that patients with the combined phenotype had longer unadjusted duration of invasive ventilation, ICU stay, and hospital stay and greater mortality. However, once multivariable models were applied with hospital mortality as the outcome and groups as time-dependent variables, it was patients with the agitated phenotype who were more likely to die than patients with other phenotypes, especially among those patients receiving mechanical ventilation.

Antipsychotic medications remain widely used for the treatment of disturbed behaviour in critically ill patients [36, 37] with 70% of antipsychotic medication used in acute care prescribed for its treatment [38]. Further, several studies have reported higher rates of administration for agitated patients [23, 39]. In our study, we also found antipsychotic medication use was significantly more likely in the combined NLP-Dx-BD group followed by the agitated groups (both groups having an agitated component). This implies that it is the presence of agitation that drives antipsychotic drugs prescription.

Using intermittently applied structured-assessment tools, a previous study of non-ventilated patients suggested that mortality did not differ between the agitated, non-agitated and combined states [40]. However, in our study, after appropriate adjustments and assessing the group as a time-dependent variable (an adjustment not applied to previous studies), ICU mortality was highest for the NLP-Dx-BD agitated group. This suggests that NLP assessment of continuous caregiver observation may identify a cohort of patients that differs from that identified by intermittently applied structured-assessment tools.

Our findings imply that NLP-Dx-BD may be used to identify three clinically relevant phenotypes. Further, our findings suggest that the combined phenotype may be dominant. However, they also suggest that the early trajectory of these phenotypes is complex with dynamic changes from one phenotype to the other, indicating that such phenotypes are unstable. Moreover, on multivariable analysis and overall, patients with an agitated component to their NLP-Dx-BD (agitated phenotype or combined phenotype) appear significantly more likely to receive antipsychotic medications. These observations imply that future studies of pharmacologic intervention should primarily focus on patients who present an agitated component to their phenotype, either in isolation or in a combination (combined phenotype).

Our study has several strengths. It is the first to use NLP to study the prevalence of behavioural phenotypes in critically ill patients. It is also the first to describe the early trajectory of these phenotypes in critically ill patients and to demonstrate their dynamic and unstable nature. Moreover, to the best of our knowledge, our study presents the first detailed information of the use of antipsychotic medications according to phenotype, thus providing further evidence of the validity of the NLP-Dx-BD construct. Finally, with current software, as soon as a key word describing an agitated state is entered into the electronic notes, such entry can be used to trigger alerts to clinicians, thus facilitating randomisation into interventional trials.

We acknowledge several limitations. First, our study was undertaken in a large intensive care unit system involving three ICUs within a university affiliated tertiary hospital in a resource-rich country. Therefore, our findings may not apply to other intensive care units in low or middle-income countries or to other ICUs with a different approach to the management of disturbed behaviour. Moreover, patients were not assessed for the presence or absence of delirium by independent adjudication personnel. However, we investigated behavioural disturbances. This is different in focus, concept, and technique from the assessment of delirium [8]. Thus, we cannot comment of the relationship between our findings and their relevance to intermittently applied delirium screening tools. Future investigations of such relationship may be of interest. The use of medications in the treatment of behavioural disturbance in critically ill patients is controversial. However, antipsychotic medications remain an important tool for moderating behaviour, are a reasonable proxy for disturbed behaviour (the target of our investigation), and are widely used in critically ill patients [22, 41]. Finally, clinicians may be unfamiliar with NLP techniques, which might generate scepticisms about our findings. However, clinicians are the very generators of the words we used in our study; NLP-based technology is fast becoming accepted in response to the arrival of Large Language Model Generative Pre-trained Transformer strategies, and our study is the first step toward machine learning approaches to the behavioural management of critically ill patients.