Validation of a risk score to differentiate autoimmune and viral encephalitis: a Nationwide Cohort Study in Denmark

The study was carried out as a retrospective nationwide cohort study including all patients with a documented episode of definite viral or autoimmune encephalitis in Denmark from 2009 to 2023. A cohort of 119 patients with presumed infectious encephalitis (PIE) but no verified pathogen was additionally identified and assessed with the risk score.

All patients were identified from two nationwide databases:

The study was initiated in January 2023 by the Department of Infectious Diseases at Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark. Basic demographic and diagnostic information was obtained from the databases. Additionally, electronical patient records were reviewed by four different investigators to ensure the fulfillment of inclusion criteria of all cases as well as to extract the additional and necessary clinical and paraclinical information for performing the individual score-assessment.

Data extraction was completed by August 2023. Results are reported in accordance with the STARD-2015 guidelines on reporting of diagnostic accuracy studies [12].

A total of 641 patients in a consecutive series were screened for inclusion and 377 patients with AE (n = 90) and VE (n = 287) were enrolled in the study. We additionally assessed the score performance on a group of 119 patients with presumed infectious encephalitis, but no verified infectious or autoimmune aetiology (PIE). Inclusion criteria were identical with those in development study [11] (supplementary appendix I).

VE: Cases of VE were identified from the DASGIB Database. All enrolled cases of VE fulfilled the International Encephalitis Consortium (IEC) criteria for confirmed encephalitis and had a viral neurotropic pathogen verified in cerebrospinal fluid (CSF), either by direct PCR or with a positive pathogen-specific intrathecal antibody test [13] (supplementary appendix I).

AE: Cases of AE were identified from the NDAE. All enrolled cases of AE fulfilled the consensus criteria for AE and had a positive test for a disease-specific autoantibody in CSF and/or blood [9].

In this validation study, we aimed to assess the accuracy of the risk score for AE developed by Granillo et al.[11] (Table 1). The score ranges from 0 to 4, with higher scores indicating a higher risk of AE.

The Charlson Comorbidity Index (CCI) was established by information on age and comorbidities documented at admission[14]. Psychiatric complaints were defined as symptoms of true psychiatric character (hallucinations, delusions, mania, depression, or severe anxiety). Memory complaints were defined as new onset or exacerbated memory deficits as part of the acute disease. The terms “confusion” or “disorientation” were not considered a memory or a psychiatric complaint.

Onset of encephalitis was defined as the onset of new neurological symptoms and the duration of symptoms was defined as the number of days from onset to first hospital contact. The number of white blood cells (WBC) and the level of protein in CSF were recorded from the lumbar puncture performed at the primary admission for encephalitis.

If standard test results were not available or assessment of the risk score was not possible due to missing data, cases were excluded.

Data were analyzed using ‘R Studio’ 4.2.2 software [15]. The accuracy of the risk score for identification of AE cases was determined by calculating receiver-operating characteristics curve (ROC) and the area under the ROC curve (AUC) using the ‘pROC’ package in ‘R’. Confidence intervals for the AUC estimates were calculated by performing 1000 resampling-bootstraps using the ‘boot’ package in ‘R’. Chi-squared and Mann–Whitney U tests were performed to calculate the statistical difference between the score results of the different diagnostic groups. We chose a confidence interval of 95% and considered p values < 0.05 significant.

The DASGIB cohort was approved by the Danish Board of Health (3–3013-2579/1 and 3–3012-3168/1) and The Danish Data Protection Agency (2012–58-0018). Additional permission to access patient files was approved by the Regional Office for Journal Data (R-23004082). Approval for the AE cohort was similarly obtained from the Danish Data Protection Agency (3–3013-3124/1) and the Danish Board of Health and granted with a waiver for individual consent.

Risk-score values were significantly higher in the AE cohort compared to the VE cohort [median of 3 (IQR 2–3) vs median of 1 (IQR 0–1), p < 0.001)] (Fig. 2A). Score values were similar across aetiological subgroups in the AE cohort and only showed slight variations in the VE cohort (Fig. 2B). The risk score reached an AUC of 0.94 (95% CI 0.92–0.97) for predicting AE (Fig. 3A). The score performance in the VE cohort varied across aetiologies from AUC = 0.98 (95% CI 0.95–0.99, VZV) to AUC = 0.88 (95% CI 0.80–0.96, TBEV) (Fig. 3B). The score performance was high and only showed marginal variation across AE aetiologies (Fig. 3C). A score ≥ 3 resulted in a PPV of 87% and an NPV of 91% (Table 3). Patients < 18 years of age in the AE cohort (n = 13), had significantly higher score values than those ≥ 18 years (n = 77) [medians and (IQR): < 18 years: 3 (2,3) vs. ≥ 18 years: 3(3,4) p value < 0,001].

The performance of the score in the PIE cohort (Fig. 3B) was consistently high yet slightly inferior to the overall VE cohort reaching an AUC of 0.88 (95% CI 0.84–0.93), corresponding to a sensitivity of 69% and specificity of 93% with a score of ≥ 3 (supplementary appendix III). Slightly higher score values were observed for patients in the PIE cohort compared to the VE cohort, but the values were significantly lower than those in the AE cohort. The PIE cohort diverted most from the VE cohort by having a higher proportion of patients with a CCI < 2 (Table 4).

The score distribution in the AE, VE, and PIE cohorts (Table 4 and Fig. 4A) followed a Gaussian-like pattern with clear differences in approximated means. Based on score values, 8% of patients in the AE cohort and 85% of patients in the VE cohort were classified as ‘low AE risk’ (score value 0–1) (p < 0.001), whereas 23% in the AE cohort and 0% in the VE cohort were classified as ‘high AE risk’ (score value 4) (p < 0.001) (Table 4).

For all score elements, patients in the AE cohort had the highest fulfillment proportion (Table 4 and Fig. 4B). The relative difference in the proportion of patients with ‘subacute onset’ and ‘psychiatric/memory complaints’ was more pronounced compared to the remaining score variables (Fig. 5).

The NMDAR and AMPAR subgroups had less comorbidities than patients in the LGI1 and GABA-B subgroups and patients in the NMDAR subgroup more frequently had robust CSF inflammation than patients in the other subgroups. In the VE cohort, patients in the TBEV subgroup had less comorbidities and a higher frequency of subacute disease presentation. Patients in the HSV-2 and VZV subgroups had more comorbidities and a higher likelihood of robust CSF inflammation compared to patients in the HSV-1 group (Fig. 5).

The score reached an AUC of 0.94 for the prediction of AE in this Danish cohort of 377 encephalitis patients. A score ≥ 3 predicted AE with a PPV of 87% and an NPV of 91%.

The score performance was comparable to what was found in the development study, even though patients in our cohorts were slightly older than those in the development study (mean of 62.5 years ± 20.8 versus mean of 50.6 years ± 19.7) but had a similar sex distribution with 52% females. The VE cohort in this study was dominated by herpes viruses, whereas the original study included an approximately equal number of cases with herpes- and flaviviruses.

We found it reassuring that the performance of the score was not negatively affected by the higher age and CCI for patients in the LGI1 group and the higher rate of CSF inflammation in NMDAR.

While reaching high AUCs for all diagnostic subgroups included in this study, we did observe relatively better performance for herpes virus aetiologies compared to TBEV. This difference was driven by less comorbidity and a higher frequency of subacute presentation among patients in the TBEV group. Although the inter-aetiological variation in score performance in the VE cohort does raise a question of the translational potential of the score to settings with significantly different aetiological patterns, we expect that the score will perform well in a European setting where aetiologies for AE and VE are comparable[10, 16, 17]

Patients in the PIE group were younger, more often males and had a lower but still acceptable score performance compared to the VE cohort (AUC = 0.88 versus 0.94) translating into a sensitivity of 69% and specificity of 93% with a score of ≥ 3. The score distribution of the PIE cohort deviated most significantly from the VE cohort by a higher proportion with a CCI < 2, likely corresponding to the lower mean age. The somewhat inferior performance of the score in the PIE cohort could potentially be caused by undiagnosed or sero-negative cases of AE. Alternatively undiagnosed atypical infectious aetiologies with differing clinical and paraclinical presentations (e.g., the TBEV group) could also contribute to an impaired performance of the score.

AE cases differed most significantly from VE cases on the longer time from symptom onset to presentation and on the presence of psychiatric- and/or memory complaints, suggesting that these parameters are essential to differentiate between the two populations. This reflects findings in the previous studies[18, 19]. Also worth noticing is that these variables are likely more robust to future epidemiological transitions. As an example, the CCI item could be affected by a change toward older mean ages in future AE populations. Likewise, the absence of robust inflammation in CSF could be affected by an increasing proportion of immunosuppressed individuals in future VE populations [20].

We did not include additional parameters than those reported in the development study, since this was solely intended to be a validation study. However, it would be interesting to compose and test a modified risk score for AE by adding some of the parameters highlighted in recent observational studies where absence of fever, absence of hyponatremia, and the presence of movement disorders were identified as significant risk factors for AE [10, 19]. Some of these risk factors were also identified in the development study[11]. Inclusion of more variables could bring the secondary advantage of increasing the numeric value of the score and thereby potentially allowing for a more accurate risk stratification.

Even though the score could probably be improved by modifications, we believe that the consistently high performance in this external validation cohort should be sufficient to qualify the risk score for prospective testing.

Only allowing inclusion of patients with verified aetiologies of AE and VE may create a risk of partial verification bias and disease spectrum bias as described by Kennedy et al. potentially causing an overestimation of sensitivities and specificities[22]. The unblinded-retrospective study design causes an inherent risk of assessment bias. In this study, the risk of assessment bias was primarily related to the registration of psychiatric symptoms. We aimed at minimizing this risk by establishing criteria for the registration of psychiatric symptoms that were followed by all assessors (see supplementary appendix II). Arguably a prospective study with wider inclusion criteria would be ideal for testing the real-life performance of the score; however, such investigation would need to be performed as an extensive multi-center study with a long inclusion period given the low incidence of encephalitis. We believe that our external validation study is an essential stepping-stone to such large-scale prospective testing.