Predictive Model for Nursing Home Dependency Loses Accuracy Outside VA Data – Bioengineer.org

Predictive Model for Nursing Home Dependency Loses Accuracy Outside VA Data  Bioengineer.org

A machine learning tool designed to predict future dependency in daily activities among nursing home residents has been put to one of the toughest tests in clinical artificial intelligence: external validation in a completely different population. The Computational Barthel Index Tool, or CBIT, was originally built from data on U.S. Department of Veterans Affairs patients and can forecast declines in activities of daily living up to one year ahead. In a new study, researchers examined how well the pretrained models performed on a large, non-VA nursing home cohort, and whether adapting them to the new population could restore their accuracy.The validation relied on SEER-Medicare linked data covering 843,836 patients and more than 8 million Minimum Data Set assessments.

The team reconstructed the predictor and outcome structures of the original CBIT models within this cohort and then evaluated the unmodified VA-trained models for discrimination and calibration at four prediction horizons: current status, 3 months, 6 months, and 12 months.Under strict external validation, discrimination dropped substantially compared with the original VA models, and the decline was most pronounced for longer-term predictions. Mean AUC for the evaluation models fell from 0.795 in the original models to 0.693 at current prediction, 0.595 at 3 months, 0.595 at 6 months, and 0.592 at 12 months. Re-evaluation models retained stronger performance but showed a similar downward