Mount Sinai Team Shows AI Can Detect Serious Neurologic Changes in Babies in the NICU Using Video Data Alone
Study findings could serve as a foundation for broader neuro-monitoring applications across intensive care units globally
A team of clinicians, scientists, and engineers at Mount Sinai trained a deep learning pose-recognition algorithm on video feeds of infants in the neonatal intensive care unit (NICU) to accurately track their movements and identify key neurologic metrics.
Findings from this new artificial intelligence (AI)-based tool, published November 11 in Lancet’s eClinicalMedicine, could lead to a minimally invasive, scalable method for continuous neurologic monitoring in NICUs, providing critical real-time insights into infant health that have not been possible before.
Every year, more than 300,000 newborns are admitted to NICUs across the United States. Infant alertness is considered the most sensitive piece of the neurologic exam, reflecting integrity throughout the central nervous system. Neurologic deterioration in NICUs can happen unexpectedly and has devastating consequences. However, unlike cardiorespiratory telemetry, which continuously monitors the heart and lung function of babies in the NICU, neurotelemetry has remained elusive in most NICUs despite decades of work in electroencephalography (EEG) and specialized neuro-NICUs. Neurologic status is evaluated intermittently, using physical exams that are imprecise and may miss subacute changes.
The Mount Sinai team hypothesized that a computer vision method to track infant movement could predict neurologic changes in the NICU. “Pose AI” is a machine learning method that tracks anatomic landmarks from video data; it has revolutionized athletics and robotics.
The Mount Sinai team trained an AI algorithm on more than 16,938,000 seconds of video footage from a diverse group of 115 infants in the NICU at The Mount Sinai Hospital undergoing continuous video EEG monitoring. They demonstrated that Pose AI can accurately track infant landmarks from video data. They then used anatomic landmarks from the video data to predict two critical conditions—sedation and cerebral dysfunction—with high accuracy.
“Although many neonatal intensive care units contain video cameras, to date they do not apply deep learning to monitor patients,” said Felix Richter, MD, PhD, senior author of the paper and Instructor of Newborn Medicine in the Department of Pediatrics at Mount Sinai. “Our study shows that applying an AI algorithm to cameras that continuously monitor infants in the NICU is an effective way to detect neurologic changes early, potentially allowing for faster interventions and better outcomes.”
The research team was surprised by how well Pose AI worked across different lighting conditions (day vs. night vs. in babies receiving phototherapy) and from different angles. They were also surprised that their Pose AI movement index was associated with both gestational age and postnatal age.
“It’s important to note that this approach does not replace the physician and nursing assessments that are critical in the NICU. Rather, it augments these by providing a continuous readout that can then be acted on in a given clinical context,” explained Dr. Richter. “We envision a future system where cameras continuously monitor infants in the NICU, with AI providing a neuro-telemetry strip similar to heart rate or respiratory monitoring, with alert for changes in sedation levels or cerebral dysfunction. Clinicians could review videos and AI-generated insights when needed, offering an intuitive and easily interpretable tool for bedside care.”
The team noted the limitations of the study, including that the AI models were trained on data collected at a single institution, meaning that this algorithm and neurologic predictions need to be evaluated on video data from other institutions and video cameras. The research team plans to test this technology in additional NICUs and to develop clinical trials that will assess its impact on care. They are also exploring its application to other neurological conditions and expanding its use to adult populations.
“At Mount Sinai, we are committed to ensuring that new artificial intelligence possibilities are investigated and leveraged to advance care for our patients,” said Girish N. Nadkarni MD, MPH, System Chief of Data Driven and Digital Medicine, Director of the Mount Sinai Clinical Intelligence Center, Director of The Charles Bronfman Institute for Personalized Medicine and a study co-author. “AI tools are already advancing clinical care across the Mount Sinai Health System, including by shortening length of stay, reducing hospital readmissions, aiding in cancer diagnostics and therapeutic targeting, and delivering real-time care to patients based on physiological data generated from wearables, to name a few. We are excited to now be bringing this non-invasive, safe, and effective AI tool into the NICU to improve outcomes for our smallest, most fragile patients.”
About the Mount Sinai Health System
Mount Sinai Health System is one of the largest academic medical systems in the New York metro area, with 48,000 employees working across eight hospitals, more than 400 outpatient practices, more than 600 research and clinical labs, a school of nursing, and a leading school of medicine and graduate education. Mount Sinai advances health for all people, everywhere, by taking on the most complex health care challenges of our time—discovering and applying new scientific learning and knowledge; developing safer, more effective treatments; educating the next generation of medical leaders and innovators; and supporting local communities by delivering high-quality care to all who need it.
Through the integration of its hospitals, labs, and schools, Mount Sinai offers comprehensive health care solutions from birth through geriatrics, leveraging innovative approaches such as artificial intelligence and informatics while keeping patients’ medical and emotional needs at the center of all treatment. The Health System includes approximately 9,000 primary and specialty care physicians and 11 free-standing joint-venture centers throughout the five boroughs of New York City, Westchester, Long Island, and Florida. Hospitals within the System are consistently ranked by Newsweek’s® “The World’s Best Smart Hospitals, Best in State Hospitals, World Best Hospitals and Best Specialty Hospitals” and by U.S. News & World Report's® “Best Hospitals” and “Best Children’s Hospitals.” The Mount Sinai Hospital is on the U.S. News & World Report® “Best Hospitals” Honor Roll for 2024-2025.
For more information, visit https://www.mountsinai.org or find Mount Sinai on Facebook, Twitter and YouTube.