Since September 2022, Henry Fuchs, Danielle Szafir and others have collaborated with medical researchers at Carolina on the multifaceted Parkinson’s Project. Their PD-Insighter software combines graphics and data visualisation to allow clinicians treating Parkinson’s disease to watch and analyse patient movement in real-world home scenarios.
AR headsets and motion sensors track patient movements and interactions with their surroundings to provide the data.
“We could only capture this data with technology that you’re wearing every day,” said Fuchs, the Federico Gil Distinguished Professor of Computer Science. “So what is it that you’ll wear every day 10 years from now? We think it’ll be your glasses, which will have some cameras that could look out, look down and look in, so that they could capture your facial expression and capture your body and so on.”
The metaverse is the “next big leap” in tech, says Fuchs, who believes AR glasses will eventually replace phones the same way phones replaced laptops. AR headsets let Parkinson’s patients live their daily lives with minimal disruption, while also collecting data for clinicians to study.
“What do you do when you have all that data?” said Szafir, an assistant professor of computer science. “How do you make it intuitive? How do you help people leverage their own expertise so that you don’t have to be professional data scientists to make sense of all this information that’s flowing in?”
Doctoral research assistant Jade Kandel played a lead role fleshing out the PD-Insighter software and its desktop-based overview dashboard, which labels the patient’s actions throughout the day and highlights “freezes” or motor deficits that clinicians can sort through in minutes instead of hours.
The team also worked on an immersive replay component, which allows clinicians – through their own AR headsets — to replay a patient’s activities. The replay allows clinicians to view a 3D body skeleton of their patients and even see digital recreations of the patient’s environment, such as furniture the patient may use for support.
