The PRIORI project is inspired by the power of artificial intelligence and personal technology to monitor the human condition and augment health. Health changes do not occur in a vacuum. Time is a critical factor. In advance of experiencing episodes of ill health, individuals’ behaviors change. When these changes are visible to the individual or to friends, family, and caregivers, there is an opportunity for intervention. But often, these changes occur outside of familiar observers or lack external evidence. Our team is developing methods to observe and quantify critical personal and social changes.
In our preliminary work, supported by NIMH R34, NSF CAREER, and foundation grants and awards, we developed a smartphone mental health monitoring system consisting of:
1. A deployable smartphone app that captures audio from telephone conversations and securely stores and transmits the data and
2. Computational strategies to detect mood.
The initial data set included 51 individuals with bipolar disorder and nine healthy controls who used the phone as their primary device for up to one year (avg. 26 weeks). The PRIORI app recorded their side of every phone call that they made or received. We obtained measures of mood through weekly structured telephone interviews (‘assessment calls’) with study clinicians using the Hamilton Depression Scale and the Young Mania Rating Scale. We refer to all calls made or received outside of these structured interviews as ‘personal calls.’
Our researchers in the Electrical Engineering and Computer Science department analyzed the initial dataset and found that although we could predict the mood state of the participant when the person was talking to a clinician, the methods were not effective on personal calls. We realized that we needed to change how we were thinking about the problem.
We needed something to bridge the divide between speech acoustics, which vary at the millisecond timescale, and mood, which varies over days to weeks to months! Emotion was our bridge. We conducted an emotion recognition proof-of-concept study on the initial dataset. Our team annotated 13,611 segments of speech (6-8 seconds each) from 12 participants and developed accurate emotion recognition algorithms using these data. We used the algorithms’ predictions to differentiate between individuals at risk for suicide and healthy controls and to determine when interventions may be necessary to help the patient. We also transcribed the 13,611 segments and used the transcriptions to verify the accuracy of an automatic speech recognition (ASR) system. Finally, we used the ASR output to measure how changes in the patterns of language are associated with changes in mood.
Great progress has been made on the PRIORI project over the past year. We have shown that speech collected in clinical environments and in natural personal interactions can be measured to detect changes in mood symptom severity. We remain excited to continue to improve our approaches, partnering with our research participants and donors, to achieve our goal of improving care and quality of life for individuals with bipolar disorder.
