AI-model found to recognise patients' emotions in therapy session, including fleeting ones

An AI-driven system was found to be able to recognise facial expressions conveying emotions such as happiness and sadness, even fleeting ones, that researchers say could help support psychotherapy.

The AI system that the researchers used for the study was a freely available artificial neural network trained to detect six basic emotions - happiness, surprise, anger, disgust, sadness and fear - using more than 30,000 facial photos. Artificial neural networks are a type of machine learning, a sub-field of AI, and are built on the principles of connections in biological neural networks found in animal brains.

The researchers then made the AI model process and analysed more than 950 hours of video recordings of therapy sessions with 23 patients having borderline personality at the Center for Scientific Computing, University of Basel, Switzerland.

The international team compared the model-generated analyses with those of three trained therapists and found "a remarkable level of agreement".

They said that along with gauging the facial expressions of the patients as reliably as a trained therapist, the model was able to detect the most fleeting of emotions, displayed for less than a millisecond, such a brief smile or an expression of disgust.

The AI was thus more sensitive than therapists towards such momentary displays of emotions, that could potentially be missed by therapists or may only be perceived subconsciously, the team said. They have published their findings in the journal Psychopathology.

"We wanted to find out whether AI systems can reliably determine the emotional states of patients in video recordings," said Martin Steppan, psychologist at the Faculty of Psychology at the University of Basel and corresponding author of the study.

The researchers further found that the model's analysis uncovered another trend - patients exhibiting emotional involvement by smiling at the start of a therapy session stuck to psychotherapy more than those who seemed emotionally indifferent towards their therapist.

The smile could therefore be a "good predictor" of the success of therapy sessions in people having borderline personality disorder, they said.

"We were really surprised to find that relatively simple AI systems can allocate facial expressions to their emotional states so reliably," said Steppan.

AI could thus become an important tool in therapy and research and could help support the supervision of psychotherapists, the team said, even as they added that currently "therapeutic work is still primarily about human relationships, and remains a human domain."