Researchers made a sonar-equipped earphone that can capture facial expressions

Researchers at Cornell University have developed an earphone that uses sonar to detect the wearer's facial expression to create an avatar of their face. The so-called "earable" system is called EarIO.

It works by bouncing sound off the wearer’s cheeks — the audio is emitted from speakers on each side of the earphone. A microphone captures the echoes, which change as the face moves and the wearer speaks. The system then uses a deep learning algorithm to turn the echoes into a replica of the person’s expression. EarIO can transmit the facial movements to a mobile device in real time and the avatar can be used in video calls.

Camera-based devices that track face movements are “large, heavy and energy-hungry, which is a big issue for wearables,” said Cheng Zhang, principal investigator of the Smart Computer Interfaces for Future Interactions Lab, who co-authored a paper on EarIO. “Also importantly, they capture a lot of private information.” A sonar-based approach can bolster privacy, affordability, comfort and battery life, he said.

In initial testing, the team found the device works while wearers are sitting and walking, and factors like background chatter, wind and ambient road noise don't impact the acoustic signaling. However, the high sensitivity of the sensing method can cause some issues. “It’s good, because it’s able to track very subtle movements, but it’s also bad because when something changes in the environment, or when your head moves slightly, we also capture that," said co-author Ruidong Zhang, an information science doctoral student. The researchers hope to mitigate such disruptions in future models.

EarIO has some limitations as things stand. The device runs for around three hours on a single charge despite being far more energy efficient than a camera-based system the team previously used. The researchers hope to improve the battery life in the future. They also aim to make EarIO a plug-and-play device but it currently needs 32 minutes of facial data training before the first use.