Automatic Smile and Frown Recognition with Kinetic Earables

In this paper, we introduce inertial signals obtained from an ear- able placed in the ear canal as a new compelling sensing modality for recognising two key facial expressions: smile and frown. Bor- rowing principles from Facial Action Coding Systems (FACS), we first demonstrate that an inertial measurement unit of an earable can capture facial muscle deformation activated by a set of tem- poral micro-expressions. Building on these observations, we then present three different learning schemes - shallow learning with statistical features, hidden Markov model, and deep neural net- works to accurately recognise smile and frown expressions from inertial signals. The experimental results show that in controlled non-conversational settings, we can identify smile and frown across a diverse user population with high accuracy (F1 score: 0.88).