Cross-Modal Approach for Conventional Well-being Monitoring with Multi-Sensory Earables

In this work, we propose a cross-modal approach for conversational wellbeing monitoring with a multi-sensory earable. It consists of three sensing models on earables, motion model, audio model, and BLE model. Using the IMU sensor, the microphone, and BLE scanning, the models detect speaking activities, stress and emotion, and participants in the conversation, respectively. We discuss the feasibility in qualifying conversations with our purpose- built cross-modal model in an energy-efficient and privacy-preserving way. In addition to these techniques, we present a mobile application that uses the inferred conversation qualifications to provide personalised feedback on social well-being to the end-users.