Multi-View Multimodal Driver Monitoring Systems

In this project, we developed a multi-view multimodal driver monitoring system (DMS) using masked multi-head self-attention. The system aims to detect drivers’ actions that are irrelevant to driving, such as drinking and texting. Our DMS (called MHSA) achieves the state-of-the-art performance on the DAD dataset, and it is also robust against view/modality collapse. For more details, please refer to our paper Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention.

We have also designed a real-time DMS through assessing how informative each view or modality is and reached the conclusion that the top view infrared modality is more helpful than any other view/modality in DMS. The paper Real-Time Driver Monitoring Systems through Modality and View Analysis provides more details.