Publications

Workshop2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention

Yiming Ma, Victor Sanchez, Soodeh Nikan, Devesh Upadhyay, Bhushan Atote, Tanaya Guha

Overview figure from the paper “Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention”

TL;DR

Multi-sensor driver monitoring systems fuse their data at the decision level and may not fully exploit how the sensors complement each other; fusing views and modalities at the feature level with multi-head self-attention reaches 97.0% AUC-ROC, ahead of all baselines, and training with patch masking makes the system robust when a view or modality fails.

Abstract

Driver Monitoring Systems (DMSs) are crucial for safe hand-over actions in Level-2+ self-driving vehicles. State-of-the-art DMSs leverage multiple sensors mounted at different locations to monitor the driver and the vehicle’s interior scene and employ decision-level fusion to integrate these heterogenous data. However, this fusion method may not fully utilize the complementarity of different data sources and may overlook their relative importance. To address these limitations, we propose a novel multi-view multimodal driver monitoring system based on feature-level fusion through multi-head self-attention (MHSA). We demonstrate its effectiveness by comparing it against four alternative fusion strategies (Sum, Conv, SE, and AFF). We also present a novel GPU-friendly supervised contrastive learning framework SuMoCo to learn better representations. Furthermore, We fine-grained the test split of the DAD dataset to enable the multi-class recognition of drivers’ activities. Experiments on this enhanced database demonstrate that 1) the proposed MHSA-based fusion method (AUC-ROC: 97.0%) outperforms all baselines and previous approaches, and 2) training MHSA with patch masking can improve its robustness against modality/view collapses. The code and annotations are publicly available.

Cite

@INPROCEEDINGS{10208640,
  author={Ma, Yiming and Sanchez, Victor and Nikan, Soodeh and Upadhyay, Devesh and Atote, Bhushan and Guha, Tanaya},
  booktitle={2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  title={Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention},
  year={2023},
  pages={2617-2625},
  doi={10.1109/CVPRW59228.2023.00260}}