Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention
How should four heterogeneous video sources be fused, and can the same model remain useful when one or more sources fail?
01 / Problem
What is this work trying to solve?
Averaging final decisions from separate sensor branches discards the spatial-temporal relationships among sources and cannot learn which source matters for a particular action. At deployment, camera or modality collapse creates a second problem: a model trained only with all four sources may depend too strongly on any one of them.
02 / Idea
What is the central technical intuition?
Separate R3D-18 backbones encode the four synchronized sources. Their patches receive source and position embeddings before multi-head self-attention, so the fusion module can compare semantics across views, modalities, space, and time. SuMoCo supplies memory-efficient supervised contrastive training, while random patch masking acts as both regularization and a simulation of partial source collapse.
03 / Evidence
What does the current evidence show?
With all four sources, MHSA reaches 97.0% AUC-ROC and 97.8% mAP for binary non-driving-activity detection, and 91.6% mAP for fine-grained classification after unseen test activities are excluded. Across tests that remove one, two, or three sources, increasing the training mask ratio consistently reduces degradation; the 90% mask setting is the most robust.
04 / Limits
Where are the limits of this work?
The paper explicitly reports overfitting and severe class imbalance: normal driving is 84.7% of training data, and unseen activities are removed from the fine-grained evaluation. Missing sensors are simulated by clean source removal rather than real corruption; the authors also flag demographic imbalance and privacy risks from face-containing driver video.