Interact with me: Joint Egocentric Forecasting of Intent to Interact, Attitude and Social Actions
From one second of egocentric video, can an agent identify who will engage, whether the attitude is positive or negative, and what social action comes next?
01 / Problem
What is this work trying to solve?
A social agent should prepare before an interaction begins, including when several people are visible. Existing work usually studies intent, attitude, or action separately, and the available JPL dataset originally labels only the completed action of the interacting person rather than all visible people.
02 / Idea
What is the central technical intuition?
SocialEgoNet runs separate graph convolutional networks over face, body, and hand keypoints, fuses the spatial cues with multi-head attention, and uses a bidirectional LSTM for the 30-frame sequence. A chained classifier passes information from intent to attitude to action. The accompanying JPL-Social annotations expand 200 videos into 290 person-level tracks with labels for all three tasks.
03 / Evidence
What does the current evidence show?
SocialEgoNet reports 88.10% intent accuracy, 91.11% attitude accuracy, and 70.24% action accuracy, averaging 83.15%. Its 3.18M-parameter task model is much smaller than MS-G3D's 12.82M and has lower post-keypoint-processing latency, although MS-G3D retains the highest average accuracy at 85.08% versus 83.15%.
04 / Limits
Where are the limits of this work?
JPL-Social contains only 200 laboratory videos from 16 subjects across six settings, so in-the-wild generalization remains untested. The reported network latency is measured after whole-body pose extraction; end-to-end speed and accuracy still depend on person tracking and keypoint quality.