SkeletonVis: Interactive Visualization for Understanding Adversarial Attacks on Human Action Recognition Models

crown jewel figure
The interface of SkeletonVis, visualizing how the Fast Gradient Method manipulates the left foot joints detected by the Detectron2 Keypoint R-CNN model. (A) The Skeleton Views hows the joints perturbed to unexpected locations. (B) Timeline View reveals the attacked joints spuriously jumping around from one frame to the next, leading to a “spike” in the average joint displacement across attacked frames. These manipulations finally sway the ST-GCN action detection model into misclassifying the attacked frames as “exercising with exercise ball,” instead of the correct “lunge” classification.
Demo Video
Abstract
Skeleton-based human action recognition technologies are increasingly used in video based applications, such as home robotics, healthcare on aging population, and surveillance. However, such models are vulnerable to adversarial attacks, raising serious concerns for their use in safety-critical applications. To develop an effective defense against attacks, it is essential to understand how such attacks mislead the pose detection models into making incorrect predictions. We present SkeletonVis, the first interactive system that visualizes how the attacks work on the models to enhance human understanding of attacks.
Citation
SkeletonVis: Interactive Visualization for Understanding Adversarial Attacks on Human Action Recognition Models
@article{park2021skeletonvis,
  title={SkeletonVis: Interactive Visualization for Understanding Adversarial Attacks on Human Action Recognition Models},
  author={Park, Haekyu and Wang, Zijie J. and Das, Nilaksh and Paul, Anindya S. and Perumalla, Pruthvi and Zhou, Zhiyan and Chau, Duen Horng},
  booktitle={AAAI, Demo},
  year={2021}
}