Hand gesture classification using early fusion based multimodal deep learning; 초기 융합 기반 다중 모드 딥 러닝을 사용한 손 제스처 분류

초록

In this paper, we propose a new hand gesture classification strategy using early fusion based multimodal deep learning. The structure and parameters of the state-of-the-art deep learning models such as ResNet152, DenseNet201, EfficientNetB0 for the source task of image classification are reused in the target task of hand gesture classification using surface electromyograph(EMG) and finger's kinematic data. The time-domain EMG and kinematic signals are normalized and then transformed into combined 2-D images for the early-fusion network. The experimental results support the superiority of the proposed method in terms of classification accuracy. The transfer learning model with the EfficientNetB0 shows the 93.94% accuracy for 40 gestures of 40 participants in the Ninapro DB2. © 2021 Korean Institute of Electrical Engineers. All rights reserved.

키워드