Traditional electroencephalograph(EEG)-based emo-tion recognition requires a large number of calibration samples to build a model for a specific subject,which restricts the application of the affective brain computer interface(BCI)in practice.We attempt to use the multi-modal data from the past session to realize emotion recognition in the case of a small amount of calibration samples.To solve this problem,we propose a multi-modal domain adaptive variational autoencoder(MMDA-VAE)method,which learns shared cross-domain latent representations of the multi-modal data.Our method builds a multi-modal variational autoencoder(MVAE)to project the data of multiple modalities into a common space.Through adversarial learning and cycle-consistency regularization,our method can reduce the distribution difference of each domain on the shared latent representation layer and realize the transfer of knowledge.Extensive experiments are conducted on two public datasets,SEED and SEED-Ⅳ,and the results show the superiority of our proposed method.Our work can effectively improve the performance of emotion recognition with a small amount of labelled multi-modal data.

Yixin Wang;Shuang Qiu;Dan Li;Changde Du;Bao-Liang Lu;Huiguang He

Research Center for Brain-inspired Intelligence,National Laboratory of Pattern Recognition,Institute of Automation,Chinese Academy of Science,Beijing 100190;University of Chinese Academy of Sciences,Beijing 100049;Beijing Institute of Control and Electronic Technology,Beijing 100038,China;School of Mathematics and Information Sciences,Yantai University,Yantai 264003,China;Department of Computer Science and Engineering,Shanghai Jiao Tong University,Shanghai 200240,China;Center for Excellence in Brain Science and Intelligence Technology,Chinese Academy of Science,Beijing,China

自动化学报（英文版）

[1]Yixin Wang;Shuang Qiu;Dan Li;Changde Du;Bao-Liang Lu;Huiguang He-.Multi-Modal Domain Adaptation Variational Auto-encoder for EEG-Based Emotion Recognition)[J].自动化学报（英文版）,2022(09):1612-1626

electroencephalograph,MMDA

Multi,Modal,Domain,Adaptation,Variational,Auto,EEG,Based,Emotion,Recognition,Traditional,recognition,requires,large,number,calibration,samples,model,specific,subject,which,restricts,application,affective,brain,computer,interface,BCI,practice,We,attempt,use,from,past,session,realize,emotion,case,small,amount,To,solve,this,problem,we,domain,adaptive,variational,autoencoder,method,learns,shared,cross,latent,representations,Our,builds,MVAE,project,multiple,modalities,into,common,space,Through,adversarial,learning,cycle,consistency,regularization,our,can,reduce,distribution,difference,each,layer,transfer,knowledge,Extensive,experiments,conducted,two,public,datasets,SEED,results,show,superiority,proposed,work,effectively,improve,performance,labelled

0.594555