A timely book containing foundations and current research directions on emotion recognition by facial expression, voice, gesture and biopotential signals

This book provides a comprehensive examination of the research methodology of different modalities of emotion recognition. Key topics of discussion include facial expression, voice and biopotential signal-based emotion recognition. Special emphasis is given to feature selection, feature reduction, classifier design and multi-modal fusion to improve performance of emotion-classifiers.

Written by several experts, the book includes several tools and techniques, including dynamic Bayesian networks, neural nets, hidden Markov model, rough sets, type-2 fuzzy sets, support vector machines and their applications in emotion recognition by different modalities. The book ends with a discussion on emotion recognition in automotive fields to determine stress and anger of the drivers, responsible for degradation of their performance and driving-ability.

There is an increasing demand of emotion recognition in diverse fields, including psycho-therapy, bio-medicine and security in government, public and private agencies. The importance of emotion recognition has been given priority by industries including Hewlett Packard in the design and development of the next generation human-computer interface (HCI) systems.

Emotion Recognition: A Pattern Analysis Approach would be of great interest to researchers, graduate students and practitioners, as the book

• Offers both foundations and advances on emotion recognition in a single volume
• Provides a thorough and insightful introduction to the subject by utilizing computational tools of diverse domains
• Inspires young researchers to prepare themselves for their own research
• Demonstrates direction of future research through new technologies, such as Microsoft Kinect, EEG systems etc.

Amit Konar is a Professor of Electronics and Tele-Communication Engineering, Jadavpur University, India, where he offers graduate-level courses on Artificial Intelligence and directs research in Cognitive Science, Robotics and Human-Computer Interfaces. Dr. Konar is the recipient of many prestigious grants and awards and is an author of 10 books and over 350 research publications. He offered consultancy services to Government and private industries. He served editorial services to many journals, including IEEE Transactions on Systems, Man and Cybernetics (Part-A) and IEEE Transactions on Fuzzy Systems.

Aruna Chakraborty is an Associate Professor with the Department of Computer Science and Engineering, St. Thomas' College of Engineering and Technology, India. She is also a Visiting Faculty with Jadavpur University, where she offers graduate-level courses on Intelligent Automation and Robotics, and Cognitive Science. Her research interest includes human-computer interfaces, emotional intelligence and reasoning with fuzzy logic.

Preface xix

Acknowledgments xxvii

Contributors xxix

1 Introduction to Emotion Recognition 1
Amit Konar, Anisha Halder, and Aruna Chakraborty

1.1 Basics of Pattern Recognition, 1

1.2 Emotion Detection as a Pattern Recognition Problem, 2

1.3 Feature Extraction, 3

1.4 Feature Reduction Techniques, 15

1.5 Emotion Classification, 17

1.6 Multimodal Emotion Recognition, 24

1.7 Stimulus Generation for Emotion Arousal, 24

1.8 Validation Techniques, 26

1.9 Summary, 27

References, 28

Author Biographies, 44

2 Exploiting Dynamic Dependencies Among Action Units for Spontaneous Facial Action Recognition 47
Yan Tong and Qiang Ji

2.1 Introduction, 48

2.2 Related Work, 49

2.3 Modeling the Semantic and Dynamic Relationships Among AUs With a DBN, 50

2.4 Experimental Results, 60

2.5 Conclusion, 64

References, 64

Author Biographies, 66

3 Facial Expressions: A Cross-Cultural Study 69
Chandrani Saha, Washef Ahmed, Soma Mitra, Debasis Mazumdar, and Sushmita Mitra

3.1 Introduction, 69

3.2 Extraction of Facial Regions and Ekman's Action Units, 71

3.3 Cultural Variation in Occurrence of Different AUs, 76

3.4 Classification Performance Considering Cultural Variability, 79

3.5 Conclusion, 84

References, 84

Author Biographies, 86

4 A Subject-Dependent Facial Expression Recognition System 89
Chuan-Yu Chang and Yan-Chiang Huang

4.1 Introduction, 89

4.2 Proposed Method, 91

4.3 Experiment Result, 103

4.4 Conclusion, 109

Acknowledgment, 110

References, 110

Author Biographies, 112

5 Facial Expression Recognition Using Independent Component Features and Hidden Markov Model 113
Md. Zia Uddin and Tae-Seong Kim

5.1 Introduction, 114

5.2 Methodology, 115

5.3 Experimental Results, 123

5.4 Conclusion, 125

Acknowledgments, 125

References, 126

Author Biographies, 127

6 Feature Selection for Facial Expression Based on Rough Set Theory 129
Yong Yang and Guoyin Wang

6.1 Introduction, 129

6.2 Feature Selection for Emotion Recognition Based on Rough Set Theory, 131

6.3 Experiment Results and Discussion, 137

6.4 Conclusion, 143

Acknowledgments, 143

References, 143

Author Biographies, 145

7 Emotion Recognition from Facial Expressions Using Type-2 Fuzzy Sets 147
Anisha Halder, Amit Konar, Aruna Chakraborty, and Atulya K. Nagar

7.1 Introduction, 148

7.2 Preliminaries on Type-2 Fuzzy Sets, 150

7.3 Uncertainty Management in Fuzzy-Space for Emotion Recognition, 152

7.4 Fuzzy Type-2 Membership Evaluation, 157

7.5 Experimental Details, 161

7.6 Performance Analysis, 167

7.7 Conclusion, 175

References, 176

Author Biographies, 180

8 Emotion Recognition from Non-frontal Facial Images 183
Wenming Zheng, Hao Tang, and Thomas S. Huang

8.1 Introduction, 184

8.2 A Brief Review of Automatic Emotional Expression Recognition, 187

8.3 Databases for Non-frontal Facial Emotion Recognition, 191

8.4 Recent Advances of Emotion Recognition from Non-Frontal Facial Images, 196

8.5 Discussions and Conclusions, 205

Acknowledgments, 206

References, 206

Author Biographies, 211

9 Maximum a Posteriori Based Fusion Method for Speech Emotion Recognition 215
Ling Cen, Zhu Liang Yu, and Wee Ser

9.1 Introduction, 216

9.2 Acoustic Feature Extraction for Emotion Recognition, 219

9.3 Proposed Map-Based Fusion Method, 223

9.4 Experiment, 229

9.5 Conclusion, 232

References, 232

Author Biographies, 234

10 Emotion Recognition in Naturalistic Speech and Language-A Survey 237
Felix Weninger, Martin W ollmer, and Björn Schuller

10.1 Introduction, 238

10.2 Tasks and Applications, 239

10.3 Implementation and Evaluation, 244

10.4 Challenges, 253

10.5 Conclusion and Outlook, 257

Acknowledgment, 259

References, 259

Author Biographies, 267

11 EEG-Based Emotion Recognition Using Advanced Signal Processing Techniques 269
Panagiotis C. Petrantonakis and Leontios J. Hadjileontiadis

11.1 Introduction, 270

11.2 Brain Activity and Emotions, 271

11.3 EEG-ER Systems: An Overview, 272

11.4 Emotion Elicitation, 273

11.5 Advanced Signal Processing in EEG-ER, 275

11.6 Concluding Remarks and Future Directions, 287