Multi-lingual emotion detection in speech using hybrid neural network architectures

Abstract

The advancement of Deep Learning has revolutionised numerous fields, including humancomputer interaction (HCI) and emotion detection. In speech recognition, voice assistants such as Apple's Siri, Microsoft's Cortana, and Google's Assistant have become integral to everyday technology use. However, current emotion detection systems often need more support for African languages, presenting a significant research gap. The aim and objectives of the study are to develop a Speech Emotion Recognition (SER) system tailored to South African languages, with a specific focus on Afrikaans. The primary objective is to compile an Afrikaans speech corpus for emotion recognition. Furthermore, it explores hybrid neural network architectures combining CNN, Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks for improved accuracy. Finally, the aim is to investigate the optimisation of SER models for multilingual cross-language support. This study adopts a pragmatic research paradigm, utilising the Design Science Research Framework (DSR). Data is collected quantitatively. The literature review covers human speech, speech physiology, neural network architectures, speech corpus, and data extraction methods. Hybrid architectures and preprocessing techniques are examined to identify the most effective configurations. The Afrikaans speech corpus is sourced from Creative Commons (CC) sources, with data augmentation using Generative Adversarial Networks (GANs). The study successfully compiled an Afrikaans speech corpus. Synthetic speech samples were created using frameworks such as Tacotron 2 and the WaveNet vocoder, thereby enhancing sound quality. For the Hybrid Neural Network Architectures: The novel Dendritic Convolutional Long Short-Term Memory (DCLSTM) architecture outperformed traditional CNN and LSTM models, effectively capturing nuanced and long-term emotional dependencies in audio data. The second Kalman filter variant was identified as the optimal preprocessing technique for noisy speech signals. A final accuracy of 78.50% was achieved, improving on baseline CNN and LSTM models. For the Multilingual SER System, the DenCaps model achieved up to 71.91% accuracy on multilingual datasets, demonstrating improved generalisation and better capture of temporal dynamics and spatial hierarchies in emotional speech. The findings highlight the effectiveness of hybrid neural network architectures and advanced preprocessing techniques in improving SER accuracy. The development of the Afrikaans speech corpus and the introduction of the DenCaps model represent significant advancements in emotion detection. These results underscore the potential for creating robust, multilingual SER systems that cater to diverse linguistic needs. The research objectives were successfully achieved through the development of a specialised Afrikaans speech corpus, the exploration of innovative neural network architectures, and the optimisation of models for multilingual support. The academic contributions pave the way for more accurate, contextually aware human-computer interaction systems that address the research gap in emotion detection for African languages.