Chapter 1 Introduction

1.1         Background

 Emotion recognition is a growing and rapidly advancing area of study that has great potential in a wide range of fields, such as computer-human interaction, psycho- logical monitoring, and customised user experiences. Conventional approaches to emotion recognition have mainly depended on external indicators like facial expres- sions, speech, and physiological signals. These methods utilize observable behaviors to infer emotional states, which, while useful, often face limitations in accuracy and reliability. Observable behaviors can be easily masked, altered, or influenced by external factors, leading to inaccuracies in emotion detection.

1.2         Limitations of Traditional Methods

Traditional emotion recognition techniques often rely on facial expressions, verbal sounds, and physiological signals such as heart rate and pores. While these methods have been extensively researched and applied, they possess inherent limitations:

1. Facial Expressions: Facial expressions can be consciously controlled or sup- pressed, making it difficult to accurately gauge genuine emotional states. Cul- tural differences also play a significant role in how emotions are expressed
2. Speech: Emotional cues in speech can be ambiguous and influenced by con- text, language, and individual speaking styles. Background noise and speech disorders can further complicate accurate emotion
3. Physiological Signals: Physiological responses such as heart rate variability and skin conductance are influenced by a variety of factors beyond emotional states, such as physical activity and environmental

1.3         Emotions

Emotion plays an essential role in promoting communication and interaction among individuals. It has significant effects on the brain, which subsequently affects various cognitive and physiological processes either directly or indirectly [3, 4, 5, 6]. Emo- tions are an essential component of human existence, influencing our experiences, choices, social connections, and overall state of being. Consequently, comprehend- ing and precisely identifying emotions holds significant significance for a wide range of applications in personal and professional settings.

1.3.1       Emotion Models

Characterizing emotions is a complex task, primarily due to the challenge of dividing the emotional space. Emotions can be categorized into two main models: discrete and dimensional. Each model offers a different perspective on how emotions are represented and understood

Discrete Model

In 1971, Paul Ekman proposed a distinct structure of emotion, which highlights the cross-cultural universality of specific emotions. Ekman’s research involved analyz- ing the relationship between human emotions and facial expressions among subjects from diverse cultural backgrounds [7]. He proposed the presence of a universal set of fundamental emotions, which are recognizable regardless of cultural differences. These basic emotions include:

• Surprise
• Pleasure
• Anger
• Fear
• Sadness
• Disgust

These emotions are distinguished from each other through various physiological the- ories and are universally experienced and expressed in similar ways. According

to the discrete model, at any given time, if a person is experiencing an emotional change, it can be categorized as one of these basic emotions.

Dimensional Model

In 1999, James A. Russell introduced the dimensional model of emotion, which is based on a cognitive theory [8]. Unlike the discrete model, which identifies specific emotions, the dimensional model represents emotions on a two-dimensional plane. This model generalizes emotions by mapping them according to two key dimen- sions: valence and arousal.

• Valence: This dimension measures the degree of positivity or negativity asso- ciated with an emotion. The spectrum spans from disagreeable (negative) to enjoyable (positive).
• Arousal: This dimension represents the degree of activation or intensity of an emotion. The range of emotional states varies from tranquilly (minimal arousal) to excitement (strong arousal).

The combination of these two dimensions creates a comprehensive space where all emotions can be positioned. For instance, emotions like excitement are characterized by high arousal and positive valence, while emotions like sadness are characterized by low arousal and negative valence.

Figure 1.1 illustrates the valence-arousal emotion model with its two-dimensional space. In this model, each emotion can be plotted according to its valence and arousal levels, allowing for a nuanced representation of emotional states. The fig-

Figure 1.1: Valence-Arousal Emotion Model [1]

Figure 1.2 illustrates a color wheel that visually represents various emotions with their essential impacts, further demonstrating the dimensional model’s application.

These models provide different frameworks for understanding and categorizing emotions, with the discrete model focusing on universally recognized basic emo- tions and the dimensional model offering a more flexible representation of emotional states. Both models are widely utilized in research and have important consequences for disciplines such as neuroscience, psychology, and artificial intelligence.

1.4         Affective Computing

In the past few decades, affective computing has become an important area of re- search, with the goal of improving communication between machines and human. 

Figure 1.2: Plutchik’s Wheel of Emotions [2] [9]. The main objective of this discipline is to create systems capable of identify- ing and reacting to human emotions, thus enhancing HCI. Due to advancements in HCI, it is now crucial for machines to understand and react to the emotional states of users. This ability is crucial for developing interfaces that are more intuitive and user-friendly.

1.4.1    Advancement and Significance

Affective computing aims to close the gap among human emotions and machine responses. The industry has experienced substantial expansion as a result of its capacity to enhance interactions by making them more authentic and effective. The capacity to perceive and comprehend emotions can have a substantial influence on

multiple areas, such as communication, planning, and learning.

1.5         Emotion Recognition

Emotion detection is a crucial element in the field of affective computing. This involves determining an individual’s emotional state by observing different signs that include facial expressions, voice pronunciation, body movements, and physiological signals. Out of these, physiological signals such as EEG are especially valuable because they are non-invasive and have a high temporal resolution [10, 11].

1.5.1    Emotion Detection Using EEG

The objective of this study is to develop a model that can accurately identify emo- tions by analysing EEG data. EEG signals provide a highly accurate assessment of brain activity, which makes them an outstanding tool for identifying emotional states. Through the analysis of these signals, it is feasible to categorise various emotions, which can subsequently be utilised to enhance the responsiveness and adaptability of human-computer interactions.

The Function and Placement of EEG Electrodes

In 1924, Hans Berger pioneered the development of EEG, a method that detects electrical activity in the brain. This activity, often referred to as the ”Berger wave,” represents the rhythmic alpha waves generated by neuron and synaptic excitations within the cerebral cortex. These electrical impulses are critical for understanding

the central mechanisms of brain function.

EEG works by measuring the electrical signals produced by neurons in the cerebral cortex. During an EEG recording session, electrodes are placed on the scalp following the internationally recognized 10-20 system, ensuring standardized and reliable data collection. This method enables the non-invasive monitoring of brain activity, making EEG a widely used and dependable technique for neurological analysis.

Recording Techniques: Invasive vs. Non-Invasive

EEG signals can be recorded using either invasive or non-invasive techniques. Inva- sive recordings involve placing electrodes directly on the cerebral cortex, providing high-precision data but requiring surgical procedures. Non-invasive recordings, on the other hand, capture brain activity from the scalp and are more commonly used due to their simplicity and safety

Frequency Bands of EEG Signals

The EEG signal encompasses a frequency range from 0.5 to 50 Hz. This range is divided into five distinct rhythms, each associated with different brain activities:

• Delta () Waves: 5 – 4 Hz, associated with deep sleep stages.
• Theta () Waves: 4 – 8 Hz, linked to light sleep, relaxation, and meditative
• Alpha () Waves: 8 – 13 Hz, indicative of relaxed wakefulness and
• Beta () Waves: 13 – 30 Hz, related to active thinking, problem-solving, and
• Gamma () Waves: 30 – 50 Hz, connected to high-level information processing and cognitive functioning

These frequency bands provide a comprehensive framework for analyzing EEG data, enabling the identification of various cognitive and neurological states

Figure 1.3: EEG Rhythm

1.6         Advantages of EEG-Based Emotion Recognition

EEG data offers a promising alternative by providing a direct assessment of brain activity. EEG is a technique that detects the electrical signals produced by the brain. It does this by placing electrodes on the scalp, without the need for surgery. This rich source of information captures the underlying neural processes associated with emotional states, allowing for a more precise and reliable identification of emotions. 

Table 1.1: Frequency bands of EEG Signal

EEG-based emotion recognition leverages the nuances of brain activity to classify emotions, bypassing the limitations of external indicators.

EEG-based methods have several advantages:

1. Direct Measurement: EEG captures electrical activity directly from the brain, providing a more immediate and less interpretable measure of emotional states compared to external
2. Non-Intrusive: Modern EEG systems are relatively non-intrusive and can be used in various settings without causing significant discomfort to the
3. Rich Data Source: EEG data provides a comprehensive view of brain activity, enabling the detection of subtle and complex patterns associated with different emotional states.

1.7         Importance of Accurate Emotion Recognition

The importance of accurate emotion recognition cannot be overstated. In human- computer interaction, understanding user emotions can lead to more intuitive and responsive interfaces, enhancing user satisfaction and engagement. For instance, adaptive systems can modify their responses based on the user’s emotional state, providing a more personalized experience. This can be particularly beneficial in educational technologies, where understanding student emotions can lead to better- tailored learning experiences, or in customer service, where systems can respond empathetically to user frustration.

In mental health monitoring, emotion recognition can offer objective measures for diagnosing and tracking emotional disorders, such as depression and anxiety. By providing real-time insights into a patient’s emotional state, healthcare providers can tailor interventions more effectively, improving treatment outcomes. Emotion recognition can also play a role in detecting early signs of emotional distress, en- abling timely intervention and support.

1.8         Challenges in EEG-Based Emotion Recognition

Despite its potential, EEG-based emotion recognition faces substantial challenges. Traditional machine learning models and simpler neural networks often struggle to capture the complex spatial and temporal patterns inherent in EEG signals. These models typically rely on manually engineered features and basic temporal analyses, leading to suboptimal accuracy and robustness.

Key challenges include:

1. Complex Patterns: EEG signals are characterized by their complex spatial and temporal patterns, which are influenced by various physiological and psy- chological factors. Capturing these patterns accurately is crucial for effective emotion recognition.
2. Feature Extraction: Manually engineering features from EEG data is a labor- intensive process that may not fully capture the richness of the Traditional methods often fall short in representing the intricate dynamics of brain activity.
3. Temporal Dependencies: Emotions are not static and evolve over time. Mod- els need to account for these temporal dependencies to accurately track and classify emotional states.
4. Data Variability: EEG signals can vary significantly between individuals due to differences in brain structure, physiology, and environmental Models must be robust enough to handle this variability.

1.9         Need for Advanced Models

Therefore, there is a need for advanced models that can automatically extract and integrate these intricate patterns to improve emotion recognition performance. Deep learning methods, especially CNN and RNN based approaches, present encouraging answers to these problems. While RNNs, particularly LSTM networks, are appro- priate for capturing temporal dependencies, CNNs are excellent at obtaining spatial features using EEG spectrograms.

By combining these architectures, it is possible to develop a model that effec- tively captures both the spatial and temporal patterns in EEG data, leading to more accurate and robust emotion recognition. Additionally, incorporating advanced acti- vation functions such as Swish can enhance model performance by improving gra- dient flow and convergence.

1.10         Motivation

The motivation behind this research stems from the significant potential of EEG- based emotion recognition to enhance various applications. In human-computer in- teraction, accurately recognizing users’ emotional states can lead to more intuitive and responsive interfaces. In mental health monitoring, it can provide objective mea- sures for diagnosing and tracking emotional disorders, offering valuable insights for therapeutic interventions. Additionally, personalized user experiences can be signif- icantly improved by adapting systems based on real-time emotional feedback.

The current landscape of emotion recognition using EEG data presents several challenges. Current models frequently lack the complexity necessary to fully rep- resent the complex temporal and spatial patterns found in EEG signals. Traditional machine learning approaches depend heavily on feature engineering, which may not fully represent the complexity of EEG data. Simpler neural networks, while au- tomating feature extraction, often fail to account for the temporal dependencies crit- ical for understanding emotional states over time. These limitations result in lower accuracy and limited generalizability of the models

To address these challenges, there is a pressing need for advanced models that can seamlessly integrate spatial and temporal information from EEG data. In order to close this gap, the proposed research will create a hybrid model which combines the advantages of RNNs and CNNs. This approach leverages CNNs for spatial fea- ture extraction from EEG spectrograms and RNNs, particularly LSTM networks, for capturing temporal dependencies. By doing so, the model aims to achieve higher accuracy and robustness in emotion recognition.

1.11         Objective

The main goal of this work is to combine CNN and RNN to create a reliable model for emotion recognition from EEG data. The goal of this hybrid model is to combine the best aspects of both architectures: RNNs, especially LSTM networks, for cap- turing temporal dependencies, and CNNs for extracting spatial features using EEG spectrograms. Furthermore, it is suggested to use Swish activation functions in order to improve the model’s performance.

The specific goals of this study are:

1. To preprocess and normalize EEG data, transforming it into a suitable format for model training.
2. To design and implement a hybrid CNN-RNN-Bi-LSTM-Attention model for emotion recognition.
3. To train and optimize the model using a comprehensive EEG dataset, ensuring high accuracy and robustness.
4. To evaluate the model’s performance against baseline models and analyze its strengths and limitations.

1.12         Overview of the Project

In this project, we have developed a novel approach to emotion recognition utilizing a hybrid model that integrates CNN and RNN. The model leverages the capabilities of CNNs to extract spatial features from EEG spectrograms, which are graphical representations of EEG signals over time. These spatial features capture the complex patterns inherent in EEG data.

Following the extraction of spatial features, the model employs LSTM layers to capture temporal dependencies. Because LSTM networks can maintain long- term dependencies and handle the vanishing gradient issue that traditional RNNs frequently face, they are particularly suitable for the purpose. By combining these two architectures, the model can effectively capture both spatial and temporal pat- terns in the EEG data, leading to more accurate emotion recognition.

Additionally, the model incorporates Swish activation functions to enhance per- formance. Swish, a self-gated activation function, has been shown to outperform traditional activation functions like ReLU in various deep learning tasks. Its smooth and non-monotonic properties contribute to improved gradient flow and model con- vergence, resulting in better performance.

We have implemented a comprehensive preprocessing pipeline to ensure the quality and uniformity of the input data. This involves normalizing the EEG signals,

generating spectrograms to capture the time-frequency representation, and mapping labels based on valence, arousal, and dominance ratings. These preprocessing steps are crucial for improving the reliability and accuracy of the emotion recognition process.

The hybrid CNN-RNN-Bi-LSTM-Attention model was trained on an extensive EEG dataset, and hyperparameters were optimized through cross-validation. The model was evaluated against baseline models, showing a significant improvement in accuracy and robustness. The results indicate that the model effectively captures complex patterns in EEG data, leading to a recognition accuracy of 92

The discussion highlights the model’s durability and its capacity for utilization in real-time monitoring of emotions and diagnosing mental health disorders. Fu- ture endeavors will focus on enhancing the model for immediate application and improving its overall efficiency.

1.13         Thesis Organization

This thesis is organized into six chapters, each detailing a critical aspect of the re- search:

• Chapter 1: Introduction: Provides an overview of the background, motiva- tion, and objectives of the study.
• Chapter 2: Literature Review: Reviews existing research on emotion recog- nition using EEG data, highlighting key studies and identifying research
• Chapter 3: Proposed Methodology: Describes the proposed hybrid model, including data preprocessing, model architecture, and training
• Chapter 4: Experimental Setup and Analysis: Details the dataset used, pre- processing steps, experimental setup, and analysis of
• Chapter 5: Results and Discussion: Presents the experimental results, com- pares them with baseline models, and discusses the implications and limita-
• Chapter 6: Conclusion and Future Work: Summarizes the findings, draws conclusions, and suggests directions for future

This structured approach ensures a comprehensive exploration of the research prob- lem, providing valuable insights into the potential of EEG-based emotion recogni- tion.

Chapter 2 Literature Review

2.1         Introduction

Emotion recognition using EEG signals has become a significant area of research, offering profound insights into the neural correlates of emotional states. This chap- ter reviews existing literature on EEG-based emotion recognition, highlighting key studies, methodologies, and findings.   The objective is to identify research gaps and establish a foundation for the proposed hybrid CNN-RNN-Bi-LSTM-Attention model.

2.2         Overview of EEG-Based Emotion Recognition

EEG-based emotion recognition leverages the electrical activity of the brain to iden- tify emotional states. This approach provides a direct and non-invasive method of assessing brain activity, making it a valuable tool for various applications, includ- ing mental health monitoring, human-computer interaction, and personalized user experiences.

2.3         Key Studies and Methodologies

2.3.1       Deep Learning Approaches

Algarni et al. (2022) explore a deep learning-based approach for emotion recogni- tion using Bi-LSTM networks. Their methodology involves the utilization of EEG signals to train the Bi-LSTM model, achieving high accuracy in detecting arousal, valence, and liking emotions. The study demonstrates the effectiveness of deep learning techniques in capturing complex temporal patterns in EEG data. The re- searchers highlight that traditional models often fail to account for the bidirectional nature of emotional experiences, where past and future contexts are equally impor- tant. By employing Bi-LSTM, they could effectively model these dependencies, leading to a significant improvement in recognition performance. The study also discusses the pre-processing steps involved, including normalization and artifact re- moval, which are crucial for ensuring the quality of the input EEG data [12].

Bazgir et al. (2018) present a robust emotion recognition system based on the valence/arousal model. The researchers employed advanced signal processing tech- niques, including DWT and PCA, to extract features from EEG signals. Their system demonstrated exceptional precision using classifiers such as SVM, KNN, and ANN. The use of DWT allowed for the decomposition of EEG signals into different fre- quency bands, capturing both time and frequency domain features. PCA helped in reducing the dimensionality of the feature space, minimizing computational complexity while retaining critical information. This combination of techniques enabled the system to achieve high accuracy and robustness in emotion classification [13].

Houssein et al. (2022) provide a comprehensive review of EEG-based emo- tion recognition methods, focusing on feature extraction and machine learning tech- niques. They highlight the potential of deep learning algorithms to address chal- lenges in emotion recognition, emphasizing the need for multi-channel EEG signal analysis to capture the complexities of emotional states. The review covers various deep learning architectures, including CNNs and RNNs, and their applications in emotion recognition. The authors discuss the strengths and limitations of different approaches, providing insights into how deep learning can automate feature extrac- tion and improve the representation of spatial and temporal patterns in EEG data. Their findings underscore the importance of integrating multiple channels and using sophisticated models to enhance recognition performance [14].

2.3.2       Feature Extraction and Classification

Liu et al. (2021) review methodologies for emotion recognition using EEG sig- nals, emphasizing the importance of feature extraction and classification techniques. They delve into a number of strategies, including modern deep learning architectures and conventional machine learning techniques, highlighting the necessity of reliable feature extraction to increase the accuracy of emotion recognition. The review high- lights that manually engineered features often fail to capture the full complexity of EEG signals. Instead, automated feature extraction methods that leverage the hier- archical structure of deep learning models can provide a more comprehensive representation of the data. The authors also explore different classification techniques, such as SVMs, decision trees, and neural networks, comparing their effectiveness in various emotion recognition tasks [15].

Gannouni et al. (2021) introduce novel methods for enhancing emotion recog- nition using EEG signals. Their research focuses on the adaptive selection of chan- nels and the identification of epochs during emotional states, showcasing promis- ing outcomes in identifying emotions across diverse categories. The authors pro- pose an adaptive channel selection method that dynamically selects the most rele- vant EEG channels for emotion recognition, thereby improving classification perfor- mance. Additionally, their epoch identification technique helps in isolating the most informative segments of EEG data, further enhancing the recognition accuracy. This approach addresses the challenge of data variability by focusing on the most sig- nificant features and reducing noise, leading to more accurate and reliable emotion classification [16].

2.3.3       Applications in Mental Health and Human-Computer Interaction

Al-Nafjan et al. (2017) classify human emotions from EEG signals using a deep neural network. Their study demonstrates the application of EEG-based emotion recognition in mental health monitoring, providing insights into emotional disorders and potential interventions. The authors highlight the potential of EEG-based sys- tems to offer objective measures for diagnosing and tracking emotional disorders, such as depression and anxiety. Their method, which uses deep neural networks to classify different emotional states with high accuracy, is a useful tool for mental

health experts. The study also discusses the practical implications of using EEG- based emotion recognition in clinical settings, including the potential for real-time monitoring and intervention [17].

Du et al. (2020) provide a technique for recognising emotions in EEG signals to evaluate activities. The study highlights the application of EEG-based emotion recognition in human-computer interaction, emphasizing its potential to enhance user experiences in gaming environments. The authors develop a system that cap- tures players’ emotional states in real-time, allowing game developers to adapt game dynamics based on the detected emotions. This approach not only improves user en- gagement but also provides valuable feedback for game design and development. The study also explores the technical challenges involved in implementing real-time emotion recognition, such as latency and computational efficiency, and proposes so- lutions to address these issues [18].

Mudgal et al. (2020) discuss the advancements in BCIs and their applications in neurosciences, including emotion recognition. They address the challenges and potential of BCIs in various fields, underscoring the significance of accurate emo- tion recognition for improving user interaction. The authors review different BCI systems and their applications, highlighting the role of EEG-based emotion recog- nition in enhancing the functionality and user experience of BCIs. Their discussion points to the need for more sophisticated models that can accurately decode emo- tions from EEG signals, providing a foundation for developing more intuitive and responsive BCIs. The study also considers the ethical and practical considerations of using BCIs for emotion recognition, such as privacy and user consent [19]

Table 2.1: Summary of Key Studies on EEG-Based Emotion Recognition

2.4         Challenges in EEG-Based Emotion Recognition

Despite significant advancements, EEG-based emotion recognition faces several chal- lenges. Accurate emotion recognition depends on being able to capture the complex temporal and spatial patterns observed in EEG signals. Traditional models often fail to represent these intricate dynamics adequately. Manually engineering features from EEG data is labour-intensive and may not fully capture the richness of the data. Advanced feature extraction techniques are needed to improve model performance.

Emotions evolve over time, necessitating models that can account for these temporal dependencies. RNNs, particularly LSTM networks, are well-suited for this task but require sophisticated training and optimization. Additionally, EEG signals vary significantly between individuals due to differences in brain structure, phys- iology, and environmental factors. Models must be robust enough to handle this variability to generalize well across different subjects.

2.5         Addressing the Challenges: Advanced Models and Techniques

To address these challenges, recent studies have proposed various advanced models and techniques. Combining CNNs and RNNs has shown promise in capturing both spatial and temporal patterns in EEG data. For instance, Li et al. (2016) developed a convolutional recurrent neural network for emotion recognition, demonstrating improved performance by leveraging both architectures [20].

A hybrid model of GCNs as well as LSTM for EEG emotion detection was proposed by Yin et al. (2020), demonstrating the possibility of fusing various neu- ral network architectures to improve model performance. Their approach leverages the strengths of GCNs in capturing spatial dependencies and LSTMs in modeling temporal dynamics, resulting in improved emotion recognition accuracy [21].

Integrating EEG data with other physiological signals can improve emotion recognition accuracy. Ranganathan et al. (2016) explored multimodal emotion recognition using deep learning architectures, demonstrating the benefits of com- bining EEG with other data sources. Their multimodal approach integrates EEG with physiological signals such as heart rate and skin conductance, providing a more comprehensive understanding of emotional states and improving classification per- formance [22].

Incorporating advanced activation functions such as Swish has been shown to enhance model performance. Swish improves gradient flow and convergence, lead- ing to better learning outcomes. Studies by Algarni et al. (2022) and Bazgir et al. (2018) have demonstrated the way Swish works to enhance deep learning models’ ability to recognise emotions [12, 13].

2.6         Gaps in Research and Future Directions

While significant progress has been made, several research gaps remain. Most stud- ies focus on offline analysis of EEG data, highlighting the need for models that can perform real-time emotion recognition with minimal latency. Addressing class imbalance in EEG datasets is crucial for improving model performance across all

emotion categories. Methods like weighted loss functions and data augmentation can help reduce this problem.

Ensuring that models generalize well across different subjects and conditions is a significant challenge. Future research should focus on developing robust mod- els that can handle variability in EEG signals. Combining EEG data with other physiological signals, such as heart rate and skin conductance, can provide a more comprehensive understanding of emotional states. Future studies should explore multimodal approaches to enhance emotion recognition accuracy.

Chapter 3

3.1         Overview

Current models for emotion recognition using EEG data frequently face numerous challenges, primarily because they are unable to fully harness the intricate and multi- faceted characteristics of EEG signals. Conventional machine learning models often rely on manually designed features, which might not accurately represent the com- plex spatial and temporal patterns inherent in EEG data. These manually engineered features typically include statistical measures such as mean, variance, and frequency domain features. While useful, these features may fail to capture subtle yet impor- tant patterns in the data.

Furthermore, less complex neural networks, although they automate the process of feature extraction, frequently neglect the temporal relationships that are essential for comprehending emotional states over time. Basic neural networks might learn from raw data but often lack the depth required to capture intricate spatial charac- teristics. Additionally, they may not possess the necessary structure to accurately

represent temporal dependencies, which are crucial for understanding how emotions evolve.

These constraints result in suboptimal performance, characterized by reduced accuracy and a limited ability to generalize to novel data. The inability to cap- ture the full spectrum of spatial and temporal features in EEG signals means that conventional models may overlook critical information needed for precise emotion recognition. Consequently, the performance of these models is often compromised, leading to inaccuracies in identifying emotional states and challenges in applying the models to new and diverse datasets.

In summary, the limitations of current models for emotion recognition using EEG data stem from their reliance on manually crafted features and the insufficient complexity of basic neural networks. These factors hinder the ability to fully exploit the rich, multi-dimensional nature of EEG signals, resulting in less accurate and less generalizable emotion recognition systems.

Emotion detection, commonly referred to as facial emotion recognition, is a captivating field within artificial intelligence and computer vision. It entails the identification and interpretation of human emotions based on facial expressions. The accuracy and efficacy of emotion detection have far-reaching applications in various domains, including human-computer interaction, customer feedback analysis, and mental health monitoring. CNNs and RNNs have emerged as transformative tools in this domain, significantly enhancing our ability to decode and interpret emotional cues from facial images.

3.2         Proposed Hybrid Model

In order to address these difficulties, we suggest employing a hybrid model that in- tegrates CNNs and RNNs. This approach will allow us to accurately record and analyse both the spatial and temporal characteristics of EEG data. The CNN module examines the spatial characteristics of EEG signal spectrograms by utilising its ca- pacity to independently obtain spatial hierarchies from the input data. CNNs exhibit exceptional efficiency in analysing data representations that resemble images, such as spectrograms, owing to their utilisation of convolutional filters. The filters in the network detect local patterns, including edges, textures, and complex shapes, as the network goes through its layers.

On the other hand, the LSTM layers in the RNN model are specifically de- signed to handle sequential data, allowing them to effectively capture the temporal relationships found in EEG signals. LSTMs are equipped with gating mechanisms that enable them to store information for long durations by regulating the flow of information. This makes them appropriate for analysing sequences that occur over time.

In addition, we utilise Swish activation functions to optimise the performance of the model by enhancing the flow of gradients and facilitating convergence during the training process. The Swish activation function has been demonstrated to surpass conventional activation functions such as ReLU by enabling a more seamless gradi- ent flow. Our proposed hybrid model combines the strengths of CNNs and RNN net- works. By using advanced activation functions, our model aims to provide accurate and reliable analysis of EEG data. It effectively handles the spatial and temporal complexities that are inherent in these signals.

3.3         CNN Model

The CNN model is a powerful deep learning architecture designed to categorize ob- jects into distinct classes based on their spatial properties [23]. CNNs are particularly effective for evaluating and processing image data, making them suitable for appli- cations such as image and video recognition, decision support, picture classification, segmentation, computer vision, text analysis, central nervous system linkages, and economic time series analysis.

3.3.1       Key Features and Structure of CNNs

Shared-Weight Design

CNNs utilize a shared-weight architecture, where the same weights (or filters) are applied across different parts of the input image. This design mimics the Fourier transform by scanning the input data with convolutional filters, capturing transla- tionally invariant features. The shared weights help reduce the number of param- eters and computational complexity, enabling CNNs to learn efficiently from large datasets

Multilayer Perceptrons (MLPs)

CNN variants include multilayer perceptrons, which are fully connected neural net- works where each neuron in one layer connects to every neuron in the next layer. Although MLPs are powerful, they often suffer from overfitting, especially with large input sizes like images. Regularization techniques such as dropout and weight decay are employed to mitigate this issue.

Feature Extraction with Convolutional Layers

The core component of a CNN is the convolutional layer, which consists of a se- ries of filters (or kernels) that slide over the input image to extract spatial features. Each filter is convolved with the input image to produce a feature map, highlighting specific patterns such as edges, textures, and more complex shapes. Convolutional layers enable the network to capture hierarchical features, from low-level details in initial layers to high-level abstractions in deeper layers.

Figure 3.1: Overview of the CNN architecture

Pooling Layers

 To reduce the spatial dimensions of feature maps and the number of parameters, pooling layers are introduced. Pooling layers perform down-sampling operations, such as max pooling or average pooling. Max pooling selects the maximum value from a group of activations, while average pooling computes the mean value. Pool- ing layers help to make the feature maps invariant to small translations and distor- tions in the input image.

Input Representation and Processing 

CNNs process input images represented as 3D tensors with dimensions correspond- ing to the number of samples, height, width, and channels (e.g., RGB channels for colour images). The convolutional layers apply filters to these input tensors, creating feature maps with reduced dimensions. The hyperparameters of the convolutional layers, such as filter size, stride, and padding, are carefully chosen to balance com- putational efficiency and feature extraction quality.

Fully Connected Layers and Activation Functions

After the convolutional and pooling layers, the network typically includes fully con- nected layers that integrate the extracted features to perform classification. These layers use activation functions such as ReLU to introduce non-linearity and im- prove the model’s ability to learn complex patterns. Regularization techniques, like dropout, are applied to prevent overfitting and improve generalization.

3.3.2       Convolutional Layer Mechanics

Receptive Field

Each convolutional neuron processes a small region of the input image, known as the receptive field. This localized processing allows CNNs to detect specific features within confined areas of the image, which are then combined across layers to form a comprehensive representation

Weight Sharing and Local Connectivity

By sharing weights across different regions of the input image, CNNs reduce the number of parameters and enhance learning efficiency. Local connectivity ensures that each neuron focuses on a specific spatial region, contributing to the hierarchical feature extraction process.

Filter and Feature Map Generation

Filters are small-sized matrices that slide over the input image, performing element- wise multiplication and summation to produce feature maps. These feature maps highlight the presence of specific patterns, such as edges or textures, within the input image

3.3.3       Pooling Layers

Max Pooling

Max pooling selects the maximum value from each region of the feature map, effec- tively reducing the dimensionality while retaining the most important features. This operation provides translation invariance and robustness to minor variations in the input image

Average Poolinng

Average pooling computes the average value within each region of the feature map, offering a smoother and more generalized feature representation. Although less commonly used than max pooling, average pooling can be beneficial in certain sce- narios.

3.3.4       Regularization Techniques

Dropout

Dropout is a regularization technique that randomly sets a fraction of neurons to zero during training, preventing the network from relying too heavily on specific neurons and promoting better generalization.

Weight Decay

Weight decay adds a penalty term to the loss function, proportional to the sum of the squared weights,

encouraging the network to learn smaller weights and reducing overfitting.

3.3.5       Advanced Architectures

FCNs

By adding convolutional layers in place of fully connected layers, FCNs expand CNNs’ capabilities to handle input images of any size and perform operations like image segmentation

Residual Networks (ResNets)

 ResNets introduce shortcut connections that bypass one or more layers, allowing gradients to flow more easily during training and enabling the construction of very deep networks without the vanishing gradient problem.

Global and Local Pooling

Global pooling computes a single value (e.g., max or average) across the entire fea- ture map, reducing the data to a single vector. Local pooling, in contrast, computes values over smaller regions, preserving more spatial information.

3.4         RNN Model

RNNs address the disadvantages of FFNNs, specifically their inefficiency in pro- cessing sequential data [24]. RNNs, unlike FFNNs, are specifically designed to address the limitations of processing only the current input without taking into ac- count previous inputs. Sequential data analysis is achieved by integrating present and previous inputs, which is made possible by their internal memory. Figure 3.2 depicts the differences in information propagation between RNNs and FFNNs.

Figure 3.2: Recurrent neural network and feed-forward neural network

RNNs are utilised in various domains including image captioning [25], time se- ries prediction [26], machine translation [27], as well as natural language processing (NLP) [28, 29]. The networks are classified into four categories: one-to-one, many- to-one, one-to-many, and many-to-many [30]. Figure 3.3 classifies these types of RNNs, while Fig

3.5         Model Architecture of Hybrid Model

The algorithm depicted in Figure ?? provides a comprehensive breakdown of the precise steps involved in the proposed hybrid CNN-RNN model for emotion recog- nition utilising EEG data. The algorithm initiates by preprocessing the data, which involves normalising the EEG signals and converting them into spectrograms. Sub- sequently, valence, arousal, and dominance ratings are associated with emotion la- bels through mapping. The model is built by combining convolutional layers to abstract spatial features as well as the Bidirectional LSTM layers to capture tempo- ral dependencies. The last layers are completely connected and employ the Swish activation function to improve performance. The training process incorporates early stopping and model checkpointing, while the evaluation step employs accuracy, con- fusion matrix, classification report, and F1 scores to assess the model’s efficacy.

3.5.1       Input Representation

The input to our model is structured as a 3D tensor representing spectrograms of EEG signals, denoted as X ∈ RN×T ×F , where:

• N is the number of samples,
• T is the number of time steps,
• F is the number of frequency

This representation allows the model to capture both temporal and frequency-domain characteristics of EEG signals simultaneously.

3.5.2       Convolutional Layers

Convolutional layers are pivotal in extracting spatial features from the spectrograms. The process begins with the application of k filters of size ( f , f ) with a stride:

Z₁ = Conv2D(X,W₁) + b₁

where W₁ and b₁ are the weights and biases of the first convolutional layer, respec- tively. The Swish activation function is then applied:

A₁ = Z₁ · σ(Z₁)

 Algorithm 1 Proposed Hybrid CNN-RNN Model for Emotion Recognition 1:  Input: EEG data X                     RN×T ×F , Valence, Arousal, Dominance ratings 2: Output: Emotion labels Y