Developing an Intelligent Saudi Sign Language Translator for Inclusive Communication

In this blog, I will walk you through a audio integration and final implementation on hand gesture recognition system that utilizes the Mediapipe framework along with machine learning models. Audio Integration: After spending several weeks on how to detect hand signatures, I finally have a model capable of detecting and recognizing all the hand gestures mentioned in previous posts. Now it’s time to go beyond text-to-speech. The basic idea is to move from just detecting and printing the detected sign on the screen to having the program speak the detected sign out loud. The main task now is to explore the options available to implement this feature. During my online research, I identified two main options: Using Python libraries to convert text to speech Using an online service to convert text to speech To give you an idea of the pros and cons of each option, I will describe the information I found under both topics. Built-In Option Since the main program is bu...

Hyper-parameter tuning is the process of enhancing a machine learning model's performance by selecting the optimal set of hyper-parameters. These values are defined before the training begins and play a crucial role in guiding the model's learning process. Unlike the model's parameters (such as neural network weights), which are learned during the training phase, hyper-parameters are set manually and significantly impact the model’s accuracy and generalization ability. Benefits of Hyper-Parameter Tuning: Improved Model Performance: Proper tuning helps the model achieve better accuracy and generalization by identifying the most suitable configuration. Prevents Overfitting/Underfitting: Optimizing hyper-parameters such as regularization and model complexity can help avoid overfitting (when a model learns noise) or underfitting (when a model fails to learn). Efficient Resource Use: Well-tuned models make efficient use of computational resources, saving time and re...

This blog provides an in-depth look on the R&D phase about MLP, along with detailed insights from the training and testing phases of the Multilayer Perceptron (MLP) model. In the realm of artificial intelligence, neural networks have become a cornerstone for solving complex problems. One such neural network that has garnered significant attention is the Multilayer Perceptron (MLP). After creating the dataset, we need to have a clear understanding of MLPs. But before we dive into the technical details, let’s first get acquainted with what an MLP truly is. A Multilayer Perceptron is a type of neural network composed of multiple layers, including an input layer, one or more hidden layers, and an output layer. Each of these layers contains neurons, the fundamental building blocks of the network. MLPs are renowned for their versatility and are widely used in tasks like forecasting models and image pattern recognition. [1] Exploring the Architecture and Functioning of a Multilaye...

The whole project development process can be divided into 3 main segments:  • Data generation • Building, training and testing of the model • ROS integration.  The following blog post will be focused on the data generation process. Selecting all the signs to be identified by the model was done and explained in the previous post and this will focus on the development of the program to capture hand landmarks. A brief introduction of the mediapipe library is also included in the post for your reference.  Google Mediapipe:    This is an open source project maintained by google which provides customizable machine learning solutions[1]. The hand detection implementation of media-pipe is used in this project to collect hand information to generate the dataset.MediaPipe Hands is a hand-tracking technology that predicts hand skeletal data using a single RGB camera. Two interdependent models—a palm detector model and a hand landmark model—make up the solution. To identify...

The data-collection program has been completed, and the next step is to begin collecting data to construct the dataset. For this purpose, 38 different signs will be collected, with a minimum of 800 data records expected for each sign. The data collection procedure will be executed in the following order: 1.The program will be executed via the terminal with the specified sign name. 2.The individual will present the hand sign to the webcam and commence data collection by pressing ‘0’. The advantage of this data collection program is that the background does not affect the quality of the collected data. This is due to the capability of the MediaPipe framework to recognize the hand regardless of other visual noise in the background. Additionally, the distance between the hand and the camera does not impact data quality, as the program normalizes the joint distances from the wrist joint. To enhance the robustness of the collected data, the following measures will be implemented: ...

In Saudi Arabia, a nation rich in traditions and diverse cultures, Saudi Sign Language (SSL) serves as a crucial communication method for the deaf community, embodying both the Kingdom's heritage and adaptability. The Origin of Saudi Sign Language (SSL) developed naturally within deaf communities in Saudi Arabia. Initially based on informal sign systems used in families and local settings, SSL evolved into a recognized language with unique grammar, syntax, and regional variations. Government Acknowledgment and Support Efforts:   to formally recognize and support SSL have increased, with the Ministry of Education and other institutions promoting SSL as a cultural and linguistic asset. Steps have been taken to integrate SSL into educational settings, providing resources and support for deaf students. Usage and Community Influence: Estimating the number of SSL users is challenging, but it remains a primary communication tool for many deaf individuals in Saudi Arabia. SSL ...