Development of AI-Driven Text2Sign-Sign2Text System for Enabling Inclusive Education of Pupils with Hearing Impairment


Hearing impairment is one of the major impediments to achieving inclusive education. As a result, hearing-impaired students face difficulties getting absorbed into an inclusive learning environment along with stigma in social and out-of-classroom interactions. Besides, educational institutions in sub-Saharan Africa are not equipped with appropriate infrastructure and technological resources to achieve a reasonable degree of inclusion of the hearing-impaired in formal learning, thus making the interactions among the hearing, hearing-impaired students and their teachers difficult and sometimes impossible, thus creating a barrier to tapping into the potentials of hearing-impaired individuals to make economic and social contributions to their community. 


The goal of this project is to develop a sign-to-text and text-to-sign software system that will enable seamless interactions among the students-hearing and hearing impaired- and their teachers. Therefore, five (5) most pressing challenges hindering inclusion of hearing-impaired pupils in secondary education from three (3) secondary schools each in Nigeria and Republic of Benin will be identified, two (2) most important of the identified challenges that are directly related to teaching and class interactions from hearing impaired students, hearing students and their teachers will be ascertained, requirements for designing machine learning models will be gathered while data for classroom communication in American and French sign languages will be gathered, an AI-enabled system that will convert sign to text and text to sign will be developed and piloted, and the system will be evaluated for acceptability, satisfaction, and engagement. 


Methods to be adopted include interview and questionnaire administration techniques for data and requirements gathering. Transformer-architecture for deep learning already in use for such a purpose will be adopted to carry out transfer learning on data that we will gather of classroom sessions with sign communication interpretation for the hearing impaired. Professional sign translators will also sign audio recordings of class sessions. The AI model will be integrated into an android software developed for this project and installed on mobile devices. The android software-based solution will be given to participants for communication between hearing impaired and unimpaired participants and hence evaluated for feasibility and usability. 


The project will provide as outputs a platform that will render sign to text and text to sign during class instruction, a database of moving images for sign language alphabets, and vocabularies used for education in the Republic of Benin and Nigeria. The expected outcomes include, enhanced interactions between hearing and hearing-impaired students, enhanced interaction between teachers and hearing-impaired students, improved inclusive education, reduction of stigma toward the hearing-impaired students, improved government policy on inclusive education and improved sub-Saharan regional experience. This project will make possible rendering from text to sign and sign to text, a robust system which to the best of our knowledge has not been in existence. The project will provide economic, social and technological impacts ranging from making the hearing- impaired knowledgeable and employable to educational and socially inclusive society and availability of life-long learning for the hearing-impaired through technology-assisted learning environment.


LOCATION: Ile Ife, Nigeria

FUNDING: $46,598