Subproject: Multimedia

ANLoc 2024, Nairobi, 23 February 2024

Subproject: Linguistic Data (LD) and Multimedia

Problem statement
• Exclusion of people with disabilities from the digital world
o Also banking and public services, e.g. ATMs don’t talk to you, it’s all text based
• Video games (education, training, entertainment) are not localized into underserved languages, incl. African languages.
o Leading to missed opportunities for
• Educators
• Business
• …
• Multimedia access in countries with low literacy rates is the key to access and interaction with content. Yet the opportunities presented by this situation are not explored.
o Compare impact of local radio with huge growth rates to that of local language newspapers with huge losses.
• Video games are cool – take up is virtually guaranteed. Yet few games have been localized into underserved languages. Could be just for fun but have incredible impact
• Angry Birds (on Android): everybody is playing it!!! – So why not localize it?
• Doctor - Patient interaction is difficult sometimes because patients don’t have enough knowledge about medical terminology, especially if doctors and patients do not speak the same language.

Research questions
• How to reduce access limitations for people with disabilities using LD and LT, especially multimodal?
• How can LD make the localization of video games easier?
• How could localized multimedia games help in education?
o Imagine a game in Acan, where the game “reads out” the numbers shown on a dice when playing a game like ludo.
• Can localized games be used for learning
o Languages
o Maths
o Strategy
o …
• What role does the fun-factor play in the take up of localized content and in the recognition of its importance.
• Which localized multimedia systems could help improve doctor-patient communication?

Resources
• LD for sign languages
• Video dictionary for sign languages
• Audio corpus
• E-Speak corpora for 12 ACALAN languages to support open text-to-speech / speech-to-text (starting with Swahili and Arabic)
o on (mobile) Android
o Keyword-based speech recognition corpora for 12 ACALAN languages (not sure what is meant here?)
• Recorded audio sermons (exist)
• MONEY & PEOPLE’s TIME
• Subtitles
• Transcription of audio/video
• Dubbing
• Encyclopedic dictionaries: audiovisual illustrations of African plants, animals (cow patterns: Songhy has ~150 words for cows)
• Create repository of animal sounds: e.g. how do dogs bark in Swahili, Acan, …