Huawei StorySign Uses AI To Translate Kids Books Into Sign Language

Huawei has unveiled StorySign, a new free product that uses artificial intelligence to help kids who are deaf or hard of hearing learn to read. The app utilizes a smartphone's camera to follow along with the text on a storybook page, presenting an animated character who translates the words into sign language. The app is powered by Huawei AI.

In the video below, Huawei explains the trouble deaf children have with learning to read — a problem it aims to help solve with its artificial intelligence technology. Deaf children often have delayed language skills due to their unique difficulties, but Star — an animated avatar — functions to augment reality and help these kids read.

The app supports a number of kids books, which a child reads using a smartphone. While reading, the child points the phone's camera at the text on the page, which the app detects and fills into the screen. At this point, Star the avatar appears alongside the text and translates it into sign language.

While Star signs, the app highlights the word that is being translated. Kids are able to follow along with the words as they're signed, correlating the two and picking up reading skills. Huawei worked with both the British Deaf Association and European Union of the Deaf while developing the product.

At this point, the app supports classic kids books from Penguin Random House. The app can be downloaded for free from both the Huawei AppGallery and Google Play Store. The product doesn't appear to be available on iPhones. The app supports 10 European languages, and there is one kid's book per language supported by the app.