There are issues around AI that require careful consideration, but that shouldn’t curb excitement around what AI can do for people living with disabilities.
«Accessibility by design» is an important concept for Microsoft, and one that underpins many of its artificial intelligence-powered products, including Seeing AI.
Announced on Wednesday among a series of other AI tools, Seeing AI is a free mobile application designed to support people with visual impairments by narrating the world around them. The app — which is an ongoing research project bringing together deep learning and Microsoft Cognitive Services — can read documents, making sense of structural elements such as headings, paragraphs, and lists, as well as identify a product using its barcode.
It can additionally recognise and describe images in other apps, and even pinpoint people’s faces and provide a description of their appearance, though camera quality and lighting might influence its description.
At the Microsoft Future of Artificial Intelligence event in Sydney, Kenny Johar Singh, a Melbourne-based cloud solutions architect at Microsoft, demonstrated Seeing AI, which he uses to help navigate the physical world.
Although he lost 75 percent of his vision due to a degenerative retinal condition, technology has been an «empowering force» that compelled him to pursue a career in the industry, Singh said.
Where before Singh was reliant on his wife to bridge the information gap between him and the physical world, by using Seeing AI he is able to be more independent.
In front of guests, Singh used the app to scan a product, which the app correctly identified as Bounce’s coconut lemon-flavoured natural energy ball.
«What it’s actually done for me is that, now that it has detected the ball, I have the calorie calibration — proteins, carbohydrates, fats, and so forth. So I know what I’m picking and using which basically means that I can be totally independent now and my wife absolutely loves it because she doesn’t need to be dragged into stuff like this,» he told guests at the Microsoft Summit.
Singh also pointed his phone’s camera at Jenny Lay-Flurrie, Microsoft’s chief accessibility officer, and the Seeing AI app described Lay-Flurrie as a «54-year-old woman with brown hair wearing glasses looking happy». The last three descriptive pieces of information were accurate, though it was off-the-mark about her age.
«People who are blind are not the best at taking pictures, and we often get a lot of edge cases coming through with those pictures. Pictures that aren’t in the middle bracket of the high quality pictures that you can use to… adapt your algorithms and get some machine learning. They are in a lot of ways sometimes dirty pictures, but the overall quality of cognitive services will increase exponentially because you’re including this data sample,» Lay-Flurrie told guests at the Microsoft Summit.
Microsoft’s latest accessibility features in platforms such as Office 365 and Windows 10 are being used by the Australian National University’s law lecturer Cameron Roles, who said in a statement, «Now is definitely, in my view, the most exciting time in history to be blind.»
One of triplets, Roles, who is also a director with Vision Australia, was born three months early and the oxygen that saved his life left him blind. Of particular use to him are the latest accessibility features in platforms such as Office 365 and Windows 10.
For example, Microsoft has recently integrated its alternative text engine into the core of Windows 10, which Lay-Flurrie said was a «serious step». This means visually impaired users that use third-party screen readers or Microsoft’s own Narrator, which reads text aloud and describes events, will be able to get a description of the contents in an image, rather than simply knowing there’s an image on screen.
The company also recently introduced Eye Control, a built-in eye tracking feature for Windows 10, enabling people living with motor neurone disease (MND) and other mobility impairments to navigate their computers. The feature currently only works with Swedish eye-tracking vendor Tobii ‘s Eye Tracker 4C, though Microsoft is working to add support for other similar devices.
Within Eye Control is a capability called «shape writing», aimed at speeding up typing by allowing the user to look at the first and last letters of a word and «simply glancing at letters in between». Microsoft said in August that a «hint of the word predicted will appear on the last key of the word», and if the prediction is incorrect, the user can select other predicted alternatives.
The company has additionally introduced colour-blindness filters in Windows 10, a condition that Lay-Flurrie said is more common that people think, affecting one in nine people.
Lay-Flurrie, who has a hearing impairment, said that while we need to consider the potential implications of AI in areas such as privacy and security, we should also look at the positives. She said there are 1 billion people living with disabilities globally, and AI can empower these people both in their day-to-day lives and in workplace environments.
«I also look at my daughter, who has autism, she’s 10. With autism, you don’t always understand social cues and social language, and facial expressions are not obvious to her what they mean. She often misinterprets what we’re saying… So I love the potential and the power and beginning to see a wave of innovation in the area of cognitive and mental health where you’re understanding those social cues, you’re using that visual stimulus to give examples, you’re helping to prompt what would be the next step to your learning through real life as opposed to sitting there… some of the therapeutic applications [require you] to sit there and watch YouTube videos,» she said.
«I think there’s real-time applications that can change the lives and include in the same way you do in the workplace with PowerPoint Designer and position people as the geniuses that they are… And you need to be able to perform at the same level as anyone else. Cognitive services and some of these beautiful engines could give us that capability.»
Lay-Flurrie also said people with disabilities can lead or contribute significantly to innovation.
«People with disabilities have a unique lens on the world that could really give a massive input of innovation here and accelerate our path with AI,» she said at the Microsoft Summit.
Microsoft’s chief storyteller Steve Clayton communicated a similar sentiment at the event, saying innovations designed by and for people living with disabilities can prove to be useful more broadly.
«When I started to learn about inclusivity in design, the dropped kerb was originally invented for people who are in wheelchairs. It turns out that the dropped kerb on a sidewalk is also incredibly useful if you’re carrying groceries or if you’re on a skateboard. There are these serendipitous moments I think we’ve found where we said, ‘Hey, we’re going to create a piece of technology that is for people with visual impairment or other disabilities’ that actually turned out to be incredibly useful for the rest of the world,» he said.
Microsoft has been able to integrate AI into its products, while offering new AI-powered products, because of advances in computer vision, speech recognition, and natural language understanding.
The company has developed technologies that can recognise speech with an error rate of 5.1 percent and identify images with an error rate of 3.5 percent.
Microsoft is also currently leading a competition run by Stanford University that uses information from Wikipedia to test how well AI systems can answer questions about text passages. The competition is expected to generate results that can be applied in areas such as Bing search and chatbot responses.
