Interaction with learners

Microsoft: How AI can foster social interaction between blind children and their peers

For children born blind, social interaction can be particularly difficult. A child may have difficulty directing their voice to the person they are talking to and putting their head on their desk instead. Linguistically advanced young people may find it difficult to maintain a topic of conversation, only talking about something that interests them. In particular, many blind children and young people find it difficult to engage and befriend those in their age group despite a strong desire to do so. This is often deeply frustrating for the child or young person and can also be deeply frustrating for their support network of family members and teachers who want to help them forge these important bonds.

  • PUBLICATION PeopleLens The PeopleLens is an open AI system that provides people who are blind or visually impaired with additional resources to understand and interact with their immediate social environment.

PeopleLens is a new search technology we created to help young people who are blind (called learners in our work) and their peers interact more easily. A device worn on the head, the PeopleLens reads aloud in spatialized sound the names of known individuals when the learner looks at them. This means that the sound comes from the direction of the person, helping the learner to understand both the relative position and the distance of their peers. The PeopleLens helps learners build a people map, a mental map of those around them necessary to effectively signal communicative intent. The technology, in turn, tells the learner’s peers when peers have been “seen” and can interact – a replacement for eye contact that typically initiates interaction between people.

For children and young people who are blind, PeopleLens is a way to find their friends; however, for teachers and parents, it is a way for these children and young people to develop their competence and confidence in social interaction. An accompanying work program aims to guide the development of spatial attention skills believed to underpin social interaction through a series of games that learners using PeopleLens can play with their peers. It also sets up situations in which learners can experience agency in social interaction. A child’s realization that he can choose to strike up a conversation because he spots someone first or can stop a talkative sibling from talking by looking away is a powerful, motivating moment. deepen their attention and that of others.

The PeopleLens is an advanced research prototype that runs on Nreal Light augmented reality glasses connected to a phone. Although not available to buy, we are recruiting learners in the UK aged 5-11 who are supported by a teacher to explore technology in a multi-stage research study steps. For the study, led by the University of Bristol, learners will be asked to use PeopleLens for a three-month period starting in September 2022. For more information, visit the research study information page.

research foundation

The work program, co-authored by collaborators Professor Linda Pring and Dr Vasiliki Kladouchou, draws on research and practice in psychology and speech therapy to provide technology-related activities. The PeopleLens is based on the hypothesis that many social interaction difficulties in blind children stem from differences in how children with and without vision acquire fundamental attentional processes as infants and young children. For example, as they grow up, sighted children learn to internalize a joint visual dialogue of attention. A young child points to something in the sky and the parent says “Bird”. Through these dialogues, young children learn to direct the attention of others. However, there is not enough research to understand how joint attention manifests in blind children. A review of the literature suggests that most research does not account for a missing meaning, and research specific to visual impairment does not provide a framework for joint attention beyond 3 years of age. We conduct research to better understand how the development of joint attention can be enhanced in early education and augmented by technology.

How does PeopleLens work?

PeopleLens is a sophisticated AI prototype system that aims to provide people who are blind or visually impaired with a better understanding of their immediate social environment. It uses a head-mounted augmented reality device in combination with four state-of-the-art computer vision algorithms to continuously locate, identify, track and capture gaze directions of nearby people. It then presents this information to the wearer through spatialized sound from the person’s direction. The real-time nature of the system gives a sense of immersion in the people map.

The PeopleLens is a revolutionary technology that was also designed to protect privacy. Among the algorithms that underpin the system is facial recognition of people registered in the system. A person checks in by taking multiple photos of themselves with the phone attached to the PeopleLens. The photographs are not stored, but converted into a vector of numbers that represent a face. These differ from any vectors used in other systems, so recognition by PeopleLens does not lead to recognition by any other system. No video or identifying information is captured by the system, which ensures that the images cannot be used for malicious purposes.

The system uses a series of sounds to help the wearer place people in the surrounding space: A percussive jolt indicates when their gaze has met a person up to 10 meters away. The bump is followed by the person’s name if the person is registered in the system, is within 4 meters of the wearer, and both of the person’s ears can be detected. The sound of wooden blocks guides the wearer in finding and centering the face of a person that the system has seen for 1 second but has not identified, changing pitch to help the wearer adjust their gaze accordingly. (People who are not registered are recognized by an audible click.) Gaze notification can alert the wearer when being watched.

Community collaboration

The success of PeopleLens, and similar systems, depends on a prototyping process that includes close collaboration with the people it is meant to serve. Our work with blind children and their support systems has set us on the path to building a tool that can have practical value and empower those who use it. We encourage those interested in PeopleLens to contact us to participate in our study and help us evolve the technology.

To learn more about the PeopleLens and its development, see the Innovation Stories Blog on technology.