Artificial Intelligence and Haptics Technology
Becoming the Eyes for the Visually Impaired

Two Scenarios Released Early

“Children with visual impairments can recognize animals through touch. However, existing educational materials primarily use techniques like folded paper or Braille to show animal or character outlines. When they later encounter a stuffed or real animal, the feel is often different from what they learned, often making identification confusing for them.”

This was the reason why Professor Yoo chose Accessible Zoo as the first scenario for the Dot Vista application, a visual image-to-haptic conversion solution developed in collaboration with Microsoft and Dot Incorporation. When visually impaired users upload images of animals like whales, camels, or dragonflies via Windows 11, the application automatically identifies and describes the animal. The Dot Pad then converts this into a tactile form, allowing users to feel the animal's appearance, position, and size in detail by just using their fingertips.

Dot Vista utilizes computer vision AI models to detect objects and their surroundings, using algorithms to interpret the images and provide users with sound or haptic conversion. While ChatGPT-4 can describe images, Dot Vista goes further by providing tactile feedback on aspects like size, position, and composition, which are easily perceived visually but difficult to fully convey only through words. The second scenario, Fruits Classification, identifies the type of fruit in an image and assesses its freshness, conveying this information through touch.

“The second scenario, which determines whether fruit is fresh, was developed to simulate real-life situations like shopping in a market or grocery store, thereby helping visually impaired individuals gain independence and training for societal participation. In other words, we are trying to help them be more independent through this technology. These were the two scenarios we presented as examples at Microsoft Ignite,” explained Professor Yoo.

Dot Vista service uses computer vision AI models to transform images so they can be felt through touch.

Innovative Technology Helping the Socially Vulnerable

For over a decade, Professor Yoo has been working on various projects using haptics (touch technology) to assist those with disabilities, patients, and the elderly.

“My research field is essentially about developing AI and computing technologies for those in need. I combine AI with virtual and augmented reality, haptics, and robotics to create technologies that are useful to users and bring them happiness. For instance, I am using technology to teach beginner learners to use robots and VR based on expert hand techniques, and helping the elderly and patients improve their physical and mental health through AI robots.”

These focuses have led Prof. Yoo to collaborate with Dot Inc., and they have worked together for a long time. Dot Inc. is a social enterprise that developed Dot Pad, the world’s first Braille Pad, as well as other smart devices for the visually impaired, resulting in the winning of the Best of Innovation Award at CES. He continues to partner with global companies to improve accessibility for the visually impaired. Microsoft has also collaborated with Dot Inc. to improve Windows accessibility for visually impaired users, and Dot Inc. proposed working with Professor Yoo, an expert on AI and haptic technologies. This is how the industry-academia collaboration between these three parties began. Since Microsoft wanted to develop and release the service as soon as possible, as it was urgently needed for the visually impaired, the three parties worked intensively to develop and present “Dot Vista” at Microsoft Ignite 2024 in just two months.

: Professor Yongjae Yoo has conducted various human-computer interaction studies using tactile technology over the past decade.

Riding the Rocket of Innovation

“Dot Vista was designed to use the Neural Processing Unit (NPU), a new AI computation device integrated into recent AI laptops and smartphones, instead of relying on large servers. This means that assistive functions for the visually impaired can now run on personal computers. During the Microsoft Ignite presentation, engineering and social contribution teams from Microsoft, along with many other visitors, expressed great interest, and several collaborative projects were initiated through business meetings."

Dot Vista is expected to be released as a Windows 11 application in 2025. As a result, visually impaired users will soon be able to use Dot Vista through Dot Pad and Windows. This technology has vast potential for expansion and ongoing academic research is focusing on increasing the scope. For example, the research team has recently published a paper converting sports commentary into haptic feedback, to improve comprehension, and converting textbooks and children's picture books into tactile formats to provide better educational resources.

Professor Yoo’s vision is for ongoing research to be developed into technologies that can deliver real, practical help and happiness to many people. Through AI and collaboration with global companies, Professor Yoo says he is riding on a ”rocket of innovation”.