Treffer: Self-guided interpretation of images for blind users based on multi-layered interaction.

Image spatial composition, information about the number of objects and their characteristics, and additional details such as the colors and attributes of the people in the scene can provide additional information in an image or photograph. However, the usual approaches to make images accessible to blind persons focus mainly on describing the image's content, without inquiring into other aspects such as spatial composition, colors, background, known faces, etc. In doing so, much information that is present in the image but not included in the description is missing for a blind user. This work explores the combination of image captioning and object detection techniques with the final goal of making images more accessible to blind users. The approach is twofold: (1) state-of-the-art algorithms of image captioning and object detection will be combined so blind users can extract different kinds of information from a given image; and (2) blind users will guide the exploration of the images, so they can gather all the information in a personalized manner and make their own interpretation. This novel interaction is designed as a self-guided multi-layered approach, enabling blind users to personalize their exploration of image content by both guiding the process and accessing the different layers of information as desired. We implemented a mobile application based on requirements obtained from blind users and performed an evaluation that provided promising results. The participants were reasonably satisfied with the usability of the prototype, and in general they were able to gather additional information through their tactile exploration that was not present in the initial image descriptions. However, some issues that were detected in the evaluation, and functionalities that could not be implemented, will be addressed in future work. [ABSTRACT FROM AUTHOR]