But while that system needed precise matches, the fuzzy logic of machine learning systems and the speed of iPhones’ built-in AI accelerators means that Screen Recognition is much more flexible and powerful. The idea is not a new one, exactly Bigham mentioned a screen reader, Outspoken, which years ago attempted to use pixel-level data to identify UI elements. We can look at the pixels on screen and identify the hierarchy of objects you can interact with, and all of this happens on device within tenths of a second.” “In iOS 13 we labeled icons automatically - Screen Recognition takes it another step forward. “We looked for areas where we can make inroads on accessibility, like image descriptions,” said Fleizach. The idea is not to make manual labeling obsolete - developers know best how to label their own apps, but updates, changing standards and challenging situations (in-game interfaces, for instance) can lead to things not being as accessible as they could be. And by “every,” they mean every - after all, screen readers need to be aware of every thing that a sighted user would see and be able to interact with, from images (which iOS has been able to create one-sentence summaries of for some time) to common icons (home, back) and context-specific ones like “…” menus that appear just about everywhere. The result is that in any app now, users can invoke the feature and a fraction of a second later every item on screen will be labeled. Such systems are very flexible - depending on the data you give them, they can become expert at spotting cats, facial expressions or, as in this case, the different parts of a user interface. Screen Recognition, available now in iOS 14, is a computer vision system that has been trained on thousands of images of apps in use, learning what a button looks like, what icons mean and so on. But the company just unveiled a brand new feature that uses machine learning to identify and label every button, slider and tab automatically. Apple has always gone out of its way to build features for users with disabilities, and VoiceOver on iOS is an invaluable tool for anyone with a vision impairment - assuming every element of the interface has been manually labeled.

Iphone accessibility screen reader