Apple's latest AI advancement could improve iPhone accessibility and more

Siri on iOS 17 on an iPhone 14 Pro
(Image credit: Future / Apple)

With the Apple AI focus set to continue right through to iOS 18 and beyond, and the company continues to work on creating the tools and features required to make that happen. One example of that is a new research paper that details Ferret-UI, a multimodal large language model that can make sense of what's on a user's display.

Large language models are what power chatbots like ChatGPT while a multimodal LLM, or MLLM, can make more sense of things like images and videos. Now, Apple's researchers have unveiled Ferret-UI, an MLLM that potentially can understand what's on an iPhone's display.

The uses for that could be many, and Apple doesn't go into too many details. But accessibility seems like an obvious use case, as does app testing among developers.

The AI is coming

The research paper hints at exactly that, too, although Ferret-UI could be used in other ways as well.

Detailing potential use cases, the paper says that Ferret-UI could be a "valuable building block for accessibility, multi-step UI navigation, app testing, usability studies, and many others."

Normally, making sense of an iPhone's display would be problematic due to the shape and size of its various components. Buttons are smaller than what these AI systems normally work with, and tapping one of those buttons changes everything. Ferret-UI can work with those new complications, but it remains to be seen how Apple implements it.

One potential use case is iOS 18, the AI-laden update that is expected to be announced at WWDC in June. Ferret-UI could help Siri make sense of an app's interface and potentially help users figure out how to carry out certain tasks, for example — something that could be extremely useful as interfaces become ever more sparse.

More from iMore

Oliver Haslam

Oliver Haslam has written about Apple and the wider technology business for more than a decade with bylines on How-To Geek, PC Mag, iDownloadBlog, and many more. He has also been published in print for Macworld, including cover stories. At iMore, Oliver is involved in daily news coverage and, not being short of opinions, has been known to 'explain' those thoughts in more detail, too. Having grown up using PCs and spending far too much money on graphics card and flashy RAM, Oliver switched to the Mac with a G5 iMac and hasn't looked back. Since then he's seen the growth of the smartphone world, backed by iPhone, and new product categories come and go. Current expertise includes iOS, macOS, streaming services, and pretty much anything that has a battery or plugs into a wall. Oliver also covers mobile gaming for iMore, with Apple Arcade a particular focus. He's been gaming since the Atari 2600 days and still struggles to comprehend the fact he can play console quality titles on his pocket computer.