An AI developer has hacked his iPhone to become the eyes of GPT-4, turning his iPhone into an all-seeing AI-powered assistant.
Mckay Wrigley took to Twitter this week to reveal how he had created an AI assistant that can see using his iPhone.
Wrigley used a wild combination of software, YoloV8 for object detection, a vision model, GPT-4 for the AI, OpenAI Whisper for voice, Google Custom Search Engine, and Python, hacked together "in a fun way." He says it was "100x easier" than it looks to create and that "this is what the world will soon look like."
GPT-4 with eyes
I used my iPhone to give GPT-4 eyes.My AI vision assistant:- learned what a Keto diet is- identified Keto foods in my fridge- searched the internet- found a recipe with the ingredientsThe future is here! pic.twitter.com/bOMhJfWRP0May 2, 2023
So what does that world look like, exactly? As you can see from his video, GPT-4 can look around and identify objects using his iPhone's camera. Wrigley says he used the vision assistant to learn what a Keto diet is, identify Keto food in his fridge, and search the internet for a recipe with ingredients.
Wrigley clarifies that he's using a Mac to run the vision model, which shows the video stream, as the iPhone can't quite handle the heavy lifting. But this could be a glimpse at what an AI-powered future using Apple VR could look like. Apple is expected to unveil its new Reality Pro headset at WWDC 2023, a mixed-reality headset for gaming, collaboration, fitness, and more.
What's really exciting is how a tool like this could one day be used for accessibility, helping blind and vision-impaired iPhone users to identify objects in their house using AI, possibly transforming the way they interact with things around the home. There's presumably headroom here for tools that can read out loud things like food labels or instructions on packaging. Who knows, accessibility might well be a sleeper surprise of the AI revolution.
Master your iPhone in minutes
iMore offers spot-on advice and guidance from our team of experts, with decades of Apple device experience to lean on. Learn more with iMore!
Stephen Warwick has written about Apple for five years at iMore and previously elsewhere. He covers all of iMore's latest breaking news regarding all of Apple's products and services, both hardware and software. Stephen has interviewed industry experts in a range of fields including finance, litigation, security, and more. He also specializes in curating and reviewing audio hardware and has experience beyond journalism in sound engineering, production, and design.
Before becoming a writer Stephen studied Ancient History at University and also worked at Apple for more than two years. Stephen is also a host on the iMore show, a weekly podcast recorded live that discusses the latest in breaking Apple news, as well as featuring fun trivia about all things Apple. Follow him on Twitter @stephenwarwick9