What you need to know
- Apple has told developers about its new framework, Vision.
- It can be used to help an app detect body and hand poses in photos and videos.
- It can be used to analyze the movement, and even trigger an action based on a gesture, like taking a photo.
A new Apple developer session for WWDC 2020 has detailed how developers can use Apple's new 'Vision' framework to detect body and hand poses in photos and videos.
Apple is adding a new capability for developers to iOS 14 and macOS Big Sur, which will allow developers to make apps that can detect hand and body poses and movements in videos and photos. From the video summary:
Explore how the Vision framework can help your app detect body and hand poses in photos and video. With pose detection, your app can analyze the poses, movements, and gestures of people to offer new video editing possibilities, or to perform action classification when paired with an action classifier built in Create ML. And we'll show you how you can bring gesture recognition into your app through hand pose, delivering a whole new form of interaction. To understand more about how you might apply body pose for Action Classification, be sure to also watch the "Build an Action Classifier with Create ML" and "Explore the Action & Vision app" sessions. And to learn more about other great features in Vision, check out the "Explore Computer Vision APIs" session.
The technology can be used for apps in fields like sports analysis, classification apps and image analysis, object tracking, and more. Not only can it be used to analyze what's happening on the screen, but it can also be used to detect gestures that then trigger action responses. For example, making a certain pose to a camera with your hand could make your iPhone automatically take a photo. The framework has a few limitations, it doesn't perform as well with people wearing gloves, bending over, being upside down or wearing flowy or robe-like clothing.
We are grateful to Setapp for sponsoring our coverage of WWDC 2020. The content of this article reflects solely our own editorial opinion.
Go beyond the Wall in Apple Arcade's Game of Thrones: Tale of Crows
"Game of Thrones" fans can now explore the lands beyond the Wall on their iPhone, iPad, Mac, and Apple TV.
Review: The Porsche Taycan can stream Apple Music, no phone or data needed
The 2020 Porsche Taycan is a sweet ride, but it's also the first to fully integrate Apple Music into its own infotainment system — without your phone or data.
iPhone 12 lens supply delayed due to coating quality issues
A major Apple supplier is to delay shipment of its lenses because of quality assurance issues, forcing other manufacturers to pick up the slack.
Apple-themed Pride shirts you'll love to wear
Apple's iconic rainbow logo has often been associated with the LGBT+ movement. Show your support by wearing an Apple-themed Pride t-shirt, including the one we like the best.