How Apple used deep neural networks to bring face detection to iPhone and iPad

Apple doesn't want to store your data on its servers. There's just no way to guarantee your privacy once your data leaves your device. But providing services on-device is a huge challenge as well.

From the Apple Machine Learning Journal:

We faced several challenges. The deep-learning models need to be shipped as part of the operating system, taking up valuable NAND storage space. They also need to be loaded into RAM and require significant computational time on the GPU and/or CPU. Unlike cloud-based services, whose resources can be dedicated solely to a vision problem, on-device computation must take place while sharing these system resources with other running applications. Finally, the computation must be efficient enough to process a large Photos library in a reasonably short amount of time, but without significant power usage or thermal increase.

The rest of this article discusses our algorithmic approach to deep-learning-based face detection, and how we successfully met the challenges to achieve state-of-the-art accuracy. We discuss:

  • how we fully leverage our GPU and CPU (using BNNS and Metal)
  • memory optimizations for network inference, and image loading and caching
  • how we implemented the network in a way that did not interfere with the multitude of other simultaneous tasks expected of iPhone.

Fascinating insight into the direction more and more of our computational experiences are going. Read the journal for much more, including flow diagrams.

Also, check out the WWDC 2017 session on the Core ML Vision framwork for a really good breakdown of the results.

Rene Ritchie

Rene Ritchie is one of the most respected Apple analysts in the business, reaching a combined audience of over 40 million readers a month. His YouTube channel, Vector, has over 90 thousand subscribers and 14 million views and his podcasts, including Debug, have been downloaded over 20 million times. He also regularly co-hosts MacBreak Weekly for the TWiT network and co-hosted CES Live! and Talk Mobile. Based in Montreal, Rene is a former director of product marketing, web developer, and graphic designer. He's authored several books and appeared on numerous television and radio segments to discuss Apple and the technology industry. When not working, he likes to cook, grapple, and spend time with his friends and family.