How Apple can detect objects in your photos without creeping your data

Photos on iPhone
Photos on iPhone (Image credit: iMore)

As part of the live Talk Show at WWDC 2016, Apple senior vice president of software engineering, Craig Federighi, and senior vice president of worldwide marketing, Phil Schiller, provided more details on how deep learning and artificial intelligence is being used in iOS 10 to surface search results without requiring you to share your data with Apple.

How will Apple index my existing photos?

When you first download iOS 10, if you have existing photos in your library, your iPhone or iPad will begin to process them in the background at night when you're plugged in. That way you won't see any performance degradation or excessive power drain during the day when you're trying to use your iPhone or iPad.

Once complete, all your old photos will be indexed for the new, better search.

What about on macOS?

Same thing. iOS typically gets released a few weeks before macOS (formerly OS X), though, and not everyone with an iPhone or iPad has a Mac, so Apple wants you to be able to enjoy the new search benefits immediately.

When the macOS Sierra update arrives later this fall, it'll index your Mac Photos library the same way.

Wait, won't the search index just sync between devices?

Not right now, but maybe one day. A system would need to be built that securely, privately shared metadata and index information between iPhone, iPad, Mac, and other products.

What about new photos? Do I have to wait for them to index?

Nope! Apple enjoys a tremendous lead when it comes to chipset architecture in mobile, and they're "spending" some of it on deep learning and AI processing immediately when photos are capture.

The image signal processor (ISP) inside the Apple A9 already handles an incredible amount of calculations for everything from white balance to burst selection. Deep learning and AI take a couple billion more, but the A9 GPU can still handle those near-instantly.

Why doesn't Apple need me to give them and their servers my photos data to get the indexing done?

According to Federighi, Apple doesn't need our photos to figure out what a mountain looks like in a photo. Their "detectives" managed to look at public domain images and figure that out.

But will it work as well as services that do require photo data sharing?

To be determined. We'll have to wait for iOS 10 to launch this fall and really put it through its paces. Personally, privacy is as valuable to me as money, time, or attention, so having the option is great for customers.

I don't use Google or Facebook photos today, so for me any upgrade will be great. If you aren't sure yet, you'll need to see how much functionality it gives you, weigh the options, and make the best choice for you.

Everything about WWDC 2020
○ WWDC 2020 remote lineup
Download the Apple Developer app (opens in new tab)
iOS/iPadOS 14
○ macOS 10.16
watchOS 7
tvOS 14
Discussion forums

Rene Ritchie

Rene Ritchie is one of the most respected Apple analysts in the business, reaching a combined audience of over 40 million readers a month. His YouTube channel, Vector, has over 90 thousand subscribers and 14 million views and his podcasts, including Debug, have been downloaded over 20 million times. He also regularly co-hosts MacBreak Weekly for the TWiT network and co-hosted CES Live! and Talk Mobile. Based in Montreal, Rene is a former director of product marketing, web developer, and graphic designer. He's authored several books and appeared on numerous television and radio segments to discuss Apple and the technology industry. When not working, he likes to cook, grapple, and spend time with his friends and family.

  • This seems to be the feature that Apple acquired from the Perceptio acquisition. Google currently has a lead in this area, so it's nice to see Apple catch up. The bonus for Apple users is that you get this functionality without compromising your privacy as you would with Google's Photos service.
  • I love the double standard. We praise Apple for "not creeping out data" with the way they index photos, however you back up those SAME photos to iCloud, where Apple has access to them. So really, what's the difference between Google's method and Apple's? (Here's a hint: there isn't) In the end, no one is creeping anyone's data. You place a certain amount of trust in the company you chose to purchase the device from. This is nothing more than a huge marketing gimmick and I can't believe how far this thing is getting ridden on. Posted with the Nexus 6, Nexus 5, or Surface Pro 3
  • Using the cloud component with Apple's photos tool is optional. It's not optional with Google's tool. If you do use Apple's cloud service, it's for backup and syncing purposes only. If you use Google's tool it's for the same, plus the creepy server side photo identification and categorization. What Google is doing with that information is unknown. However, fundamentally, Google's motivations will always be in question when generally speaking, their services are intended to monetize their user's information for advertisers. Likewise, the double standard you speak of is based on a reality that you choose to ignore.
  • Read Jerry's post on Android central about what Google does with your data some months ago. If it turns out Google is doing naughty stuff with my data then I will be the first to leave, until then I trust them.
    We need to find out more about how Apple is doing this and make sure it what it is and not just not marketing speak. Apple too collects user data but makes money off of hardware. What ever the case I actually agree with the last paragraph and I don't offten agree with René! Posted via the iMore App for Android
  • 1. Google is the advertiser. It's not to to their benefit to scatter your information throughout the internet. 2. Read the EULA, it generally is better than conjecture. 3. Apple putting the processing on the device also functions to trap you on the platform, if we really want get into the whole evil corporate overlord talk. Apple makes their money by keeping you on Apple hardware for as many purchases as possible. The article Waldon Newman mentioned (I believe):
  • Truth and reality has no place here... Fear mongering is more at home in these forums.
  • I'm aware of Google's privacy policies. Google has also had their own privacy violations. Do a simple search, they're not hard to find. Anyway, the point of my post is that you don't know what Google is doing with your data. The fact that Apple is keeping the analysis and indexing local to the device should be an inherently more private approach to the same problem. Seriously, this is just common sense... no conspiracy theories required here.
  • Even more reason to make sure Apple is doing what it says it is and that it won't be "creeping out" any of my pictures. Nothing wrong methinks in questioning what they or anybody else is doing a with our stuff. Posted via the iMore App for Android
  • And it likely won't work nearly as well as Google's competing service based on what we have seen in the past when comparing Apple services to Google's too. Nice attempt to copy G Maps and G Photos features, though. Posted from my Nexus 6P
  • It's funny how the zealots condemn a product before it's even released.
  • I'm very underwhelemed with the new Photos features. I have my own (thematic) filing system so auto sorting isn't useful for me plus it doesn't always work in the existing Photos app (some of my regular photos are incorrectly auto sorted into Selfie and Screen Cap folders and no way to change this). I'd rather have more manual control over my photos, especially the option to rename photos in iOS.
  • Will this be similar to how Samsung's Gallery allows you to search for things? If I type "people" in it, it shows me pictures of people. It seems that its search is not too comprehensive (it doesn't find "hamburger", although "food" works just fine), and not as good as Google Photos.
  • Will it work for voiceover users? People who are visually impaired like myself Sent from the iMore App