On Siri and sequential inference

I use Siri, Apple's virtual assistant, all day, every day. Most of the time it's an incredibly enabling technology that lets me both do my job and manage my home more easily and naturally than I'd have previously though possible. But, when it crashes, it crashes hard.

Walt Mossberg, writing for The Verge:

In recent weeks, on multiple Apple devices, Siri has been unable to tell me the names of the major-party candidates for president and vice president of the United States. Or when they were debating. Or when the Emmy awards show was due to be on. Or the date of the World Series. […]

John Gruber, writing for Daring Fireball:

Siri now does know the date and time of the next U.S. presidential debate, but where Siri fundamentally is its inability to maintain context and chain together multiple commands.

Sequential inference — that contextual awareness and chaining of commands — is something Siri's done since launch. It's what made Siri so special, even back in 2011.

Me: What's the capitol of Germany?

Siri: Berlin.

Me: Population?

Siri: The population of Berlin is about 3,610,000

Me: Turn my hallway light on.

Siri: Okay, your light is on.

Me: Make it purple.

Siri: Purple, it is!

The problem isn't that Siri doesn't do sequential inference — again, it's done it since launch — it's that there's no way for you to know which domains or queries will use it when you try.

Siri is a server-side service, and query handling is something that can be tweaked at any time, so I'm guessing any reports of poor behavior are fixable. What's frustrating is that it seems many of those sequential inferences haven't already been mapped out and account for.

That leads to inconsistency, which for customers is as bad or worse than it not existing at all.

Mossberg:

You can now use Siri to "turn the lights blue" or "turn on the bathroom heater" — integrations that Amazon's Echo and Alexa assistant have led the way on. And the always-listening Echo is faster than pressing the iPhone's home button to call up Siri, and more reliable than the "Hey Siri" command, which can be hit-or-miss.

Echo is a room device that's always connected to power and has seven beam-forming microphones. That's great, but a totally different product from iPhone, which is an always with you but mostly on battery, and with only a few phone mics.

Alexa on Echo is more reliable than Siri on iPhone when you're in the same room. When you're across town or across the globe, Siri on iPhone is infinitely more reliable than Alexa on Echo back home. Because it's back home.

I can't comment on whether or not Echo/Alexa "led the way" on home automation controls because Echo/Alexa still doesn't exist where I live, nor in the vast majority of the world. Siri, while not everywhere, is far more global and multilingual. Again, different products with different priorities.

Gruber:

To be fair, I tried the same two-step sequence (when's the next debate?; add it to my calendar) with Google Assistant running in the Allo app on Android, and it failed in the same way. I remain unconvinced that Siri is behind the competition, and even if it is, I don't think it's by much.

This stuff is incredibly hard. Apple's been accused of tying their own virtual assistant hands by instituting a privacy policy that effectively prevents them from churning through all the personal data Google uses to provide more and arguably better features.

But they're also solving for different problems. Apple is focusing on a personal assistant. Siri has a name and a Pixar-like personality, and for all sorts of personal tasks, it does sequential inference just fine. Google is focusing on a Star Trek computer. That's why Google Now and Google Assistant don't have any more of a name or personality than Star Trek's "Computer".

Apple doesn't have to match Google Now or Google Assistant feature for feature — and privacy means they won't be digging through your email or web history to do so any time soon — but Apple does have to make sure Siri can handle the kind of tasks most of Apple's customers will ask most of the time. And do it in a way that's not just delightful but reliable and consistent.

(That, and handle the much bigger part of AI that gets much less attention: The behind the scenes stuff that'll eventually make everything from code more reliable to interface preemptively faster...)

When it comes to the front facing stuff, I'm just a dumb writer, which is about as far from a product manager as you can get. That said, it seems like a lot of what Mossberg complains about, and Gruber notes, could be headed off by having someone with a Steve Jobs or Craig Federighi-like drill-down-to-the-smallest-detail approach empowered inside Apple, hammering on Siri, all day, every day, and making sure it never gets caught off its virtual guard.

Rene Ritchie
Contributor

Rene Ritchie is one of the most respected Apple analysts in the business, reaching a combined audience of over 40 million readers a month. His YouTube channel, Vector, has over 90 thousand subscribers and 14 million views and his podcasts, including Debug, have been downloaded over 20 million times. He also regularly co-hosts MacBreak Weekly for the TWiT network and co-hosted CES Live! and Talk Mobile. Based in Montreal, Rene is a former director of product marketing, web developer, and graphic designer. He's authored several books and appeared on numerous television and radio segments to discuss Apple and the technology industry. When not working, he likes to cook, grapple, and spend time with his friends and family.

45 Comments
  • The moment I read Mossberg's article on the verge I knew your Siri defense article was coming. Thank you for not disappointing me! ;)
  • The moment I read Mossberg's critique I knew he was playing Devil's Advocate in the name of "neutrality"; Mossie is no techno sap; he knows full well that Alexa's always connected to power, immovably static, has seven multi-directional microphones and has barely rolled out across one country and language so far. It's a convenient overlook that enables easy swipes to be taken at Cupertino. And they are not false swipes, HE'S NOT WRONG. Rene nailed it regarding the domain implementation of sequential inference in Siri being all over the place; if you tell Siri to set an alarm and then immediately say "CANCEL IT" (literally, not "cancel the alarm") Siri will bring up your most recent iCloud Calendar Event and ask if you're sure you want to delete it Yes/No... It will however infer correctly and cancel the alarm you created if you say "cancel the alarm". For all the bluster coming from admirers of Google, Viv, Amazon et al, AI is still pretty much in its primitive infancy. Reminds me of a childish trick question we'd ask as kids becoming aware of intelligence & inferential wit:
    Question - Antidisestablishmentarianism is a long word; can you spell it? The answer we'd reveal after several failed attempts, however correct, would be "I-T"...
  • Did you actually read the article?
  • I dunno if that poster did or not, but I did, and the way your article is written it tries to juxtapose Google Assistant and Siri as equally poor as assistants, and it's just not the case. You may feel like your'e being completely fair, but Siri is far behind Google Assistant. It may have a "pixar-like" personality, but it's definitely less capable. By next year it may be well ahead of Google Assistant, who knows? Right now, it is the case that Google Assistant can do more, and answer more questions than Siri can. There are things that Google does better Rene, and there are things that Apple does better. It's just life.
  • I admit I too am a daily user of Siri. Why can't you or the other Siri bashers point out some key capabilities that Google can do that Siri can't? Not just an answer to a question, because I can find as many examples where Siri can answer something that Google can't. I have used both since my significant other has a Google and we frequently test them against each other. What I find is that Google is sometimes more accurate in detecting my words, especially in a noisy environment like my car, but we have yet to find that Google has some increased overall capability that Siri doesn't.
  • Did you even read the article that Rene is trying to defend against? There are some really concrete and very embarrassing examples there of what Siri cannot do.. here's the link because you probably can't use Siri to help you find it: http://www.theverge.com/2016/10/12/13251068/walt-mossberg-apple-siri-is-...
    Oh and by the way, where are your examples?
  • The fact of the matter is Google Now has parts where it doesn't work as well, and so does Siri. I don't think Google Now is better than Siri, the same way I don't think Siri is better than Google Now. Just use whatever works for you
  • Yes, I read it, which is more than what you were able to do with my question. I asked for a list of CAPABILITIES that Google had that Siri didn't, BEYOND merely asking some specific question because both Siri and Google can answer questions that the other sometimes can't. After you wipe the embarrassment off, move on and try and answer the question I posed. Open your mind and try to be objective about the personal assistants. If you want to bash Siri, back it up with things that Google Now or Cortana or Alexa can do that Siri can't. Here's a recent short video comparison done by PhoneRadar website. You'll see that there are some things that Google Now does better, but also CAPABILITIES that Siri has, that Google Now can't do as well or can't do at all. You'll see that Siri also remains far ahead on natural conversation capability. https://www.youtube.com/watch?v=IMRkmet5hBg
  • It seems to me that Siri loves to bring up webpages where Google actually answers the question, verbally. That's just my experience
  • For me, Google Now loves to bring up "searches" instead of answering the question, but that's just my experience too. Which means it's safe to say that both have flaws
  • Gruber makes a great point in his post. That once Siri fails for someone they stop trying. And even if Apple does improve Siri most of those users won't know.
  • You spared me the effort of making this point myself, JB, and it's one that Rene's defense glosses over without mention. After my initial experience with Siri on the 4S I considered her a cute toy, but through three generations of iPhones I hardly ever used her. It wasn't until I got an Apple Watch that I began using Siri with any regularity. I am now a believer, and FWIW I consider HomeKit control via Siri to be the Watch's killer app. But even now, having become quite comfortable using Siri, I seldom find new applications that see regular usage. Why? Because more often than not Siri does not deliver the anticipated results. I'd like to make more use of Siri, but I find myself suffering from the "why bother" syndrome. I fear Apple has an uphill battle rectifying this problem; it's a competitive disadvantage now, and it will only become more of one going forward unless it is addressed.
  • Except I mentioned it:
    That leads to inconsistency, which for customers is as bad or worse than it not existing at all.
  • biggest problem with siri is when i wanna do some things hands free it ask me to unlock my iPhone wish apple would have a smart lock like android. like at the wifi network at home i don't need to lock my iPhone same thing in the car. OR?
  • You either didn't read Rene's article or were the one who "glossed over". He specifically mentioned it and it was a primary focus of it. Like Rene, I too want Siri to evolve greatly, but Siri does so many things well that it is easy to take for granted how powerful Siri is. I am skeptical that the vast majority of Siri bashers have any idea of what she can do, and, as Rene points out, that is Apple's fault. But meanwhile, every day I will enjoy the great benefits of being able to set alarms, timers, reminders (both time and geo based!), check on the weather, get great sports info (games, scores, game stat leaders), traffic updates, send texts, read me texts, get directions, read me my appointments, set an appointment,
  • True for most people. However, for those who embrace the maxim "it is better to light a candle than to curse the darkness", it's the beginning of an exciting and funny journey full of laughter, disgust and surprising discoveries
  • Since Siri is baked in, any time Apple announces a whiz-bang new feature for Siri they have a chance of getting someone to try it again. But they need to pick those moments carefully and nail them.
  • "Apple doesn't have to match Google Now or Google Assistant feature for feature — and privacy means they won't be digging through your email or web history to do so any time soon" And when the next siri update is going to do the exact same thing like google now on android (reading your email etc.) you'll just "i love how siri now works better than ever, it can scan my email and answer it without me doing anything, how great is that?! I love siri" lol rene, you really are something Sent from the iMore App
  • That wont happen - they will do it locally on the device - like the "AI" does in photos now.
  • The jury is still out on that, but I willing to be proved wrong. What happens if you upgrade your iPhone ? Posted via the iMore App for Android
  • Then you resync your email and it reads it locally
  • Apple could always decide to throw away the privacy stance, but we'd lose that stance as an option and that would be worse for consumers, I think.
  • All very interesting, but ulitmately for most people it doesnt work well enough to persist in trying it, as JBSlough says. I tried the other day to ask who Tim Cook is, seeing that Mossberg complained it went to the contact. Well, yea but that's logical if they are in the contact. for me, he clearly isnt, so Siri replies by giving me a link to Apple.com. So for me that is dumb. It KNOWS Tim Cook has something very significant to do with Apple, it seems, but then gives me a generic page! Why not send me at least to his profile page on the exec page?! At the end of the day Apple is bad at SEARCH. It cant match and react properly for Maps, App Store, or Siri.
  • I'm kind of confused as to what this article is saying. Are we saying not to compare Siri to other similar services because it's nothing like those services?
  • For most people whether Apple is solving for a different problem is irrelevant. That one is used when on-the-go and the other sits in your house is irrelevant. They're all voice interfaces that you talk to and expect an answer. When that answer isn't optimal, the service just "sucks". That Siri gives less optimal answers than Google Assistant is very much a problem for Apple regardless of what problem they're trying to solve because most people think they are the same thing and don't understand the problem space. When one assistant-any assistant-suddenly can have human-like conversations, all other services will be perceived as mentally challenged. The assistant that will win is one whose algorithms adjust for the variations of human speech, not one in which a bunch of engineers are manually connecting pre-scripted templates. A good assistant shouldn't need an engineer to tune it. It will tune itself simply by having real conversations with real people. It's in that problem space where I worry Apple isn't keeping up.
  • A computer cannot have "real" conversation with real people. Not even if you employ a brigade of Turings.
  • Siri is not aware of context, and it is not AI. Here’s an example query I try from time to time: “When is the next Ducks game?” Or, during a game, “How are the Ducks doing?” I am in Oregon. Siri invariably responds with informaton about a hockey team in California that is also called the Ducks. Siri is: (1) unaware of my location, (2) unaware of the most popular sports team in my state, (3) unware of my past requests (“Oregon Ducks” etc.), (4) unaware of the requests of people around me, (5) if I’m asking during a game, it’s unaware of the game that’s going on, (6) unaware that very few people in my area ever ask about the California Ducks. It has NO context at all. Any “context” is static, hard coded by Apple’s programmers. Furthermore, it is not AI. My query will get fixed as soon as someone from Apple notices it and manually fixes it. It’s been years. The ability to learn over time is an essential component (the essential component?) of AI. A system that has to be manually programmed for each type of query is not AI and should not be marketed as such.
  • Wow you nailed it. Siri does not learn. Teams of software engineers code Siri functionality. Google started crawling the web almost 20 years ago building the knowledge graph that now has over 70 billion nodes. So Google does not need software engineers coding everything but instead it learns with some guidance. Google solution will just get better and better in a scalable fashion. App!e needs to get serious in this space quick.
  • Siri is not marketed as AI. Never. It is just a voice based layer added to the user interface. AI is research. Siri is product. Apple is not a research institute, it is a profit based private company. It doesn't market "projects", it sells finished products and Siri as such, deserves its qualification as "product".
  • I have read the comments section from Walt's and other anti-Siri people, and many gave examples of Siri failing. Most of those actually worked for me. So I don't know how much of anti-Siri complaints are really true.
  • Anti-Siri people will find faults where none exist, the same with people who are anti-anything
  • Good that Siri is is acting smarter and now knows all this stuff about presidents, but i would prefer knowing most common stuff, that u would ask every single day, rather than focus on soccer etc.
  • Siri does know common stuff, then again, it depends what you count as common stuff
  • If my butler can't answer a question that the buildings concierge knows the answer to, he just calls down and asks. Why wouldn't a personal assistant reach out to a broader knowledge base to answer those questions that are beyond it? Sent from the iMore App
  • Each of these assistants are quite different and I've been frustrated that different lines of questioning have been used to promote one and discredit another. Siri does have some serious inconsistencies. Replace Germany/Berlin with United Kingdom/London and the population question fails to hold context. This is odd because on AppleTV, Siri is all about maintaining context. The takeaways for me are; if you want to do something useful - ask Siri, if you want trivia - ask Google Now, and if you think **** hath no fury like a fanboy scorned - ask Mossberg. Sent from the iMore App
  • You've obviously never used Android....
  • I end up talking to google more than siri on my iphone to get better results, but thats mainly due to apple maps inferior data.
    All in all, I find siri a pretty literal, and stupid assistant. it is good however at setting an alarm or adding a reminder at least, yay
  • Take Watson for example, the most advanced of all such platforms. Here is how IBM defines it: "IBM Watson is a technology platform that uses natural language processing and machine learning to reveal insights from large amounts of unstructured data."
    http://www.ibm.com/watson/what-is-watson.html So, even Watson is not "AI", according to IBM. All that AI hype is just pop culture. I don't deny pop culture, it is just a reality of our everyday life. And Apple knows well how to make profits by selling pop culture as seen with iTunes, Apple TV, Apple Music etc... That AI thing is just not part of their business.
  • I worry about Apple's ability to compete with google on this, google is the worlds search engine and leveraging that to make a personal assistant gives them such a fantastic edge over Siri it's really hard to imagine how Apple can keep up
  • I am skeptical that the vast majority of Siri bashers have any idea of what she can do, and, as Rene points out, that is Apple's fault. And, Sir has a lot more evolving to do, but in the meantime, every day I will enjoy the wonderful benefits of being able to use Siri to set alarms, timers, reminders (both time and geo based!), check on the weather, get great sports info (games, scores, game stat leaders), traffic updates, send texts, read me texts, get directions, read me my appointments, set an appointment, and a ton more. I'll bet folks would be amazed, and again I think the failure is on Apple that Siri can do all of the things on this list. There's about 500 skills https://hey-siri.io Again, Siri is far from perfect, but try out some of the things on that list and you will be amazed at how well they work and make things easier, more productive, safer, nicer, etc. With the advances in from VocalIQ technology and the other tech they are working on, the sequential inference problem Rene talks about, or what Apple refers to as "natural language" is going to see some huge advances with Siri in the near future.
  • In response to a few of the comments above. I think Apple has marketed Siri in many ways. Some are misleading. Even current descriptions. Currently on the Apple Web Site: "Talk to Siri as you would to a friend and it can help you get things done — like sending messages, placing calls, and making dinner reservations. You can ask Siri to show you the Orion constellation or to flip a coin. Siri works hands-free, so you can ask it to show you the best route home and what your ETA is while driving. It works with HomeKit to let your voice be the remote control for connected products in your home. And it’s tuned in to the world, working with Wikipedia, Yelp, Rotten Tomatoes, Shazam, and other online services to get you even more answers. The more you use Siri, the more you’ll realize how great it is. And just how much it can do for you." "Talk to Siri as you would as a friend" creates the expectation of understanding of context. And of general knowledge rather than just a "voice interface". The phrase: "And it's tuned in to the world" creates a similar expectation. Also above is "Siri works hands-free". But many times I get a "I found this on the web", which breaks the hands-free voice interface expected. For example, "Who were the last three presidents" results in a "I found this on the web". For example, "What will the weather be in San Francisco on Monday" results in a similar answer.
  • That false expectation is rooted deep in humans' brains, this is not Apple's fault. Humans just don't know how to talk without insight. But this is the problem of every such machine based service, not only Apple's. "I found this on the web" is a fairly good answer you can get to your meaningless question. Presidents of what? Add "US" to your query and try again. If it still returns web results then report to Apple on their feedback page.
  • Google doesn't have that issue
  • First of all--love iOS10--LOVE it. I just came running back to iOS after being a staunch Windows Mobile user and supporter for over three years. Unfortunately, that ship is sinking quickly. One of the first things I did with my new iPhone 7 was compare Siri and Cortana. I was sorely disappointed to see that Siri cannot come close to Cortana in accuracy or capability. I now have Cortana for iOS installed on my iP7 and it works great. You asked for a firm example: My wife sent me her flight number so I could track it and fetch her at the airport. "Siri, I need to track American flight 2063." Siri pulled up four websites to track or download flight tracking apps.
    "Cortana, I need to track American flight 2063." Cortana audibly answered, "American Airlines flight 2063 from Charlotte to Baltimore is on time to arrive at 9:43pm." Cortana also gave me a map of the inflight animation and asked if I would like to be notified when it lands. Another example: "Siri, call Brixx Gastonia." Siri pulled up and called Brixx Charlotte (wrong location) from my contacts list. The Gastonia location was not in my contacts list, but Siri did not search another database.
    "Cortana, call Brixx in Gastonia." Cortana pulled up the phone number, directions and menu for the restaurant and confirmed I wanted to call Brixx Pizza in Gastonia, NC. Sorry, but it's sad when Cortana beats Siri on Siri's native device.
  • "Sorry, but it's sad when Cortana beats Siri on Siri's native device." Technically all the assistants do their calculations in the cloud, so whether you're on the native device or not it wouldn't really make a difference. If one assistant works better for you, it's going to work better for you no matter what OS you're running it on