UPDATED: Apple: 8500 Apps to Review a Week by 40 Odd Reviews


UPDATE: Icebike makes a couple great points in the comments:

  • Updates may not be reviewed as thoroughly as new apps (though with Apple, who knows?)
  • Part of the process is no doubt automated, where test suites a

As part of their response to the FCC's investigation into the rejection of Google Voice, Apple stated that they 1) receive about 8500 apps and app updates to review each week, 2) each app is reviewed by two reviewers, and 3) employ more than 40 full-time, trained reviewers.

Assuming that (3) doesn't mean there are scads more part-time, untrained reviewers doing grunt work in the dark, sweaty back room (more on that in a moment), some math has been run by Mike Ash:

With 17,000 [8500 x 2] reviews per week and 45 reviewers, that means each reviewer performs 378 reviews per week. At 40 hours per week, this is 9.4 reviews per hour, or one review every 6.4 minutes.

Ash points out how this means months of work by a developer is left to the tender mercies of less than 10 minutes (counting overtime) with someone tasked to look at almost 400 other apps that same week. Can we get a "yikes!"

Back to part-time, untrained reviewers, Marco.org hazards to guess:

There could be 41 full-timers and 40 more part-timers. There’s a lot of evidence to indicate that most (if not all) of the front-line reviews are by non-native-English speakers and on schedules that strongly imply that they’re offshore. This may be the cause of a lot of the frustrating rejections in which the reviewer didn’t understand something about the application or description that seems clear to most Americans.

To recapitulate. Between iPhone users and 8500 weekly app submissions (each reviewed twice), stands possibly an unknown number of outsourced, untrained frontliners, 40 odd trained, full-time second liners, an unquantified star-chamber of executive reviews, and ultimately one Phil Schiller who may or may not email the developer or a blog (or two) about it?

Oh, and Steve Jobs.

Have something to say about this story? Leave a comment! Need help with something else? Ask in our forums!

Rene Ritchie

EiC of iMore, EP of Mobile Nations, Apple analyst, co-host of Debug, Iterate, Vector, Review, and MacBreak Weekly podcasts. Cook, grappler, photon wrangler. Follow him on Twitter and Google+.

More Posts



← Previously

Do Other Countries Lose Out on Apps Because of AT&T Policies?

Next up →

Apple Afraid Google is Taking Over the iPhone?

Reader comments

UPDATED: Apple: 8500 Apps to Review a Week by 40 Odd Reviews


Given the money involved, the hardware sales apps generate, and the money Apple has, and the highly public level of frustration with both the time taken and inconsistent quality of the review process there's no other way to put it other than to say 40 is fucking pathetic!

ya that is pretty effin scary. If I were a dev that just spent $20,000+ on developing a kickass app, put in blood and sweat into my product just to have some wonker spend less than 10 minutes looking over EVERYTHING in the app, I would leave to someone who cares about me. M$ seems to be shmoozing devs at the moment. Might be a good choice while they are desperate for apps...

Hold on a second....

about 8500 apps and app updates to review each week

An application might need a complete review, updates don't need much scrutiny. (The Developers know if they sneak anything in that breaks a rule they will get tossed).
So reviewing an upgrade to iSSH, for instance takes 1 minute. Test a couple connections, look for a web browser (to tag it 17+) and pass it on. (In spite of this, one reviewer did kick an upgrade of iSSH back because it used the vibrator, which the Touch does not have. The developer took it out when used on a Touch).
New apps probably take 10 to 25 minutes at each station. And I suspect each reviewer checks for certain things.
Including the church-lady who scans for dirty words.
After that, I can't see revisions getting as much scrutiny as new apps.
Also, don't forget that Apple is a Computer company.
Apple can scan the executables for calls to their APIs and images, and textual content. This is probably how a great deal of the work is automated, and the reviewers know the app has the 7 dirty words embedded before they even install it to test.

That's what I was thinking. App updates need very little review time. Simple apps probably don't take a long time, leaving more time to the more complicated apps.
Really, you can do a full tour of most iPhone Apps in under 10 minutes. Add to that a basic list of disqualifying criteria to go for first (low hanging fruit) and the math starts to look more reasonable.
But this does explain how shaken baby apps get through. Something like that could be easy to overlook during the monotony of this process.
I'm not sure about automation. It's possible there are some basic things that can be done with automation tools to look for bad things.
Still, the numbers are pretty staggering and puts it all in to perspective as to exactly why things get held up and why mistakes are made. No one's perfect.
Makes me want to go buy the App QA team a round.

How do you review an app like Hero of Sparta for six minutes? the game takes 2 to 3 hours to complete. or Monkey Ball? or Cras Bandicoot?
the later stages could have a nude competitor.

Wow, great feedback from folks breaking this process down. A lot to think about.
@icebike - That makes sense about updates being much faster to review. Though not a huge difference, 20+ minutes, for example, would seem more feasible than 7 minutes for some level of app assessment.
@jbrandonf - Good point that it would take a long time to go all the way through a game (which is a primary genre of apps for the iPhone) and ensure nothing "inappropriate" there. Hmm.
Reading about the details of all this leaves the impression that the beast being created by demand for iPhone applications is becoming a bit unwieldy! But, I suppose it's just reaping what you sow if a company makes their product insanely popular but insists on retaining that level of control over the software allowed on it.

These calculations in the main article dont make any sense (i.e. 6.5 minutes per app) Apple says they receive 8500 apps and updates to review every week, but Apple doesn't say that its people actually review that many in a week.

i dont belive that for a second. I own a small business. If that were me, i would have hired way more people already. and this is Apple we're talking about. there i no way those numbers are right.

It may not take that long to review updates to apps and some new ones may not take that long either. But only 40 people to review? Come on now, with their billions I'm sure they can at least double that for literally mere pennies if this job is out sourced.

Reviewers don't have to get all the way to the end of the game.
Game characters and scenes are embedded as images and image fragments in the executable and other resources stored in the game. Computers are really good at scanning for that stuff.
This might change in the future as the phone gets more powerful. Then characters can simply be meshes of triangles (nothing but numbers) which are imaged by the GPU. But we aren't there yet, and Apple does not forbid such content on any of its computer platforms.

Apple needs to offer the ability to sideload apps. OR approve everything minus tethering apps and such and use the killswitch if an app poses a security threat. The app store is just too big to regulate it app by app, update by update.

Forget the manpower (outsourced or not), forget the estimated time to review an individual app (new or update)...Apple said 8500 apps and updates to review each week AND that they complete 95% of that every TWO weeks...can we do the math for this one??
By this equation, I see there being twice as many left and waiting for review...
8500 apps x 4 weeks = 34,000 apps
95% of 8500 = 8075 (completed in TWO weeks)
8075 x 2 = 16,150 apps reviewed in a month.
34,000 submitted. 16,150 reviewed. Hmmm. Please...please tell me I have this completely wrong.

The formatting for this site jumbled my equation. Sorry...
8500 apps x 4 weeks = 34,000 apps...
95% of 8500 = 8075 (completed in TWO weeks)...
8075 x 2 = 16,150 apps reviewed in a month

Don't forget, there's more to it than just the review. As with any large company, I'm sure there is paperwork, and reports to complete. If something is rejected, there probably has to be a summary. Notifications have to be sent. Oh, and who works a full 40 hours of a 40-hour week?
Not trying to defend them, just want to be sure we're realistic.

As someone who submits apps to Apple, I know this system is broken first hand. The worst is they don't really test anything, and if there is a bug and the developers fix it in as fast as possible it will still take weeks to get to the customers hands. During that time every customer will be totally upset with us as the developers even though we would love to get them a fix right away. The fix stop testing stuff at all just put it live. It is up to us to comply with the agreements, and fix and test our apps. With the thousands of apps competing, a company will have to fix all issues with speed if they are going to make any money.