According to Storage Newsletter, Apple has purchased 12 petabytes (1 petabyte is 1024 terabytes, so over 12,000 terabytes) of storage from EMC's Isilon Systems, reported for use with iTunes to store music, movies, TV, and other media content online.
Obviously this will be hugely expensive -- you don't buy this level of storage from Best Buy, after all -- but storing huge amounts of data with both staging, local, and off-site backup doesn't come cheap. If Apple wants to offer something similar to Amazon's Cloud Locker service in the future, this much storage will certainly come in handy.
It's interesting Apple is choosing to go the Isilon route rather than use massive amounts of commodity drives the way Google does. Any data experts out there have a theory on that?
(And yes, if you're trying to figure out how many MKVs you could store on 12PB the answer very nearly is -- all of them.)
(Storage Newsletter via AI)
Reader comments
Apple buying 12 petabytes of storage for iTunes online
interesting logic
1 petabyte is 1024 terabytes, so over 12,000 terabytes
iTunes??
Anybody else have the old movie FreeJack come to mind?
as an ex EMCer my perspective is this:-
1.) Apple will "invest" in 12PB, they will not consume that from EMC day one, nor will they pay for it day one - it will be on a PAYG(row) basis
2.) As mentioned earlier in this thread, there are a lot of other cool technologies in play here, de-dupe, replication, backup, etc - so JBOD or GoogleDisk approach is not close to being appropriate.
3.) Google build for disposable consumption - Apples strategy needs to include a robust, scalable architecture that also supports GCR (Governance Compliance & Risk), DRM, auditing, etc, etc.
Besides this - I bet they get a cracking deal around $0.08 per GB - so why invent their own Google/ Cloud based tech (which would require additional research/ investment/ maintenance) when a vendor has it all on a platter for them (excuse the disk based pun please).
Cheers,
So I'm coming from a background of using external hard drives (I've upgraded from a western digital 500GB drive to a 2TB life saver). It is clear that the industry is going into the clouds, but for some reason I'm concerned about it. My concerns is all the content I have in physical form: how will I get it onto the cloud? Will I have to re-buy every single thing if Apple doesn't recognize it? What about music that was not purchased on iTunes? Will the cloud be stable enough to handle a users extensive library? I've been using iTunes since 2003 and I've accumulated A LOT of songs, tv shows, movies, apps, etc. Call me crazy (or a digital hoader) but I'm not ready to part with any of my content. I guess this is me freaking out to the impending change that's going to take place soon. FYI my library breaks down to this: almost 3,000 songs, nearly 600GB of TV shows, 233 apps, and near 150 movies. (yeah, I live and breathe iTunes, judge me!)
Such storage systems typically use data deduplication techniques to gain vast amounts of space, this goes far beyond storing the same song only ones.
What surprises me is that Apple is confident that they will need such an amount of storage from day one, storage tends to drop in price so they could have started with less. I guess they got a hefty rebate due to volume and decided to go for it.
As an employee of isilon systems I can tell you that our system is highly optimized for their needs. We don't use raid. Raid is simply put too slow and unreliable for such a large system.
Well now I fully believe iOS 5 will be highly cloud based.
Omgomgomgomg
i have 2 Terraybyte on my PC. 12000 Terrabyte is space for 10000 people....... but we have millions of ipad/ipod/iphoe user.....
So not true... I only have like 15 gigs of music .. And 100 of movies... So I won't use nearly a TB... my mom may have a gig of music.. So she defiantly doesn't need a TB... you do, so you will pay a premium price for it
I know folks that came from Isilon (pre acquisition). They have a system optimized for virtualizing this level of storage way beyond other solutions. "Virtualization" has much to do with striping across drives but there is TONS of optimizations that can be made after that (de-duplication technology, addressing space efficiencies, etc...). Because of those storage virtualization technologies, you can't just say "50GB per user means XYZ users total" because (at a minimum) if a file is the same for 5 users, you don't store 5 copies you just give 5 people access to the same copy (that is an incredible simplification but basic idea).
Isilon also made its bread and butter selling these systems to niche markets that included media delivery.
That is true from an efficiency point of view, but maybe not a legal one. Quite a few pundits think that Amazon's Cloud Drive might be in the clear precisely because user's have their own distinct copies -- that is, users are streaming only the bits they uploaded. Services like mymp3.com were promptly sued out of existence in part because the bits were "shared" between users.
This seems like a lot, but it isn't. Jobs has said that Apple sold 50 million iPhones 4 worldwide. That's about 250 Mb per customer. If we forget international users, that's still around 15 million iPhone users in the US, which give each one less than 1Gb. Add to that iPad and iPod users, and a hefty overhead for backup, and I suspect this storage is NOT for a service like Amazon's.
They don't need much space per user. They only need to store Michael Jackson's "Thriller" once. Then for the 15 million users who are licensed to that "Thriller" they only need to store the license key (or whatever).
This is oversimplified of course - if all 15 million users try to listen to that one Thriller file simultaneously it will work worse than RealPlayer (buffering...) - They would need a few copies of Thriller to be able to distribute load and whatnot.
But the concept is that they don't need to store every media file for every user - just the list of what the user is licensed to play...
Crap, as I scroll down I see Joel beat me to it. ;)
I know that, but the point is that it's not a service like Amazon's, where you can upload any song, bought or not at Amazon. Users have their music at different bitrates and formats, ripped from their CDs and bought from various sources. If Apple ever launches such a service, it will be a streaming service for iTunes, and not a cloud storage for music. If it's just streaming, Apple does not need the space. My guess is that this storage is just for plain iTunes content.
I'm no storage expert but I imagine there's quite a bit more to it than that. There's storage facilities, local and offsite back ups, maintenance, electricity and various other costs associated with having that much memory available to users.
"Hugely expensive?" Aren't 1TB drives available for $50? Based on that, it seems like it would only cost Apple about $1 or $2 million for 12,000TB, which is small beans for them. Or am I missing something?
Yeah you're missing some stuff. 1TB drives cost more than $50. Also there is a lot of other hardware and software and personnel and electricity and bandwidth and... well you get the idea.
Let's not forget that business class, high I/O drives are much more expensive than $50/TB. You have to pay (in a 'round about way) for the hosting company's fire protection system, backup systems, generators, on-staff support, etc.
This will be quite expensive.
Lets not forget that you can't just buy as many drives as you need to make 12PB. Instead you need to buy five times that to do a RAID 5 array to ensure no down time. How happy will you be if Apple goes for the the cloud and you end up all of your things gone because Apple didn't properly plan for failures.
Once you have that level of storage, running a large datacenter, managing the traffic (major internet pipes)and the network traffic is no small business.
I used to run the ecommerce site for a company and spent a lot of time touring datacenters and talking with their staff. It is not something you do on the cheap if you want to do it well.
Thanks for the responses, guys. As I wrote, "Am I missing something?" Clearly, I was. And now I know.
Thanks.
So that's enough space for 240,000 users to have 50 gb each. How many iOS users are there?
I am no data expert, but that has never stopped me from offering up hair-brained theories.
Google has invested significant time and effort building the software infrastructure to tie massive amounts of commodity hardware together. It is part of their competitive advantage in data processing for search, so they have made it part of their core competency.
Apple will have huge storage and IO needs, but nearly the same distributed computational needs or parallel requirements. This is not a shot at Apple; quite the opposite. They likely just realize that it makes little sense to throw money and brains at a problem that will not help their core business, so they are buying from a vendor who can solve the specific problem space they need to deal with.
CoD is easily the best poster here. Besides me, of course.
Ugh, that should read "not nearly"
Actually, I enjoy his facts and your sarcasm.
I've often wondered if (copy of) dev is actually the old poster, Truth...
Hah no...my "old" name, back when Truth was around, was just plain "Dev" (which is why Rene sometimes refers to me as Dev in posts). At some point when they updated the tipb software, "Dev" was no longer allowed as a name -- too short? A vendetta against me;) ? No idea. So, I changed to (Copy of) Dev. as that sounded somehow less silly than "Dev alias".
There shouldn't be any problem with Dev, I'll look into as well. (And I also really enjoy and learn from your comments, highly appreciated!)
It may have been a temporary glitch, or I may have messed something up, myself, just like I messed up so the above smiley didn't show. :) Don't waste your time on it.
Back to the regular thread -- that's a lot of storage
Hey, Rene: Thanks for cleaning up some of those more, um, unsavory comments. I appreciate an open and free exchange of ideas, and am happy to say I'll continue to stop by TiPb for further edification.
Cheers.
"the cloud" works, until it doesn't. At which point all cloud fetishists are screwed because they either lost their data, or it is now owned by someone else, or they have to cough up big money every time they need/want to access it. Local storage still has a lot of advantages, most notably full control over the content and an exact understanding of the cost for storing or using that data. The time when ISPs will charge data plan style fees, just like we experience it for 3G access already, will come. They just have to sucker everyone onto the band wagon first.
Local storage just works...until it doesn't. No storage medium is perfect. "The Cloud" should be used in conjunction with local storage. I prefer the cloud when I'm on the go since I see no reason to walk around with 4TB of data when I just need a few hundred songs.
Kind of makes my home media server upgrade to 15TB seem insignificant.
Wow, that's nuts. I thought I was cool with 3. Actually 3.5TB but still. Do you have every movie/tv show you have ever purchased on that? lol
I had 4.5Tb, but was getting close to the limit (300GB left). It took me 3 years to fill this.
I am beginning to shoot a lot of HD video and I take a lot of RAW photos, so I needed a little headroom.
I'm guessing this will get me through another 3-4 years.
I'm with you. I purchased the VIO POV HD helmetcam for action sports. It fills a 32GB SDHC in 4 hours on 1080P. I need more myself. My mediaserver's 10ish GB is running low.
Kinda makes Apple's announcement seem sort of insignificant to me. That's only 800x more storage than you have (or say 4000x what I have). Considering they are incredibly wealthy huge multinational corporation who's WHOLE BUSINESS is computers and data, I'm a bit underwhelmed.
Damn, I gotta stop drunk posting.
*Considering they are AN incredibly wealthy huge multinational corporation whose WHOLE BUSINESS ...
@FLskydiver: you're right, for a comany that makes twice as much in an hour as I do in a year this seems to be a small deal. However I don't have to build, staff and power a data center.
Arthur C. Clarke predefcted that you could store an entire humans lifetime experiences (in 3D, but some compression) in around 4-5Pb.
So Apple bought 3 human brains :)
Recent estimates put the size of the internet at a little over 1Xb (1024Pb).
However, extreme estimates of the size of the internet put it at 1Yb which is a million times larger.
Neat info, thanks.
I got curious and did that math. Apple posted revenue equal to more than 12.2 million dollars an hour in their most recent (1st) Quarter filing. That'd buy a couple hard drives for sure.
Hopefully the cloud is coming soon!