How to store 1.8 trillion photos on AWS

During a recent evaluation of Amazon’s Elastic File System service, we were astounded to discover that it is backed by what can only be described as a gargantuan storage volume, spanning a whopping 9 Exabytes in size.

For those of you familiar with the Unix operating system, here is a screen shot showing the 9 Exabytes in action (note the sheer number of digits in the Available space column).

To put this mammoth of a number into perspective, assume that:

  1. The average size of a photo taken with a decent smartphone these days is around 5 Megabytes (MB).
  2. It’s quite common to see many such smartphones with a capacity of 16 Gigabytes (GB), which is 16,000 Megabytes, which is around 3,200 photos.
  3. Several laptop models now come with as much as 1 Terabyte (TB) of storage, which is the equivalent of 1,000 Gigabytes (1,000,000 Megabytes), which is enough for well over 200,000 photos.

But to get from there to an Exabyte, you’d need eat your way through a further 1,000 Terabytes to get to what’s known as a Petabyte. And it’d take a further 1,000 of those to finally get to an Exabyte. And remembering that our EFS volume is 9 of those, that’s the equivalent of 1,800 billion (or 1.8 trillion) photos!

And fascinatingly, when using the more helpful alternative of the above Unix command (df -h) which shows the used space in percentage terms, you would have to copy an astonishing 90 Petabytes of information (or 18 billion photos) onto this disk volume just to get the Used column to move off 0%.

Amazon Elastic File System

The main selling points of EFS are that it:

  1. Elastically grows (and shrinks) to meet your storage needs;
  2. Runs across multiple data centres (Availability Zones);
  3. Can be attached to more than one server at a time (by virtue of the fact it’s powered by NFS).
  4. You only pay for the storage you consume.

For more, see http://docs.aws.amazon.com/efs/latest/ug/whatisefs.html.