Confused about the myriad of AWS Data Storage Options? In this post we take a look at the whole range of AWS Storage services, and the different types of storage available. By the end of the article you should know your EBS from your EFS and know which AWS storage services are right for your particular use case. But before we dive into AWS cloud storage options, let’s first take a look at the different types of storage in general, so we can relate AWS services back to industry standard terms.
TL:DR warning – this post may be too long for some, so here are some quick links to jump straight to the different AWS storage services:
Different Types of AWS Storage
Block Storage
Block Storage is the oldest type of data storage, where data is stored in fixed length blocks. Block storage is usually accessed by SCSI, SAS, SATA or Fibre Channel interfaces, and is typically used for hosting operating system data. Therefore, disk images attached to virtual machines or cloud instances will run on block storage. But block storage can also be used for hosting databases, applications, virtual machines and containers. Block storage is the fastest type of storage available, as it does not have the file searching overhead required for both Object and File Storage.
File Storage
File storage uses a file system to map where data is stored on the storage device. File storage generally resides on network or direct attached storage, and it takes care of organizing data and presenting it to user. Users can navigate a hierarchical file system from top to bottom using a unique locator such as a filename, location or URL. File storage generally sits on top of block storage, but access to the underlying blocks is restricted. File storage can be used to store files that need to be accessed by multiple users or machines, application binaries, databases and virtual machines.
Object Storage
Object Storage is a repository for unstructured data. An Object consists of the data, which can be a file or multiple files together, and some metadata – data about the data, for example the age or size of the object. Every object has a unique identifier, which means that users or applications can access objects without knowing where they are stored. Object data is accessed via APIs.
Object data is contained in Object Stores, which are very flat in structure compared to file systems. This means that Object Stores can scales to many petabytes, while still delivering high speed access to the Object data. Object stores are typically used to store large volumes of photos, video, audio, logs and analytics data.
AWS Cloud Storage Services
AWS Block Storage
Amazon Elastic Block Store (EBS)
SSD Based EBS Volumes
| Volume Type | Volume Description | Size | Durability | Max IOPS per Volume | Max Throughput per Volume |
|---|---|---|---|---|---|
| io2 | EBS Provisioned IOPS SSD | 4GB -16TB | 99.999% | 64,000 | 1,000 MB/s |
| io2 Block Express (Preview) | EBS Provisioned IOPS SSD | 4GB – 64TB | 99.999% | 256,000 | 4,000 MB/s |
| io1 | EBS Provisioned IOPS SSD | 4GB – 16TB | 99.8% – 99.9% | 64,000 | 1,000 MB/s |
| gp3 | EBS General Purpose SSD | 1GB – 16TB | 99.8% – 99.9% | 16,000 | 1000 MB/s |
| gp2 | EBS General Purpose SSD | 1GB – 16TB | 99.8% – 99.9% | 16,000 | 250 MB/s |
| Volume Type | Volume Description | Size | Durability | Max IOPS per Volume | Max Throughput per Volume |
|---|---|---|---|---|---|
| st1 | Throughput Optimized HDD | 125GB -16TB | 99.8% – 99.9% | 500 | 500 MB/s |
| sc1 | Cold HDD | 125GB -16TB | 99.8% – 99.9% | 250 | 250 MB/s |
Amazon EBS Encryption
Amazon EBS Snapshots
Amazon EBS Snapshots are a simple & cost effective way to protect your data stored on EBS volumes, or indeed on any block storage in any location. EBS snapshots are incremental, which means only the changes since the previous snapshots are stored. EBS Snapshots are stored on Amazon S3 object storage for long term retention, which means they benefit from S3’s 11 9’s (99.999999999%) durability – the chances of losing your snapshots are extremely low! EBS Snapshots can be managed by Data Lifecycle Manager (DLM) which enables the creation of policies for the creation, deletion, retention and sharing of snapshots. Logicata recommends a daily snapshot with 30 day retention as a default starting point for our AWS Managed Services clients.
If you’re looking to protect block storage outside of AWS, this can be achieved with the EBS API. This means you can snapshot your non-AWS block stores to AWS, and quickly recover to an EBS volume in AWS – a simple and cost effective way to achieve basic disaster recovery.
EBS Snapshots can easily be encrypted as with EBS volumes.
Amazon EBS Elastic Volumes
AWS File Storage
AWS has a couple of different file storage options, the choice of which is driven by whether you are a Windows or Linux shop
Amazon Elastic File System (EFS)
Amazon Elastic File System (EFS) is a managed service providing NFS shared file system storage for Linux. EFS supports the NFS 4.0 and 4.1 protocols, enabling connections from thousands of EC2 instances across multiple availability zones and regions. EFS is almost infinitely scalable, scaling to petabytes or even exabytes (EFS shares show in Logicata’s Datadog monitoring as having over 8 exabytes of capacity!). As an EFS file system grows, IOPs and throughput also scale in line with capacity, and burst capacity is available for higher throughput. If sustained higher throughput is required, this can be achieved with Provisioned Throughput – EFS Filesystems can scale to multiple GB/s of throughput. Customers are billed for the volume of data stored in EFS, and EFS file systems are elastic, meaning they grow and shrink as files are added and removed, negating the need for any capacity planning.
EFS Filesystems are highly available and durable. EFS is designed for 11 9s durability, and by default everything stored in EFS is replicated across multiple availability zones. Don’t need this level of availability? AWS has you covered with EFS One Zone storage classes, which save up to 47% on standard EFS costs.
Storing data which is not accessed often? Again AWS has you covered, with EFS Infrequent Access storage classes which can save up to 92% over EFS standard pricing – but be warned, you’ll be charged for accessing data in EFS Infrequent Access.
Here are the 4 storage classes offered by EFS:
- EFS Standard
- EFS Standard Infrequent Access
- EFS One Zone
- EFS One Zone Infrequent Access
Data stored in EFS file systems can be encrypted both in transit, using Transport Layer Security (TLS) and at rest using KMS encryption keys.
EFS now also supports containers and serverless compute options – apps running in ECS and EKS can access shared file systems, as can apps running on Lambda.
Data stored in EFS file systems can be backed up with AWS Backup, which is covered later in this post. Additionally, AWS Transfer Family and AWS Datasync can be used to rapidly transfer on premises files to EFS.
For full specs and pricing visit the Amazon EFS pages.
Amazon FSx For Windows File Server
Amazon FSx for Windows File Server is fully managed file storage accessible over the Server Message Block (SMB) protocol. As with all AWS storage services, FSx is designed to be scalable, highly available and durable. FSx can scale to up to 64TB per file system, and DFS Namespaces can span multiple FSx file systems. Throughput can scale to multiple GB/s, and additional throughput capacity can be purchased if required. Data De-duplication is available on FSx, which could save between 30-80% on data storage costs, depending on the type of data stored.
FSx is built on Microsoft Windows Server, and offers Active Directory integration – both on premise and AWS Microsoft Managed AD.
FSx for Windows File Server is available in both single and multi AZ deployments, with either SSD or HDD backed storage. It can be accessed by many AWS services including EC2, ECS, VMware Cloud on AWS, Workspaces and AppStream. FSX can also be accessed by on premises machines over AWS VPN or AWS Direct Connect. All Windows desktop and Server versions are supported from Windows 7 and Windows Server 2008 onward.
Much like NFS, on premises file systems can be easily migrated to FSx in minutes using AWS DataSync.
And like NFS, FSx file system data can be encrypted in transit and at rest with TLS and KMS.
FSx for Windows Server is backed up daily to Amazon S3 using Volume Shadow Copy Services (VSS).
For full specs and pricing visit the Amazon FSx for Windows pages.
Amazon FSx for Lustre
FSx for Lustre supports concurrent access to the same file or directory from thousands of compute instances. SSD and HDD options are available, and all FSx For Lustre file systems are supported by an SSD backed meta data server ensuring all metadata operations are delivered with sub-millisecond latencies.
FSx for Lustre can be accessed by the most popular Linux AMIs – Red Hat, CentOS, Ubuntu and SUSE Linux. Data can easily be imported from and exported to Amazon S3 via a native integration.
For full details and pricing visit the FSx for Lustre page on the AWS website.
AWS Object Storage
Amazon Simple Storage Service (S3)
I wrote an entire post about Amazon S3 last year so I’m just going to summarize here in table format for reference.
| Storage Class | Designed for Availability | Guaranteed Availability | Availability Zones | Minimum Storage Duration Charge | Retrieval Fee |
|---|---|---|---|---|---|
| S3 Standard | 99.99% | 99.9% | ≥3 | N/A | N/A |
| S3 Intelligent Tiering | 99.9% | 99% | ≥3 | 30 Days | N/A |
| S3 Standard-IA | 99.9% | 99% | ≥3 | 30 Days | per GB retrieved |
| S3 One Zone IA | 99.5% | 99% | 1 | 30 Days | per GB retrieved |
| S3 Glacier | 99.99% | 99.9% | ≥3 | 90 Days | per GB retrieved |
| S3 Glacier Deep Archive | 99.99% | 99.9% | ≥3 | 180 Days | per GB retrieved |
For full specifications and pricing visit the S3 pages on the AWS site.
AWS Snow Family
No self respecting post on types of AWS storage would be complete without a mention of the AWS Snow family – but I’ve also covered that in detail in another post.
AWS Storage Gateway
AWS Storage Gateway is a hybrid cloud storage service enabling on premises access to virtually infinite cloud storage.
AWS Storage Gateway is available as 3 different services:
File Gateway
File gateway enables customers to store files as objects in Amazon S3. Files can be accessed via the standard NFS or SMB protocols, or they can be accessed directly as objects in S3. Once files are uploaded to S3, they benefit from S3 features such as cross region replication and lifecycle management.
Tape Gateway
Volume Gateway
For more details and pricing on AWS Storage Gateway, check out the AWS Storage Gateway pages.
AWS Backup
Last but not least in my complete AWS Storage guide is AWS Backup. AWS backup offers centralized, automated backup of other AWS services including:
- EC2 Instances
- EBS Volumes
- RDS Database Instances
- Dynamo DB Tables
- EFS Volumes
- FSx for Windows & Lustre file systems
- AWS Storage Gateway Volumes
Data can be backed up to different regions and AWS accounts, making it easy to meet compliance and disaster recovery requirements.
So, there you have it – my complete guide to AWS cloud storage services. OK I’ve cheated a little and pointed you to some earlier posts, but I hope I’ve given you a useful reference point for all things AWS Storage related. If you want to keep up with AWS news, why not sign up for my weekly AWS News Roundup email? I promise not to send any marketing spam, just a once a week curated list of AWS news with all other vendor pitches weeded out by yours truly! Thanks for reading.
