EBS Volume Performance

A performance based review

Photo By Anthony Main on Flickr

The Background

When you create AWS EC2 instances, you can attach two types of volumes to them.

1. EBS volumes
2. Instance Store volumes

Out of these two volume options, EBS is the persistent storage option and Instance store is the non-persistent / ephemeral storage option.

Out of these two options, I am trying to highlight some of the key performance booster options such as credit systems and burst modes provided with some of the EBS volume types and why other volume types do not have them by default. It is always important to understand the key differences in each volume type for us to make key architectural decisions. I hope this article will be able give some insight to it.

EBS Storage Performance

Lets start the analysis by introducing some keywords in storage performance.

EBS storage performance could be measured in two ways.

1. IOPS — The number of I/O operations per second
2. Throughput — The speed of data transfer measured in megabytes per second. This is based on the number of I/O operations and the block size of each I/O operation. (Throughput = Number of IOPS * Block size)

The Latency means, the time it takes for an I/O operation to complete.

SSD vs HDD

When it comes to EBS, there are two main volume categories and they are known as SSD volumes and HDD volumes respectively.

SSD backed EBS volumes are primarily used for transactional, IOPS intensive workloads.

HDD backed volumes are primarily used for throughput intensive workloads with large I/O sizes and sequential reads (See Figure 1).

Figure 1— SSD vs HDD

EBS Volume Types

Within those two categories, there are multiple EBS volume types available.

1. General Purpose SSD
2. Provisioned IOPS SDD
3. Throughput Optimized HDD
4. Cold HDD
5. EBS Magnetic HDD

See figure 2 for a high level comparison of those.

Figure 2 — EBS Volume Types

Now, lets dig bit deeper into these EBS volume types one by one.

General Purpose SSD

There are two General Purpose SSD types available.

1. General Purpose SSD — gp2
2. General Purpose SSD — gp3

Out of these two, we have been dealing mostly with gp2 but in 2020 December, AWS came up with its new type called gp3.

GP2 Volumes

This is “still” the default general purpose SSD storage type. It might change in the future to gp3, but not yet.

If you’re unsure of which volume type to use for your workload, gp2 volume is the best default choice, because they offer a more balanced price or a performance for a wide variety of workloads.

The size of the storage volume could be minimum as 1GB and maximum as 16TB.

Gp2 volume type IOPS are calculated based on the size of the volume. This is known as the baseline performance.

Baseline Performance

We can calculate the baseline performance for a given gp2 volume based on its volume size (see the formula below).

IOPS for the gp2 volume = (size of the volume in GiB x 3)

Though the gp2 IOPS are based on the above formula, the minimum baseline performance for a gp2 volume is considered as 100 IOPS.

This means, up to 33.3 GB volume capacity will have the support up to 100 IOPS (33.3 * 3). After 33.3 GB, the above generic formula will be applied.

Lets do some simple calculations to understand this further.

Example 1: [gp2 volume size = 30 GB]

Applying the generic formula -> IOPS for the gp2 volume = 30 * 3 = 90 IOPS

Since gp2 volume size < 33.3 GB, IOPS for the gp2 volume =100 IOPS

Example 2: [gp2 volume size = 50 GB]

Applying the generic formula -> IOPS for the gp2 volume = 50 * 3 = 150 IOPS

Since gp2 volume size > 33.3 GB, IOPS for the gp2 volume = 150 IOPS

I/O Credits and Burst Mode

I/O credits represent the available bandwidth that the gp2 volume can use to burst when more than the baseline performance is needed. The more credits your volume has for I/O, the more time it can burst beyond its baseline performance level and the better it performs when more performance is needed (See Figure 3).

Figure 3 — gp2 burst bucket (Source: AWS Documentation[5])

Each volume receives an initial balance of 5.4 million I/O credits to its gp2 burst bucket. This initial balance of 5.4 million I/O credits is enough to support maximum performance of 3000 IOPS for 30 minutes continuously (3000 IOPS * 60 * 30 = 5.4 million I/O Credits).

This initial credit balance of 5.4 million I/O credits is primarily designed to provide a fast initial boot cycle for EBS boot volumes and relatively a good bootstrapping experience for other applications whenever required.

3000 IOPS means 1 TB (1000 GB). That means, volumes till 1 TB can be burst up to 3000 IOPS and volumes which are over 1 TB size can move up to 16,000 IOPS as the maximum (at 5.33 TB volume size). From 5.33 TB to 16 TB baseline performance remains at 16,000 IOPS (See Figure 4 and 5).

Figure 4— The Baseline Performance of a Volume

Figure 5— Burst IOPS vs Baseline Performance [Source AWS Documentation [5]]

Once the gp2 volume uses all its I/O credit balance, the maximum IOPS performance of the volume remains at the baseline IOPS performance level and it will replenish the burst bucket during idle times or when performing low levels of I/O.

This design allows AWS to provide consistent and predictable performance for all gp2 users. In practice, very few gp2 volumes ever completely deplete their burst-bucket.

If you notice that your volume performance is frequently limited to the baseline level (due to an empty I/O credit balance), you should consider two options.

1. By increasing the volume size to a higher one. Since gp2 baseline performance is relying on the size of the volume this will work. You can try this by using the “Modify volume” option under EBS volumes.
2. By switching to a more robust gp3 volume.

GP3 Volumes

This is the next generation general purpose SSD volume type.

Unlike gp2, the baseline performance is independent of the storage capacity and provides the baseline performance of 3000 IOPS irrespective of the the storage size. Hence, there is no credit system with gp3 as you did find with gp2.

This further provides 20% lower price than the gp2 volumes.

Unlike gp2, gp3 IOPS scaling can be done without provisioning any additional storage.

Provisioned IOPS SSD

There are two main volume types under Provision IOPS SSD.

1. Provision IOPS SSD — io1
2. Provision IOPS SSD — io2

io1/io2 vs io2 Block Express

There is one more iOPS SSD volume type called io2 Block Express, which will give substantial performance improvements compared to io2 standard volume type.

Like General Purpose SSD gp3 volume type, this volume type also can be adjusted independent from the volume size.

IOPS = Both io1 and io2 can achieve 64,000 IOPS limit. The io2 Block Express can go up to 256,000 IOPS.

Throughput = Both io1 and io2 can achieve 1000 MB/s throughput. The io2 Block Express can go up to 4000 MB/s.

Volume Size = Both io1 and io2 have the range from 4GB to 16TB. The io2 Block Express can range from 4GB to 64TB.

Instance Types = Both io1 and io2 volume types are available for all Amazon EC2 instance types. Block Express only supports R5b EC2 instance type.

Throughput Optimized HDD

Low cost, designed for frequently accessed throughput intensive workloads with large, sequential I/O. (For example, Big Data and Data warehousing applications).

Defines performance using throughput rather using IOPS.

Credits and Burst Performance

Just like gp2 volume type, this volume type also provides the credit based burst architecture.

Volume size determines the baseline throughput of your volume, which is the rate at which the volume accumulates throughput credits. Volume size also determines the burst throughput of your volume, which is the rate at which you can spend credits when they are available. Larger volumes have higher baseline and burst throughput. The more credits your volume has, the longer it can drive I/O at the burst level (See figure 6).

Figure 6 — Throughput Optimized HDD Burst vs Baseline throughput [Source: AWS Documentation]

Figure 7 — st1 burst bucket [Source: AWS Documentation]

Cold HDD

Extremely low cost volume type out of all SSD and HDD volumes explained before. Designed primarily for infrequently accessed, large, sequential and cold data workloads.

Credits and Burst Performance

Just like gp2 and st1 volume types, this volume type also provides the credit based burst architecture.

Volume size determines the baseline throughput of your volume, which is the rate at which the volume accumulates throughput credits and the rate at which you can spend credits when they are available. Larger volumes have higher baseline and burst throughput. The more credits your volume has, the longer it can drive I/O at the burst level.

Figure 8 — sc1 burst bucket [Source:AWS Documentation]

Figure 9 — Throughput v Volume Size on sc1 volumes [Source: AWS Documentation]

Conclusion

The above analysis could have given you some idea about how AWS designed its EBS volume types to improve their performances. You must have seen how some of the performance boosters was able to handle some of the heavy workloads even if you had selected a smaller volume size. In addition to that, you would have seen some volume types were able to natively support to larger volumes without any performance booster options. Knowing these facts would be invaluable for you to make storage related decisions in your AWS projects.

References

1. Understanding Burst vs Baseline Performance with RDS and GP2: https://aws.amazon.com/blogs/database/understanding-burst-vs-baseline-performance-with-amazon-rds-and-gp2/
2. Burst balance metric for EC2’s General purpose SSD (GP2) volumes: https://aws.amazon.com/blogs/aws/new-burst-balance-metric-for-ec2s-general-purpose-ssd-gp2-volumes/
3. Introducing new amazon EBS general purpose volumes: https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-new-amazon-ebs-general-purpose-volumes-gp3/
4. EBS volume resizing: https://crishantha.medium.com/ebs-volume-resizing-b39d36528316
5. EBS Volume Types (AWS GitHub): https://github.com/awsdocs/amazon-ec2-user-guide/blob/master/doc_source/ebs-volume-types.md
6. io2 Block Express Volumes: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#io2-block-express