AWS EC2 Performance Storage cloud

When you launch an instance in Amazon EC2, the instance type that you specify determines the hardware of the host computer used for your instance. Instance types include varying combinations of CPU, memory, storage, and networking capacity so that you can choose the appropriate mix of resources for your need.

A noticiable improvement in the current generation of instance types is the introduction of SSD storage for instance store. The disadvantage with instance store is that it is ephemeral, meaning it persists only during the lifetime of its associated instance. To keep valuable data safe, it shoud be stored in Amazon EBS which is persistant and backed up using snapshots which are stored in Amazon S3. However, with the right technique, this extremely fast SSD ephmeral storage can still be leveraged to significantly improve performance. In this series of posts we will discuss a exactly that.

There are obvious use cases where the data stored in SSD ephemeral storage is temporary or can be rebuilt again without affecting the application, and these will still benefit greatly from the improve speed provided by SSDs. In other scenarios, the lack of peristance in instance store can be improved by creating highly available architecture where the risk of losing ephmeral disks can be mitigated by replicating data to secondary instance; this indeed is a valid approach. The aim here however is to focus on techniques that combines instance store SSD with EBS to provide a hybrid like solution that hopefully offers the best of both worlds.

Over a number of posts we will cover the following:

In this first part we will set the scene and go through our test environment.

Testing

For testing I’ll use an EC2 instances that looks like this:

Instance Type: c3.8xlarge
vCPU: 32
RAM: 60
Ephemeral Disks: 2 × 320 GB SSD
Network Performance: 10 gigabit

Depending on the test I might use one of the following AMI’s:

Ubuntu 14.04 LTS: ami-0f12d078 Windows Server 2012 Base: ami-458b4b32

Testing in Linux will be done using The flexible I/O tester/benchmarker fio. For the sake of simplicity we will perform sequential and random read and write tests with the following options:

Block Size (KB): 16
IO Depth: 32
IO Engine: libaio
processes/threads: 32

1
2
3
4
$ fio --directory=/<directory>/ --size=1G --name=test \
--direct=1 --rw=write --refill_buffers \
--ioengine=libaio --bs=16k --iodepth=32 \
--numjobs=8 --time_based --runtime=10 --group_reporting

For Windows we will use SQLIO.

Pre-warming

Because of the way Amazon virtualizes both instances store and EBS disks there can be a penalty of 5 to 50 percent loss of IOPS the first time each block is accessed. Therefore it is recommended that you pre-warm the drives by reading or writing once before production use.

Note: The I2 high I/O instance type uses direct-attached solid state drives that provide maximum performance at launch time, without pre-warming.

To prewarm a volume, we simply run the following command:

1
$ sudo dd if=/dev/xvdb of=/dev/null

dd is not verbose by default so I used the following script, it shows the status of the pre warming of EBS Volumes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash
if [ -z "$1" ]; then
  echo "Usage: sudo $0 /dev/sdh1"
  exit 1;
fi
 
dd if=$1 of=/dev/null & pid=$!
while true; do
  ps -p$pid --no-heading || break;
  echo "-- $(date) ------------------";
  kill -USR1 $pid;
  sleep 60s;
done
echo "-- $(date) ------------------";
echo "DONE \o/"

On c3.8xlarge it took aprox 22 mins to pre-warm 320GB SSD ephemeral disk and 114 mins for 400GB provisioned iops EBS volume with 4000 iops. First indication of just how fast these SSD’s are.

Note: If you are doing the same make sure you run the script under a screen or a tmux session.

For Window, the approach described in AWS documentation will be used: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html#ebs-prewarm-windows

Benchmarks

As a baseline I have run fio tests against the SSD and the 4000 iops provisioned iops volume, the results look as follows:

Next is Part 2 where we will discuss Software RAID with mdadm --write-mostly .. stay tuned.

Comments