EC2 EBS-SSD vs instance-store performance on a non EBS-optimized m5d.4xlarge

dd.log

# /tmp/test = EBS-SSD
# /ephemeral/test = instance-store

[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.15671 s, 340 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.81722 s, 381 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.47532 s, 309 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.50803 s, 306 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.50256 s, 307 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.50407 s, 306 MB/s

[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.07444 s, 349 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.07443 s, 349 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/ephemeral/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.0745 s, 349 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.09322 s, 347 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.09355 s, 347 MB/s
[root@ip-172-31-26-72 ec2-user]# dd bs=1M count=1024 if=/dev/zero of=/tmp/test oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.0929 s, 347 MB/s

hdparm.log

## BENCHMARK

# /dev/nvme0n1p1 = EBS-SSD
# /dev/nvme1n1 = instance-store SSD

[ec2-user@ip-172-31-26-72 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G     0   32G   0% /dev/shm
tmpfs            32G  548K   32G   1% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/nvme0n1p1  500G  2.4G  498G   1% /
/dev/nvme1n1    280G  318M  279G   1% /ephemeral
tmpfs           6.3G     0  6.3G   0% /run/user/1000
tmpfs           6.3G     0  6.3G   0% /run/user/0

[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing buffered disk reads:  42 MB in  3.19 seconds =  13.18 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing buffered disk reads:  62 MB in  3.09 seconds =  20.09 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing buffered disk reads:  86 MB in  3.08 seconds =  27.96 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing cached reads:   18374 MB in  1.99 seconds = 9234.41 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing cached reads:   18420 MB in  1.99 seconds = 9257.47 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme0n1p1

/dev/nvme0n1p1:
 Timing cached reads:   18288 MB in  1.99 seconds = 9190.36 MB/sec

 [ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2460 MB in  3.00 seconds = 819.32 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2068 MB in  3.00 seconds = 688.76 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -t /dev/nvme1n1

/dev/nvme1n1:
 Timing buffered disk reads: 2068 MB in  3.00 seconds = 688.76 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme1n1

/dev/nvme1n1:
 Timing cached reads:   18346 MB in  1.99 seconds = 9219.70 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme1n1

/dev/nvme1n1:
 Timing cached reads:   17792 MB in  1.99 seconds = 8940.80 MB/sec
[ec2-user@ip-172-31-26-72 ~]$ sudo hdparm -T /dev/nvme1n1

/dev/nvme1n1:
 Timing cached reads:   17770 MB in  1.99 seconds = 8929.51 MB/sec

results.md

Setup

Instance type: m5d.4xlarge Instance store: 300 GB NVMe SSD EBS: gp2 500 GiB 1500 IOPS

Based on https://gist.github.com/ktheory/3c3616fca42a3716346b

Device reads (no cache)

hdparm -t /dev/nvme*

Trial	EBS-SSD (mb/s)	instance-store (mb/s)
1	13.18	819.32
2	20.09	688.76
3	27.96	688.76

Cache reads

hdparm -T /dev/nvme*

Trial	EBS-SSD (mb/s)	instance-store (mb/s)
1	9234.41	9219.70
2	9257.47	8940.80
3	9190.36	8929.51

Writes

dd bs=1M count=1024 if=/dev/zero of=/{tmp,ephemeral}/test

Trial	EBS-SSD (mb/s)	instance-store (mb/s)
1	306	340
2	307	381
3	306	309

Direct Writes

dd bs=1M count=1024 if=/dev/zero of=/{tmp,ephemeral}/test oflag=direct

Trial	EBS-SSD (mb/s)	instance-store (mb/s)
1	347	349
2	347	349
3	347	349

Conclusions:

Instance-store is over 30x faster than EBS-SSD for uncached reads.

Instance-store and EBS-SSD are equivalent for cached reads (RAM speed).

Instance-store and EBS-SSD are equivalent for writes (cache?).