EC2 settings

EC2.md

EC2 相關設定

記憶體

1. 設定swap（預設為60, 當系統使用到超過40% memory, 就會嘗試使用swap）

   sysctl -w vm.swappiness=0   # from 60 -> 0
   

2. Huge Page (especially THP)

   for mongodb: https://jira.mongodb.org/browse/DOCS-2131

   echo never > /sys/kernel/mm/transparent_hugepage/enabled
   

File System

1. File System flush issues

    # vm.dirty_ratio = 80                  # from 40
    # vm.dirty_background_ratio = 5        # from 10
    # vm.dirty_expire_centisecs = 12000    # from 3000
   

2. For Mongodb Disk Mounting

   modify /etc/fstab

   /dev/xvdf /mongodb_data ext4 defaults,auto,noatime,noexec 0 0
   /dev/xvdg /journal ext4 defaults,auto,noatime,noexec 0 0
   /dev/xvdh /log ext4 defaults,auto,noatime,noexec 0 0
   

3. 設定readahead

   sudo blockdev --setra 32 /dev/xvdf
   

   可以利用sudo blockdev --report觀看是否設定成功

   每次開完機自動設定

   echo 'ACTION=="add", KERNEL=="xvdf", ATTR{bdi/read_ahead_kb}="16"' | sudo tee -a /etc/udev/rules.d/85-ebs.rules
   

   ref: http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/

4. ulimit

   $ sudo nano /etc/security/limits.conf
   * soft nofile 64000
   * hard nofile 64000
   * soft nproc 32000
   * hard nproc 32000
   
   $ sudo nano /etc/security/limits.d/90-nproc.conf
   * soft nproc 32000
   * hard nproc 32000
   

Mongodb EC2 Related Settings

Virtual Memory & File System

sudo vim /etc/sysctl.conf

vm.swappiness = 0                       # from 60
vm.dirty_ratio = 80                     # from 40
vm.dirty_background_ratio = 5           # from 10
vm.dirty_expire_centisecs = 12000       # from 3000

1. 設定swap（預設為60, 當系統使用到超過40% memory, 就會嘗試使用swap）
2. vm.dirty_ratio 是如果 dirty page 真的超過多少就強制回寫
3. 這個參數主要是降低 Linux 寫檔案(dirty page)的 buffer 大小，如果是寫入的量很大的系統建議降低此值
4. Ref: http://parseflo.at/post/linux-tuning-for-write-heavy-system/

sudo sysctl -p /etc/sysctl.conf         # reload settings

Huge Pages

Huge Page (especially THP) for mongodb: https://jira.mongodb.org/browse/DOCS-2131

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Storage I/O

change xvdf

echo noop | sudo tee /sys/block/xvdf/queue/scheduler

/sys/block/*/queue/rq_affinity  2
/sys/block/*/queue/scheduler        noop
/sys/block/*/queue/nr_requests  256
/sys/block/*/queue/read_ahead_kb    256
mdadm –chunk=64 ...

modify /etc/fstab (using discard to activate TRIM function)

/dev/xvdf /mongodb_data ext4 defaults,auto,discard,noatime,noexec 0 0
/dev/xvdg /journal ext4 defaults,auto,discard,noatime,noexec 0 0
/dev/xvdh /log ext4 defaults,auto,discard,noatime,noexec 0 0

sudo vim /etc/security/limits.conf
* soft nofile 64000
* hard nofile 64000
* soft nproc 32000
* hard nproc 32000

Networking

sudo vim /etc/sysctl.conf

net.core.somaxconn = 1000
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_abort_on_overflow = 1    # maybe

Reference

1. http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html
2. http://parseflo.at/post/linux-tuning-for-write-heavy-system/