Ubuntu 14.04 + zfs + KVM / Free Memory Mystery

Summary.md

System is running Ubuntu 14.04 with 3.19.0-78-generic kernel and locally attached ZFS filesystem (including native root fs).

# free -m
             total       used       free     shared    buffers     cached
Mem:         72484      58779      13705          1          7        136
-/+ buffers/cache:      58634      13849
Swap:        75633          0      75633

• Where is my 57 G (58634 MB)?

# ps -o rss,comm -A|sed -re 's/^ +//'|sort -n|grep -v ^0
RSS COMMAND
164 atd
208 rpc.idmapd
208 upstart-file-br
224 upstart-socket-
1444 sed
1620 acpid
2052 getty
2068 irqbalance
2076 getty
2076 upstart-udev-br
2100 getty
2112 getty
2120 ps
2124 sort
2132 getty
2136 getty
2236 pager
2296 dnsmasq
2300 cron
2336 grep
2376 rpcbind
2528 dbus-daemon
2636 rpc.statd
2740 man
2896 nginx
3220 rsyslogd
3284 systemd-logind
3328 screen
3484 nginx
3756 nginx
3792 nginx
4064 bash
4076 bash
4088 bash
4152 systemd-udevd
4180 nginx
4244 init
5388 sshd
5516 bash
6604 sshd
20796 libvirtd
61600 qemu-system-x86
174580 qemu-system-x86
352804 qemu-system-x86
363912 qemu-system-x86
386152 qemu-system-x86
407736 qemu-system-x86
414788 qemu-system-x86
462956 qemu-system-x86
474600 qemu-system-x86
561932 qemu-system-x86
569992 qemu-system-x86
617168 qemu-system-x86
705936 qemu-system-x86
793556 qemu-system-x86
916060 qemu-system-x86
1035700 qemu-system-x86
2251916 qemu-system-x86
2269640 qemu-system-x86
4207200 qemu-system-x86
4344892 qemu-system-x86

• Sum of qemu-system-x86 processes is 21285444 (20.30 GB)
• Other processes 130100 (0.12 GB)
• Total: 21415544 (20.42 GB)

 Active / Total Objects (% used)    : 9137092 / 9404715 (97.2%)
 Active / Total Slabs (% used)      : 195765 / 195765 (100.0%)
 Active / Total Caches (% used)     : 82 / 127 (64.6%)
 Active / Total Size (% used)       : 1614992.05K / 1690771.47K (95.5%)
 Minimum / Average / Maximum Object : 0.01K / 0.18K / 16.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
6445056 6262637  97%    0.06K 100704       64    402816K kmalloc-64
681472 680470  99%    0.03K   5324      128     21296K kmalloc-32
665712 663657  99%    0.32K  27738       24    221904K taskstats
371280 350652  94%    0.09K   8840       42     35360K kmalloc-96
234576 215949  92%    0.29K   8688       27     69504K dmu_buf_impl_t
215982 215929  99%    0.10K   5538       39     22152K blkdev_ioc
214880 193526  90%    0.05K   2528       85     10112K ftrace_event_field
 64792  55303  85%    4.00K   8099        8    259168K kmalloc-4096
 63712  62713  98%    0.50K   1991       32     31856K kmalloc-512
 58632  58632 100%    0.19K   1396       42     11168K dentry
 48468  47454  97%    0.19K   1154       42      9232K kmalloc-192
 43552  42409  97%    0.12K   1361       32      5444K kmalloc-128
 38964  38964 100%    0.12K   1146       34      4584K kernfs_node_cache
 34524  34524 100%    0.87K    959       36     30688K dnode_t
 31320  31320 100%    1.07K   1080       29     34560K zfs_znode_cache
 31262  31262 100%    0.27K   1078       29      8624K sa_cache
 26578  25884  97%   16.00K  13289        2    425248K zio_buf_16384
 15872  15872 100%    0.01K     31      512       124K kmalloc-8
 14644  14603  99%    0.55K    523       28      8368K inode_cache
 13158  13007  98%    0.08K    258       51      1032K anon_vma
 11232   6509  57%    0.25K    351       32      2808K kmalloc-256
 10240   9536  93%    0.02K     40      256       160K kmalloc-16
  8800   8586  97%    1.00K    275       32      8800K kmalloc-1024
  8544   8544 100%    0.16K    356       24      1424K kvm_mmu_page_header
  6318   6166  97%    0.61K    243       26      3888K proc_inode_cache
  4256   4256 100%    0.07K     76       56       304K Acpi-ParseExt
  4068   3998  98%    8.00K   1017        4     32544K kmalloc-8192
  2920   2920 100%    0.20K     73       40       584K zil_lwb_cache
  2784   1792  64%    1.09K     96       29      3072K zio_cache
  2550   2550 100%    0.04K     25      102       100K Acpi-Namespace
  2324   2324 100%    0.57K     83       28      1328K radix_tree_node
  1848   1312  70%    0.36K     42       44       672K blkdev_requests
  1792   1697  94%    2.00K    112       16      3584K kmalloc-2048

• Kernel takes about 1.5 GB (1614992.05K)

# cat /etc/modprobe.d/zfs-arc-max.conf 
options zfs zfs_arc_max=10737418240

This should limit ZFS memory usage to 10-20 GB, however...

# cat /proc/spl/kstat/zfs/arcstats|grep ^c
c                               4    38002710528
c_min                           4    33554432
c_max                           4    38002710528

# cat /proc/spl/kstat/zfs/arcstats|grep ^size
size                            4    38003001192

ZFS arc is 35 GB (38003001192 bytes).

# cat /sys/module/zfs/parameters/zfs_arc_max
0

Summary

• 20 + 1.5 + 35 = 56.5 G

The problem seems to be that my /etc/modprobe.d/zfs-arc-max.conf did not affect the zfs module when it was loaded. Maybe because I'm using a native root fs? Probably I didn't have it in my modprobe.d when my initramfs was generated.

Solution

I did echo 10737418240 > /sys/module/zfs/parameters/zfs_arc_max and -- after a while -- the result is:

# free -m
             total       used       free     shared    buffers     cached
Mem:         72484      34268      38215          1          7        141
-/+ buffers/cache:      34119      38364
Swap:        75633          0      75633

# cat /proc/spl/kstat/zfs/arcstats|grep ^c
c                               4    10737418240
c_min                           4    33554432
c_max                           4    10737418240