Skip to content

Instantly share code, notes, and snippets.

@SchumacherFM

SchumacherFM/Stats.md

Last active October 9, 2018 05:47
Show Gist options
  • Save SchumacherFM/9150220 to your computer and use it in GitHub Desktop.
Save SchumacherFM/9150220 to your computer and use it in GitHub Desktop.
Download ZIP
Comparing different storage systems for Magento
Raw
Stats.md

Magento Caching Test

Testing three type of caches: persistent, non-persistent and two-level.

Prerequisites

  • MacBook Air
  • 13-inch, Mid 2012
  • Processor 1.8 GHz Intel Core i5
  • Memory 8 GB 1600 MHz DDR3
  • OS X 10.9.1
  • Zend OPcache 7.0.3-dev
  • PHP 5.5.8
  • redis php client 2.2.3
  • memcached 1.4.15
  • MongoDB v2.4.9 (default config)
  • MySQL 5.6.14

APCu 4.0.1 with PHP 5.5.6 results in a Segmentation fault (11)

Benchmark suit: https://github.com/colinmollenhour/magento-cache-benchmark

Before each run the caches has been flushed.

Persistent Storage Systems

File based

Magento default Zend_Cache_Backend_File

$ bash var/cachebench/opcache/run.sh
Cache Backend:  (Zend_Cache_Backend_File)
Loading opcache test data...
Loaded 10000 cache records in 25.31 seconds (24.8313 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 129 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 3358.44|  336.93|    0.48
Client  1| 3331.92|  308.01|    0.49
Client  3| 3362.99|  366.63|    0.48
Client  0| 3617.67|  372.48|    0.50
------------------------------------
ops/sec  |13671.02| 1384.05|    1.95

Rewritten File backend Cm_Cache_Backend_File

/var/cache directory was always empty before the run(s)

$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_File
Loading opcache test data...
Loaded 10000 cache records in 28.13 seconds (27.6723 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 10 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 5938.14|  254.52|  181.05
Client  1| 5943.64|  239.68|  159.26
Client  0| 5951.00|  256.85|  148.05
Client  3| 5921.19|  180.68|  173.78
------------------------------------
ops/sec  |23753.97|  931.73|  662.14

Zookal_Core_Model_Cache_OPcache

This is also a two level cache.

1st run

$ bash var/cachebench/opcache/run.sh
Cache Backend: Zookal_Core_Model_Cache_OPcache
Loading opcache test data...
Loaded 10000 cache records in 13.27 seconds (12.9285 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 111 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 3192.13|  577.17|    0.58
Client  1| 3242.26|  643.26|    0.59
Client  3| 3145.39|  603.99|    0.60
Client  0| 3346.32|  631.83|    0.60
------------------------------------
ops/sec  |12926.10| 2456.25|    2.37

2nd run

Applied Magic Trick (Loaded one time the admin backend and run then the test)

$ bash var/cachebench/opcache/run.sh
Cache Backend: Zookal_Core_Model_Cache_OPcache
Loading opcache test data...
Loaded 10000 cache records in 3.67 seconds (3.4409 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 8 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 9007.14| 1398.61|   21.23
Client  1| 9059.54| 1459.68|   21.02
Client  3| 9121.34| 1419.80|   21.21
Client  0| 9200.80| 1481.28|   21.37
------------------------------------
ops/sec  |36388.82| 5759.37|   84.83

Note: not all entries in the OPcache were written to a file ...

Rewritten OPcache to extend from Cm_Cache_Backend_File

@todo

Redis

Redis server version 2.6.16

Native PHP Redis Client

$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_Redis
Loading opcache test data...
Loaded 10000 cache records in 5.55 seconds (5.2796 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 14 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  0| 4120.33|  937.83| 1196.29
Client  1| 4037.56|  944.25| 1216.55
Client  3| 4033.28|  934.53| 1045.59
Client  2| 4016.23|  892.09| 1206.88
------------------------------------
ops/sec  |16207.40| 3708.70| 4665.31

Redis server version 2.8.6

Native PHP Redis Client

$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_Redis
Loading opcache test data...
Loaded 10000 cache records in 5.67 seconds (5.3932 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 13 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  1| 4090.03|  932.86| 1093.55
Client  0| 4092.96|  948.69| 1134.70
Client  2| 4038.49|  869.52| 1192.82
Client  3| 4041.93|  939.85| 1157.50
------------------------------------
ops/sec  |16263.41| 3690.92| 4578.57

Redis 2.8.6 && Credis_Client

Native PHP redis.so has been disabled and Credis_Client has beend used.

$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_Redis
Loading opcache test data...
Loaded 10000 cache records in 8.35 seconds (8.0632 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 19 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 2893.19|  661.12|  876.66
Client  1| 2842.48|  660.40|  988.77
Client  3| 2837.17|  541.92|  964.61
Client  0| 2825.53|  638.61|  993.46
------------------------------------
ops/sec  |11398.37| 2502.05| 3823.50

Redis config:

maxmemory 128MB
dir /usr/local/redis-data/
save 900 1
save 300 10
save 60 10000
tcp-keepalive 0
timeout 0

MongoDB

Using Cm_Cache_Backend_Mongo (Collin Mollenhour)

  • PHP Client 1.3.7 (default config)
  • mongodb://localhost:27017/?w=1&wTimeoutMS=20000
$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_Mongo
Loading opcache test data...
Loaded 10000 cache records in 6.77 seconds (6.4867 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 33 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 1593.12|  492.16|  249.19
Client  3| 1591.99|  521.11|  229.14
Client  0| 1591.77|  526.20|  252.44
Client  1| 1587.40|  660.26|  244.42
------------------------------------
ops/sec  | 6364.28| 2199.73|  975.19
  • PHP Mongo Client 1.4.5 (default config)
  • mongodb://localhost:27017/?journal=false&w=1&wTimeoutMS=20000
$ bash var/cachebench/opcache/run.sh
Cache Backend: Cm_Cache_Backend_Mongo
Loading opcache test data...
Loaded 10000 cache records in 6.76 seconds (6.4715 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 31 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 1657.72|  623.44|  297.58
Client  0| 1660.19|  536.05|  255.89
Client  3| 1657.54|  462.00|  260.35
Client  1| 1655.32|  534.34|  218.48
------------------------------------
ops/sec  | 6630.77| 2155.83| 1032.30

Using Zend_Cache_Backend_Mongo (Anton Stoeckl)

  • PHP Mongo Client 1.4.5 (default config)
  • mongodb://localhost:27017/?journal=false&w=1&wTimeoutMS=20000
$ bash var/cachebench/opcache/run.sh
Cache Backend: Zookal_Core_Model_Cache_MongoAntonStoeckl
Loading opcache test data...
Loaded 10000 cache records in 6.81 seconds (6.5148 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 21 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  3| 2460.52|  709.59|  339.09
Client  1| 2456.94|  738.01|  307.68
Client  0| 2454.49|  722.97|  311.68
Client  2| 2439.66|  811.59|  356.93
------------------------------------
ops/sec  | 9811.61| 2982.16| 1315.38

MySQL

InnoDB

$ bash var/cachebench/opcache/run.sh
Cache Backend: database (Varien_Cache_Backend_Database)
Loading opcache test data...
Loaded 10000 cache records in 35.56 seconds (35.2136 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 65 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  0|  796.25|  194.35|  170.42
Client  2|  791.21|  181.85|  171.08
Client  3|  790.03|  192.34|  170.60
Client  1|  787.10|  186.27|  154.86
------------------------------------
ops/sec  | 3164.59|  754.81|  666.96

MyISAM

$ bash var/cachebench/opcache/run.sh
Cache Backend: database (Varien_Cache_Backend_Database)
Loading opcache test data...
Loaded 10000 cache records in 35.63 seconds (35.2952 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 66 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2|  771.18|  232.36|  172.00
Client  1|  770.75|  235.54|  162.80
Client  3|  770.17|  227.29|  166.39
Client  0|  766.13|  223.44|  167.75
------------------------------------
ops/sec  | 3078.23|  918.63|  668.94

Two Level Caches

Why two level? First level fast and in-memory, second slower but persistent.

Snippet of Zend_Cache_Backend_TwoLevels to load from cache:

public function load($id)
{

    $res = $this->_fastBackend->load($id);
    if ($res === false) {
        $res = $this->_slowBackend->load($id);
        if ($res === false) {
            // there is no cache at all for this id
            return false;
        }
    }

    // maybe, we have to refresh the fast cache ?
    if ($this->_options['auto_refresh_fast_cache']) {
        ... save in fast cache
    }
    return ...;
}

If first level cache key has not been found (e.g. memcache restart) then search the second level cache for the cache key. If it has been found then save it (if configured) back in the first level.

1st OPcache, 2nd File

@see Zookal_Core_Model_Cache_OPcache

1st Memcached, 2nd Redis

$ bash var/cachebench/opcache/run.sh
Cache Backend: memcached (Zend_Cache_Backend_TwoLevels) + Cm_Cache_Backend_Redis
Loading opcache test data...
Loaded 10000 cache records in 6.78 seconds (6.5028 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 16 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 3896.71|  719.76|   38.29
Client  3| 3920.42|  740.20|   37.80
Client  1| 3903.17|  709.19|   33.33
Client  0| 3892.64|  675.84|   36.43
------------------------------------
ops/sec  |60456.94| 2852.99|  785.85

1st Memcached, 2nd Cm_Cache_Backend_File

$ bash var/cachebench/opcache/run.sh
Cache Backend: memcached (Zend_Cache_Backend_TwoLevels) + Cm_Cache_Backend_File
Loading opcache test data...
Loaded 10000 cache records in 31.33 seconds (30.8888 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 106 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 3252.10|   27.57|    0.59
Client  3| 3246.06|   28.46|    0.63
Client  1| 3387.09|   34.74|    0.59
Client  0| 3431.57|   34.40|    0.64
------------------------------------
ops/sec  |58160.82|  129.17|  642.45

1st Memcached, 2nd MongoDB

with Anton Stoeckls implementation

$ bash var/cachebench/opcache/run.sh
Cache Backend: memcached (Zend_Cache_Backend_TwoLevels) + Zookal_Core_Model_Cache_MongoAntonStoeckl
Loading opcache test data...
Loaded 10000 cache records in 8.50 seconds (8.2084 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 23 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  3| 2332.68|  454.83|   61.65
Client  2| 2303.30|  604.78|   62.03
Client  0| 2331.25|  457.32|   57.63
Client  1| 2316.01|  438.64|   54.44
------------------------------------
ops/sec  |54127.24| 1959.57|  875.75

1st APCu, 2nd Zend File

$ bash var/cachebench/opcache/run.sh
Cache Backend: Apc (Zend_Cache_Backend_TwoLevels) + Zend_Cache_Backend_File
Loading opcache test data...
Loaded 10000 cache records in 22.18 seconds (21.8155 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 127 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 5184.39|  432.71|    0.46
Client  1| 5397.87|  504.60|    0.47
Client  3| 5210.70|  484.03|    0.47
Client  0| 5423.74|  462.79|    0.49
------------------------------------
ops/sec  |21216.70| 1884.13|    1.89

Seems weired, too slow, despite apc.enable_cli is on.

1st APCu, 2nd Cm_Cache_Backend_File

$ bash var/cachebench/opcache/run.sh
Cache Backend: Apc (Zend_Cache_Backend_TwoLevels) + Cm_Cache_Backend_File
Loading opcache test data...
Loaded 10000 cache records in 27.86 seconds (27.4565 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 86 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2| 7769.04|   32.08|    0.67
Client  3| 7612.99|   32.63|    0.72
Client  1| 7599.35|   37.68|    0.68
Client  0| 7426.27|   36.56|    0.74
------------------------------------
ops/sec  |30407.65|  138.95|    2.81

Non-Persistent Storage Systems

Standalone APCu

$ bash var/cachebench/opcache/run.sh
Cache Backend: Zookal_Core_Model_Cache_Apc
Loading opcache test data...
Loaded 10000 cache records in 2.06 seconds (1.8354 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 5 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  2|18160.61| 2322.01| 7494.74
Client  3|18099.39| 2564.96| 7413.34
Client  1|18086.73| 2584.69| 7508.13
Client  0|18100.98| 2508.75| 7396.45
------------------------------------
ops/sec  |72447.71| 9980.41|29812.66

Memcache

No OPcache enabled.

Libmemached standalone

$ bash var/cachebench/opcache/run.sh
Cache Backend: Zend_Cache_Backend_Libmemcached
Loading opcache test data...
Loaded 10000 cache records in 3.75 seconds (3.4832 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 11 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  1| 5373.32| 1322.44| 5495.46
Client  3| 5359.75| 1444.04| 5633.02
Client  2| 5332.40| 1369.92| 5495.95
Client  0| 5339.84| 1411.72| 5547.89
------------------------------------
ops/sec  |66249.31| 5556.12|22812.32

Memcache standalone

$ bash var/cachebench/opcache/run.sh
Cache Backend: Zend_Cache_Backend_Memcached
Loading opcache test data...
Loaded 10000 cache records in 3.65 seconds (3.3911 seconds cache time). Data size is 4979.7K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 10 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  3| 5323.24| 1424.29| 5990.60
Client  2| 5314.45| 1401.16| 5866.16
Client  0| 5321.92| 1435.49| 6019.17
Client  1| 5300.88| 1448.42| 5800.48
------------------------------------
ops/sec  |21260.49| 5709.36|23676.41
@Vinai
Copy link

Vinai commented Feb 24, 2014

Great, thanks for publishing your benchmark results.

Just as a few notes to complement the information here:
The Zend_Cache_Backend_TwoLevel implementation is quite buggy, see Fabrizios great summary here
http://www.fabrizio-branca.de/magento-zend-frameworks-twolevels-cache-backend-mess.html for more information.

Also, the two level implementation isn't about persistance, its mainly about supporting cache tags.

Using memcached or APC without the two-level cache effectively removes Magentos ability to segment the cache. Clearing, lets say, the config cache, would remove all other cache segments, too.
Benchmarks using these options are not comparable since the functionality is flawed.

For those reasons its best to avoid memcached and APC and the two level cache backend entierly.

Since the default Zend_Cache_Backend_File and MySQL storage backends should be avoided since because they don't scale - as your benchmark has shown - that only leaves Colins REDIS or File or one of the Mongo (or any other well manufactured custom backend) as a viable options.

@colinmollenhour
Copy link

Also note that when testing APC, unless you modify the benchmark scripts to use a persistent web server, each instance will be operating on a separate (and empty) cache instance. So APC results run on CLI are invalid.

On MongoDb, first, the Anton Stoeckl version did not use the TTL feature of MongoDB and had several significant bugs so I wouldn't recommend using it in production.

In my testing on my virtual machine (quad-core laptop) MongoDb was consistently faster than Redis (http://screencast.com/t/fAYiQIz0aj). However, the Redis protocol is very simple and efficient (and single-threaded), so on a small load I wouldn't be surprised if Redis beats MongoDb, but it is when the load starts to saturate a single core and your machine has power to spare that MongoDb should easily overtake Redis.

So, perform your tests on a powerful server and bump up the number of clients to get a better picture. Otherwise, if all you care about is best performance on your MacBook then Cm_Cache_Backend_File is the way to go. :)

Also, to be exhaustive you could try testing on TokuMX, which claims to be significantly faster than MongoDb, but 100% client-compatible.

@SchumacherFM
Copy link
Author

Thanks @fbrnc for Visualization‎ https://docs.google.com/spreadsheets/d/1iNI4LK2RRToxqU9rvd7qCYfoD6NSVcqoHlTSsP2CjcY/edit?pli=1#gid=1235446123

@colinmollenhour Thanks! Will compile TokuMX on the weekend :-)

Here are the results from an Amazon EC2 m1.small instance:

$ bash var/cachebench/default/run.sh
Cache Backend: Cm_Cache_Backend_Redis
Loading default test data...
Loaded 10000 cache records in 10.80 seconds (10.2038 seconds cache time). Data size is 4999.0K
Benchmarking 4 concurrent clients, each with 50000 operations...
4 concurrent clients completed in 61 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  0|  936.20|  339.65|  919.02
Client  3|  933.84|  272.43|  201.79
Client  1|  923.63|  439.36|  543.23
Client  2|  931.63|  583.81|  963.27
------------------------------------
ops/sec  | 3725.30| 1635.25| 2627.31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment