PHPNG vs. HHVM - SugarCRM

phpng-hhvm-sugarcrm.md

Zeev Suraski (Zend CTO) recently released some benchmarking figures comparing HHVM to PHPNG (and PHP 5.6). I was particularly interested in the results for SugarCRM. They show a 14-28% speed benefit in favor of PHPNG. I decided to investigate, and ran some basic benchmarks with siege.

It came up with the following results:

	Trans/sec
PHPNG	24.12
HHVM	20.74
Diff.	16.29%

Or, find the raw results here.

These are results for the dashboard for a logged in user and they don't lie: a 16% advantage for PHPNG! Interesting...

I decided to see where HHVM was spending all its time. Enter gperftools. You can enable the CPU and Heap profiler by passing USE_GOOGLE_CPU_PROFILER=1 and/or USE_GOOGLE_HEAP_PROFILER=1 to cmake respectively.

The heap profiler has also been proven to be very useful for finding memory leaks, but that's a story for another day. Because I forgot to do this initially and being impatient, I decided to just load it via LD_PRELOAD:

LD_PRELOAD="/usr/local/lib/libprofiler.so" \
CPUPROFILE=/tmp/hhvm.prof \
CPUPROFILESIGNAL=25 \
./hphp/hhvm/hhvm -ms -vServer.Type=fastcgi -vServer.Port=9001

At this point I could just killall -25 hhvm to start profiling, and issue the same command to stop it. Obviously, this was done after a bit of warmup.

I now had a /tmp/hhvm.prof.0 file. There are many ways of analyzing it, but I decided to go with the gif option:

pprof --gif hphp/hhvm/hhvm /tmp/hhvm.prof.0 > ~/profile.gif

Which resulted in a rather large image which I have cropped to the relevant area:

pprof orders the sample trees from heaviest to least heavy from left to right. Individual nodes that use a lot of resources are enlarged. The stuff on the far right and middle is mostly related to I/O. You'll notice stat is used quite a lot, that's because I didn't use Repo Auth mode, SugarCRM still uses eval in a bunch of places.

The most interesting tree is the one on the far left, particularly compress2 > deflate > crc32_combine64 calls that originate from MemcachedData::setByKey > MemcachedData::toPayload. To this moment I'm still not quite sure why they are so slow, maybe Simon Welsh can pitch in (the MemcachedData stuff seems related to HNI conversion). I compiled PHPNG and HHVM with the same zlib.

It was time to dig into the SugarCRM docs, and I found this page that deals with caching. It seems as if SugarCRM just detects whatever caching mechanisms are available, and uses them presumably in some order that prefers memcached.

PHPNG itself doesn't seem to support memcached (yet), and it would make sense because it's a PECL extension. So naturally, I tried adding the following to my config_override.php:

$sugar_config['external_cache_disabled_memcached'] = true;

After that, it was time for another benchmark. Here are the results:

	Trans/sec
PHPNG	24.07
HHVM	35.98
Diff.	49.48%

Or, find the raw results here.

The end result: HHVM is 50% faster than PHPNG!

Conclusion:

• We probably have some work to do in Memcached
• Always do your own benchmarks / profiling :-)

Hope someone found this useful / interesting!

Interesting. Were there any similar bottlenecks for PHPNG, if you checked?

I just retested with this extra override - no visible impact on the benchmarks (190+ req/sec for PHP, 173 req/sec for HHVM).
BTW, Dmitry redid the Magento tests and got similar results to mine, and he's using SSD. So I'm not sure where the source of discrepancy between your tests and ours are coming from... Anyway, we can definitely agree on the 'always do your own benchmarks' recommendation :)

No matter how, a bottleneck with zlib has nothing to do with HHVM. Only when the code execute the same way (call the same function and reach the same branch for the same times), the compare between HHVM and PHPNG make sense.

I agree with @wwbmmm's viewpoint.
I think optimize problem, you should use the same tools compare performance diff(e.g:xphrof),and you analyze performance diff optimize diff hot point.