LocustVsAwsBztJMeter.md

• AWS/bzt/Jmeter / Apache License 2.0
  • Pros
    • The AWS solution seems to distribute fairly well . I haven't tested it though as simply running locally was sufficient for Notify's needs.
    • Using the AWS solution without JMeter makes it a fairly simple and handy tool to use. This is good for test that does not require complex scenario with parameters passing or complex authentication.
    • JMeter is battle tested by years of experience. It offers a wide range of options and plugins. It can deliver on complicated scenarios.
    • bzt is compatible with many testing tools such as Gatling, but only JMeter integration is available through the AWS integration.
    • bzt offers an abstraction for simple tests that allows us to skip JMeter scripting.
    • Can run tests on many workers in a distributed manner.
    • JMeter has a plugin system that makes it possible to extend its functionality to a certain extent.
    • It has good support to trigger different waves of load .
  • Neutral
    • Have to use AWS machinery of different components (S3, CloudFormation, etc) in order to setup the overall solution, which would involve custom jmeter scripts and parameters.
    • bzt actually offers a nice live reporting abstraction layer on top of JMeter which does not offer great live results options.
    • A non-engineer might actually benefit from JMeter UI, but it still require a good technical knowledge and complex scenarios will require some usage of Apache Java expression language or variants.
    • Stress tests need to be done with care as tests hosted in AWS and stressing other AWS components fall under the AWS stress tests policy .
  • Cons
    • The overall AWS solution is a bit complicated to launch tests.
    • JMeter: Clunky user interface -- kept freezing on my MaxOSX laptop and used my Windows machine to develop JMeter script.
    • I could not get bzt to properly install on my Windows machine too as the current pip install is currently broken.
    • JMeter scripting via the UI is laborious and counter-intuitive. A task as simple as reading from the environment variables is difficult, necessitate a plug-in. Reading Java properties is easier in that case, but it leads to more complicated setup.
    • The AWS solution could limit us on a more complicated setup: how do I pass parameters? read from the external world ?
    • JMeter threading model is killing me every time I use it. Adjusting for request per second is not so easy because all request are done synchronously in a thread that is exposed directly in the JMeter configuration. One needs to calculate the necessary number of threads by guessing the worst request response that a system under stress can take. In an article , the recommended number of threads should follow this equation: $$RPS * / 1,000$$ . Based on our recent load testing, we would plug these values in: $$5,500 * 15,000ms / 1,000ms = 85,500 threads$$ . This is insane even on system with 24 logical processors but necessary, only because each synchronous thread might block on their processed request.
    • The network calls and memory performance of JMeter suffers also due to its execution model via synchronous threads, explained in the previous point. This is generally well acknowledged even in the BlazeMeter documentation .
• Locust / MIT License
  • Pros
    • Performs very well with an asynchronous model. Locust is built on top of gevent which is built on top of libevent , a specialized event loop engine written in C .
    • Easy to target a number of request per second.
    • Writing test scenarios and users is utterly simple yet powerful as the full machinery of the Python programming language is available.
    • Web based UI works good. Charts are available and reported live as the tests runs.
    • Can run tests on many workers in a distributed manner.
    • Programmatic listeners can be accessed to process data on our own, e.g. send these to Slack or in a database, or process incoming request to calculate custom metric .
    • It has good support to trigger different waves of load , with similar configuration to JMeter if preferred.
  • Neutral
    • Locust has Docker and Helm packages available to easily deploy in the Cloud and distribute tests among different workers.
    • Locust is extensible via event hooks that seems quite easy to augment. Building new pages in the Locust UI or treating certain request differently should be not only possible but fairly good. Using it as a library is possible too for more customized approach.
  • Cons
    • No packaged solution in CDS in the Cloud (although we can simply use GitHub actions).
    • Developers in different regions might get different results if we don't run this in the Cloud.
    • It might go against the standard recommended tool in CDS and previous work.