How to load test persona on EC2

build your environment

Let's create two virtual machines:

$ scripts/deploy.js create loader -t c1.xlarge
... (lots of output) ...
$ scripts/deploy.js create loadee -t c1.xlarge
... (lots of output) ...
$ git push loadee HEAD:master
... (lots of output)...

Now you got loadee and loader, two machines with 8 processors each. Let's tweak loadee so that it can simulate a production deployment in a meaningful way.

disable SSL

In our production environment we handle SSL at a separate layer. Assume that this is infinitely scalable and disable it in the AWS environment so we can stress other parts of the system.

First kill the proxy:

$ ssh [email protected] 'forever stopall'

Now let's fiddle the router process from persona to run directly on port 8080 (which on awsbox is mapped to port 80 with fancy iptables magic so that non-root processes can bind the default HTTP port of 80). Apply this patch to ~app/post-update.js

-    process.env['PORT'] = 10000;
+    process.env['PORT'] = 8080;

And next, let's tell persona that it's actually not on SSL and that it needs to bind all interfaces with a patch to ~app/config.json:

- "public_url": "https://loadee.personatest.org",
+ "public_url": "http://loadee.personatest.org",
+ "router": { "bind_to": { "host": "0.0.0.0" } },

turn down bcrypt

because on aws the most you can get is about 8 cores, and persona deployment hardware is typically 24, on aws "request processing" daemons do much less work relatively on AWS instances. In order to shift the balance back, we can reduce the amount of work done in bcrypt to shift the relative balance and better simulate a production env. just patch config.json again:

+ "bcrypt_work_factor": 10,

create test users

See scripts/create_test_users.js for further details, but roughly this will get you a bunch of test users in your database that will be used by load generation:

$ CONFIG_FILES=whatver.json code/scripts/create_test_users.js 5000

enable "fake verification"

Load generation uses "fake verification" - that is a REST api is exposed that allows you to verify email addresses by calling an api rather than actually verifying email. This hack allows us to load test persona without worrying about email round trips. To enable this API, you need to define an environment variable. one way is to patch ~app/post-update.js

@@ -7,6 +7,8 @@ forever = require('forever'),
 path = require('path'),
 fs = require('fs');
 
 +process.env['BROWSERID_FAKE_VERIFICATION'] = 1;
 +
 function checkErr(msg, err) {
   if (err) {

restart the servers

ssh into loadee as the app user and simulate a git push/redeploy

$ cd git && ../post-update.js

Generate load

At this point your load target (loadee) is all set up and ready to go. Now let's hop on the machine we want to send the load from (loader) and fire away...

$ ssh [email protected]
(loader) $ git clone git://github.com/mozilla/browserid
(loader) $ cd browserid
(loader) $ npm install
(loader) $ bin/load_gen -o -s http://loadee.personatest.org -u 1/1000 -m 1000000

lloyd/load_testing_on_aws.md