Keystone performance benchmarking

auth.json

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
		    "user": {
			"domain": {
			    "id": "default"
			},
			"password": "password",
			"name": "admin"
		    }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "id": "default"
                },
                "name": "admin"
            }
        }
    }
}

authenticate.py

import os

from keystoneclient.v3 import client


HOST = os.environ.get('HOST', '23.253.157.126')
KEYSTONE_ENDPOINT = os.environ.get(
    'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)


project_scoped = client.Client(
    username='admin',
    password='password',
    project_name='admin',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('%s' % project_scoped.auth_token)

benchmark.sh

set -e

export HOST=23.253.157.242
export DB_HOST=23.253.157.126

echo "Ensuring Keystone is bootstrapped with the expected credentials..."
python bootstrap.py

echo "Truncating the keystone.token table..."
ssh root@$DB_HOST "mysql keystone -e 'truncate token;'"

echo "Creating a token to run benchmarks with..."
ADMIN_TOKEN=`python authenticate.py`
SUBJECT_TOKEN=`python authenticate.py`
echo "Admin token: $ADMIN_TOKEN"
echo "Subject token: $SUBJECT_TOKEN"

echo "Warming up Apache..."
ab -c 100 -n 1000 -T 'application/json' http://$HOST:35357/ > /dev/null 2>&1

echo "Benchmarking token creation..."
ab -r -c 1 -n 200 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > create_token
if ! grep -q 'Failed requests:        0' create_token; then
  echo 'WARNING: Failed requests!'
fi
git diff --color create_token | grep --fixed-strings ' [ms] (mean)' || true

echo "Benchmarking token validation..."
ab -r -c 1 -n 10000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN\n" -H "X-Subject-Token: $SUBJECT_TOKEN\n" http://$HOST:35357/v3/auth/tokens > validate_token
if ! grep -q 'Failed requests:        0' validate_token; then
  echo 'WARNING: Failed requests!'
fi
git diff --color validate_token | grep --fixed-strings ' [ms] (mean)' || true

echo "Benchmarking token creation concurrently..."
ab -r -c 100 -n 2000 -p auth.json -T 'application/json' http://$HOST:35357/v3/auth/tokens > create_token_concurrent
if ! grep -q 'Failed requests:        0' create_token_concurrent; then
  echo 'WARNING: Failed requests!'
fi
git diff --color create_token_concurrent | grep --fixed-strings ' [#/sec] (mean)' || true

echo "Benchmarking token validation concurrency..."
ab -r -c 100 -n 100000 -T 'application/json' -H "X-Auth-Token: $ADMIN_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://$HOST:35357/v3/auth/tokens > validate_token_concurrent
if ! grep -q 'Failed requests:        0' validate_token_concurrent; then
  echo 'WARNING: Failed requests!'
fi
git diff --color validate_token_concurrent | grep --fixed-strings ' [#/sec] (mean)' || true

bootstrap.py

import os

from keystoneclient.v3 import client


HOST = os.environ.get('HOST', '192.168.111.222')
KEYSTONE_ENDPOINT = os.environ.get(
    'KEYSTONE_ENDPOINT', 'http://%s:35357/' % HOST)


c = client.Client(
    token='ADMIN',
    endpoint=KEYSTONE_ENDPOINT + 'v3')

domain = c.domains.get('default')

try:
    role = c.roles.create(name='admin')
except Exception:
    role = c.roles.list(name='admin')[0]

try:
    group = c.groups.create(domain=domain, name='admin')
except Exception:
    group = c.groups.list(domain=domain, name='admin')[0]

c.roles.grant(group=group, domain=domain, role=role)

try:
    project = c.projects.create(domain=domain, name='admin')
except Exception:
    project = c.projects.list(domain=domain, name='admin')[0]

c.roles.grant(group=group, project=project, role=role)

password = 'password'
try:
    user = c.users.create(
        domain=domain, name='admin', password=password)
except Exception:
    user = c.users.list(domain=domain, name='admin')[0]

c.users.add_to_group(user=user, group=group)

services = c.services.list(name='Keystone', type='identity')
if services:
    service = services[0]
else:
    service = c.services.create(
        name='Keystone', type='identity')

endpoints = c.endpoints.list(service=service)
public_endpoints = [e for e in endpoints if e.interface == 'public']
admin_endpoints = [e for e in endpoints if e.interface == 'admin']

if not public_endpoints:
    public_endpoint = c.endpoints.create(
        service=service,
        interface='public',
        url=KEYSTONE_ENDPOINT + 'v3')

if not admin_endpoints:
    admin_endpoint = c.endpoints.create(
        service=service,
        interface='admin',
        url=KEYSTONE_ENDPOINT + 'v3')

unscoped = client.Client(
    username='admin',
    password='password',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Unscoped token: %s' % unscoped.auth_token)

project_scoped = client.Client(
    username='admin',
    password='password',
    project_name='admin',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Project-scoped token: %s' % project_scoped.auth_token)

domain_scoped = client.Client(
    username='admin',
    password='password',
    domain_name='Default',
    auth_url=KEYSTONE_ENDPOINT + 'v3')
print('Domain-scoped token: %s' % domain_scoped.auth_token)

create_token

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        886 bytes

Concurrency Level:      1
Time taken for tests:   14.596 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      553836 bytes
Total POSTed:           123400
HTML transferred:       177200 bytes
Requests per second:    13.70 [#/sec] (mean)
Time per request:       72.981 [ms] (mean)
Time per request:       72.981 [ms] (mean, across all concurrent requests)
Transfer rate:          37.05 [Kbytes/sec] received
                        8.26 kb/s sent
                        45.31 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    62   73  19.4     70     284
Waiting:       59   70  19.4     66     280
Total:         62   73  19.4     70     284

Percentage of the requests served within a certain time (ms)
  50%     70
  66%     71
  75%     71
  80%     72
  90%     80
  95%     82
  98%    123
  99%    214
 100%    284 (longest request)

create_token_concurrent

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        886 bytes

Concurrency Level:      100
Time taken for tests:   11.235 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      5538728 bytes
Total POSTed:           1234000
HTML transferred:       1772000 bytes
Requests per second:    178.02 [#/sec] (mean)
Time per request:       561.726 [ms] (mean)
Time per request:       5.617 [ms] (mean, across all concurrent requests)
Transfer rate:          481.46 [Kbytes/sec] received
                        107.27 kb/s sent
                        588.72 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   133  555 278.6    477    1862
Waiting:      125  542 274.6    463    1852
Total:        134  555 278.6    477    1862

Percentage of the requests served within a certain time (ms)
  50%    477
  66%    596
  75%    707
  80%    771
  90%    928
  95%   1107
  98%   1328
  99%   1465
 100%   1862 (longest request)

requirements.txt

python-keystoneclient

validate_token

This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        886 bytes

Concurrency Level:      1
Time taken for tests:   62.751 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      27620000 bytes
HTML transferred:       8860000 bytes
Requests per second:    159.36 [#/sec] (mean)
Time per request:       6.275 [ms] (mean)
Time per request:       6.275 [ms] (mean, across all concurrent requests)
Transfer rate:          429.84 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     5    6   0.6      6      39
Waiting:        4    5   0.6      5      38
Total:          5    6   0.6      6      39

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      7
  95%      7
  98%      7
  99%      7
 100%     39 (longest request)

validate_token_concurrent

This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 23.253.157.242 (be patient)


Server Software:        Apache/2.2.22
Server Hostname:        23.253.157.242
Server Port:            35357

Document Path:          /v3/auth/tokens
Document Length:        886 bytes

Concurrency Level:      100
Time taken for tests:   59.193 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      276200000 bytes
HTML transferred:       88600000 bytes
Requests per second:    1689.40 [#/sec] (mean)
Time per request:       59.193 [ms] (mean)
Time per request:       0.592 [ms] (mean, across all concurrent requests)
Transfer rate:          4556.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     6   59  79.7     17     470
Waiting:        4   55  79.4     14     469
Total:          6   59  79.7     17     470

Percentage of the requests served within a certain time (ms)
  50%     17
  66%     30
  75%     56
  80%    120
  90%    212
  95%    230
  98%    252
  99%    267
 100%    470 (longest request)

Hi there, thanks for the v. useful benchmarking code!

Just one q: I've been testing this on an openstack-ansible-deployed Pike environment and Keystone was rejecting all my tokens validation requests ('Invalid user token') until I removed the linefeeds from the request headers (eg from "X-Auth-Token: $ADMIN_TOKEN\n"). Is this an indication of a config issue? I'm trying to understand cause of subpar Keystone validation performance (~500ms single-threaded on lightly loaded metal controllers).