Preliminary results from talos experiments in the cloud, see: https://github.com/jonasfj/talos-experiments

aws-results.csv

Score is a relative measure of stability.

It's basically:
  for each test case:
    Rank configurations by lowest std.dev.
  for each configuration:
    compute score as mean over ranks assigned in test-cases
(this needs a little improvement)

c4.2xlarge/us-west-1/xorg/a4d43797b065,  12.090476190476192
c4.2xlarge/us-west-1/xvfb/a4d43797b065,  13.076190476190478
c4.2xlarge/us-east-1/xvfb/a4d43797b065,  13.252380952380955
m3.2xlarge/us-west-1/xvfb/a4d43797b065,  14.785714285714274
c4.2xlarge/us-east-1/xorg/a4d43797b065,  15.509523809523813
c3.2xlarge/us-west-1/xorg/a4d43797b065,  15.842857142857142
c3.2xlarge/us-west-1/xvfb/a4d43797b065,  15.866666666666655
c4.large/us-west-1/xvfb/a4d43797b065,    15.938095238095238
m3.2xlarge/us-west-1/xorg/a4d43797b065,  17.452380952380953
c4.large/us-west-1/xorg/a4d43797b065,    17.476190476190478
c4.xlarge/us-west-1/xvfb/a4d43797b065,   17.899999999999995
c4.xlarge/us-west-1/xorg/a4d43797b065,   18.12857142857143
c4.large/us-east-1/xorg/a4d43797b065,    19.271428571428572
c3.large/us-west-1/xvfb/a4d43797b065,    20.100000000000012
c4.large/us-east-1/xvfb/a4d43797b065,    20.27142857142856
g2.2xlarge/us-east-1/xorg/a4d43797b065,  20.547619047619047
c4.xlarge/us-east-1/xvfb/a4d43797b065,   20.790476190476188
c3.large/us-west-1/xorg/a4d43797b065,    20.847619047619048
c4.xlarge/us-east-1/xorg/a4d43797b065,   20.980952380952377
g2.2xlarge/us-east-1/xvfb/a4d43797b065,  22.56666666666667
g2.2xlarge/us-west-1/xvfb/a4d43797b065,  22.580952380952393
m3.2xlarge/us-east-1/xorg/a4d43797b065,  22.93809523809523
m3.2xlarge/us-east-1/xvfb/a4d43797b065,  22.990476190476194
c3.xlarge/us-west-1/xvfb/a4d43797b065,   23.257142857142867
c3.large/us-east-1/xvfb/a4d43797b065,    23.485714285714288
g2.2xlarge/us-west-1/xorg/a4d43797b065,  23.5952380952381
c3.xlarge/us-west-1/xorg/a4d43797b065,   23.8047619047619
c3.xlarge/us-east-1/xorg/a4d43797b065,   24.042857142857144
c3.xlarge/us-east-1/xvfb/a4d43797b065,   24.195238095238096
c3.2xlarge/us-east-1/xvfb/a4d43797b065,  24.43333333333333
c3.large/us-east-1/xorg/a4d43797b065,    24.76666666666667
c3.2xlarge/us-east-1/xorg/a4d43797b065,  24.809523809523814
r3.large/us-east-1/xorg/a4d43797b065,    24.814285714285717
r3.large/us-east-1/xvfb/a4d43797b065,    25.10952380952381
r3.xlarge/us-west-1/xvfb/a4d43797b065,   26.085714285714296
m3.xlarge/us-west-1/xvfb/a4d43797b065,   26.280952380952378
m3.xlarge/us-east-1/xvfb/a4d43797b065,   26.371428571428567
r3.xlarge/us-west-1/xorg/a4d43797b065,   26.604761904761915
m3.xlarge/us-west-1/xorg/a4d43797b065,   27.08571428571429
m3.xlarge/us-east-1/xorg/a4d43797b065,   27.938095238095233
r3.xlarge/us-east-1/xorg/a4d43797b065,   28.552380952380954
r3.xlarge/us-east-1/xvfb/a4d43797b065,   30.185714285714283
r3.large/us-west-1/xvfb/a4d43797b065,    30.77619047619047
r3.large/us-west-1/xorg/a4d43797b065,    33.06666666666667
m3.medium/us-east-1/xorg/a4d43797b065,   38.17142857142859
m3.medium/us-east-1/xvfb/a4d43797b065,   38.88571428571427
m3.medium/us-west-1/xvfb/a4d43797b065,   40.157142857142865
m3.medium/us-west-1/xorg/a4d43797b065,   40.31904761904763

These should be reproducible with the following config:

# Result Upload configuration
export AWS_DEFAULT_REGION='us-west-2';
export RESULT_BUCKET='jonasfj-talos-test-results';

# Test configuration
export TESTS='ts_paint:sessionrestore:sessionrestore_no_auto_restore:tpaint:a11yr:tp5o:tcanvasmark:tresize:tart:tsvgr_opacity:tsvgx:tscrollx:cart:dromaeo_css:kraken:dromaeo_dom:v8_7:tp5o_scroll';

# Test setup
export INPUT_URL='https://s3-us-west-2.amazonaws.com/jonasfj-talos-test-results/short-input.csv';
# Don't end with a slash:
export RESULT_PREFIX='results/3/aws/short-input';

S3 credentials might be required, but I suspect not as I made the bucket public.
Otherwise see config.env.template for how the config file should look.