a-b-testing.md

A/B Testing in Rails

Goals

A/B testing of sharing UI alternatives.

Requirements:

• User grouping
• Conversion metrics
• Rendering splits

Metrics tracking seems possible with Mixpanel for any solution. Not sure it is the ideal tool though.

Libraries

• Chili
• Split
• Vanity - Extremely similar to Split. Split seems slightly more feature complete and slightly more popular. Both are still pretty active repos.
• ABingo - Used to be big. No longer maintained

Chili

TL;DR

Repo Activity: Not much activity over the past year

User Grouping: #active_if callback allows a toggle injection. Need to create our own toggle logic.

Conversion Tracking: None

Rendering Splits: Does exceedingly well at this. Perhaps major overkill for the current use-case. Based on Rails Engine and Deface

Overview

Mainly handles modifying UI

Still may need something like Rollout to help split users into groups

Mounts a Rails::Engine per feature. This makes each feature isolated with it's own controllers, template overrides, and models.

Seems geared toward the toggling of larger more complex features.

Creates dirs for features under /lib/chili/<feature name>

For a 'Social' Feature:

Templates overrides: lib/chili/social_feature/app/overrides/<controller>/<template>/<feature partial>

Feature Modules # lib/chili/social_feature/lib/social_feature.rb

module SocialFeature
  extend Chili::Base
  active_if do
    # in context of controller action
    # some logic to determine user split"
  end
end

This seems like an on/off toggle per feature. To A/B two versions of the same feature we may have to make two "Features" in Chili and ensure the active_if block splits the users in a way that they will only have one or the other.

Pros / Cons

Pros:

• Clean way to implement the logic splits without polluting application
• easy throw away code when a choice is made

Cons:

• Still have to provide own user splits
• No conversions tracking
• Still need to re-integrate the winning feature implementation when the A/B test is over

Split

TL;DR

Repo Activity: Moderately active

User Grouping: Automatically splits users and ensures they always see the same option.

Conversion Tracking: Looks easy enough to set up and evaluate results. Uses redis for storage. Also can be configured to push metrics to Google Analytics.

Rendering Splits: Rudimentary #ab_test method to choose a value that can be used to toggle parts of the template or controller logic.

Overview

Inspired by Vanity and ABingo. Very similar to Vanity with a better UI.

Uses Redis as a datastore.

Includes conversions tracking.

User splits seem to be automatic and opaque. Also seems to try and keep even splits based on users who are actually visiting the site rather than some probabilistic hashing algorithm.

Seemingly rudimentary tool for implementing alternatives. This could end up being a good thing. Forces simple logic toggles

Seems geared more toward simple UI or UX flow alternatives.

Uses a simple #ab_test(test_name, [alternatives, &block]) to provide different alternatives for the user. Uses the same alternative for the user each time they see it. First alternative is the control/default, generally the option that currently exists.

Can call #finished(test_name) to track that the user has completed the desired behavior.

Select a choice: @share_template = ab_test('list_sharing', 'grouped_modal', 'prominently_exposed')

This could then be used in the template to decide which to render.

Mark a conversion as: finished('list_sharing'). No way to do this in JS so we would have to send an ajax request back to the server.

Pros / Cons

Pros:

• Baked-in conversions metric tracking with a Dashboard UI
• Has robot detection. Will not include bots in conversions
• Automatic user splits
• UI toggle happens in the app so integrating the feature at the end of the result just replaces the conditional with the winning feature choice

Cons:

• Opaque user splits
• Could get messy quickly for larger more intricate features