Toggling between machine learning models with a single attribute

Deliveroo Engineering

At Logistics Algorithms we use a lot of machine learning models to compute/predict time estimates for parts of the order journey, either before, while or after orders are placed. Everything we rollout is previously experimented via a traditional but slightly elaborated AB/testing (a blog post to some other time).

Sometimes we want to test a new model for a particular geographical region, other times we want a way to toggle between models while time goes by (ab testing), or we simply we want to quickly disable everywhere a particular model. While we use and advocate the use of global feature flags, we also need these more refined capabilities.

You can accomplish this pattern by using the datastore of your choice in which you store a setting attribute (for example in our case we might put this as a "Zone" attribute so we can test zones separately) that dictates which model should be used. The default setting should always be the "NoOp" scenario that just returns a "no-op" value. Avoiding testing nil is in general a good pratice, in fact most of strongly typed languages require you to be explicit about return values (for example None).

Managing this attribute should be also done safely, i.e. so it's always good pratice that values can be either be set via an UI or a task that can be invoked, never allow directly access to the database or a rails console for example.

A very simplistic ruby example would be:

def compute_model
  case model_version
  when "fancy_ai"
    FancyAI.new(...)
  else
    NoOp.new(...)
  end
end

class FancyAI
  ...
  def compute
    ...
  end
end


class NoOp
  ...
  def compute
    0 # of whatever default behaviour should be 
  end
end

In production sometimes we tend to do a dynamic lookup for a constant and avoid using the a case switch all together.

Hopefully this quick tip was useful!

Pedro @peduardocunha

pedrocunha/example.md

Deliveroo Engineering