README.md

Dynamic models in Ray Serve

This is an example of using Ray actor in Ray Serve to dynamically update the models. In the screencast below, I demostrated that you can dynamically register new model or update existing one with Python API. The updated model is immediately reflected in API call.

dynamic_model.py

import ray
from ray import serve

@ray.remote
class RegistryActor:
    def __init__(self):
        self.registry = dict()

    def put(self, key, value):
        self.registry[key] = value

    def get(self, key):
        return self.registry.get(key)

class DynmaicModel:
    def __init__(self):
        self.registry_actor = ray.get_actor("registry")

    async def __call__(self, http_request):
        name = http_request.query_params["model_name"]
        model_weights = await self.registry_actor.get.remote(name)
        # for this example, we will just return the model weights.
        return model_weights


def user_api_add_model(model_name, model_weights):
    # retrieve the registry actor
    handle = ray.get_actor("registry")

    # add the model to actor
    ray.get(handle.put.remote(model_name, model_weights))

def deploy_registry_actor():
    # create a global singleton actor
    RegistryActor.options(lifetime="detached", name="registry").remote()


def deploy_serve():
    client = serve.start(detached=True)

    client.create_backend("model", DynmaicModel)
    client.create_endpoint("model", backend="model", route="/predict")

if __name__ == "__main__":
    ray.init(address="auto")

    deploy_registry_actor()
    deploy_serve()



    

Execution log:

➜  ~/Desktop/ray/serve-demos/serve-registry ray start --head
Local node IP: 192.168.31.141
2021-01-18 16:02:41,044	INFO services.py:1174 -- View the Ray dashboard at http://localhost:8265

--------------------
Ray runtime started.
--------------------

Next steps
  To connect to this Ray runtime from another node, run
    ray start --address='192.168.31.141:6379' --redis-password='5241590000000000'

  Alternatively, use the following Python code:
    import ray
    ray.init(address='auto', _redis_password='5241590000000000')

  If connection fails, check your firewall settings and network configuration.

  To terminate the Ray runtime, run
    ray stop
➜  ~/Desktop/ray/serve-demos/serve-registry python dynamic_model.py
2021-01-18 16:02:43,474	INFO worker.py:655 -- Connecting to existing Ray cluster at address: 192.168.31.141:6379
(pid=96899) 2021-01-18 16:02:44,239	INFO http_state.py:70 -- Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-node:192.168.31.141-0' on node 'node:192.168.31.141-0' listening on '127.0.0.1:8000'
(pid=96926) INFO:     Started server process [96926]
(pid=96899) 2021-01-18 16:02:47,280	INFO controller.py:180 -- Registering route '/predict' to endpoint 'model' with methods '['GET']'.
➜  ~/Desktop/ray/serve-demos/serve-registry ipython
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 13:42:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ray

In [2]: ray.init(address="auto")
2021-01-18 16:02:57,022	INFO worker.py:655 -- Connecting to existing Ray cluster at address: 192.168.31.141:6379
Out[2]:
{'node_ip_address': '192.168.31.141',
 'raylet_ip_address': '192.168.31.141',
 'redis_address': '192.168.31.141:6379',
 'object_store_address': '/tmp/ray/session_2021-01-18_16-02-40_588415_96827/sockets/plasma_store',
 'raylet_socket_name': '/tmp/ray/session_2021-01-18_16-02-40_588415_96827/sockets/raylet',
 'webui_url': 'localhost:8265',
 'session_dir': '/tmp/ray/session_2021-01-18_16-02-40_588415_96827',
 'metrics_export_port': 51637,
 'node_id': '05c11c1c6e18d35828996ef6262061ba99e0ee3d3a84fd8a58cdaf1d'}

In [3]: from dynamic_model import user_api_add_model

In [4]: user_api_add_model("model_1", 100)

In [5]: !curl "http://localhost:8000/predict?model_name=model_1"
100
In [6]: user_api_add_model("model_1", 999)

In [7]: !curl "http://localhost:8000/predict?model_name=model_1"
  4 ## Dynamic models in Ray Serve
999
In [8]: user_api_add_model("model_2", 42)

In [9]: !curl "http://localhost:8000/predict?model_name=model_2"
42
In [10]: !curl "http://localhost:8000/predict?model_name=model_1"
999