Out of process interface to R using Redis

readme.md

Out of process interface to R using Redis

Trying out an idea for interfacing to an out-of-process instance of R, from any client, using Redis as a middleman.

The idea is to write the R code to a Redis key, and then publish the key as the data on a channel.

The R "server" listens on the channel, and executes a callback when data is received. The R code is then retrieved and evaluated.

Once the R server has completed the evaluation, it serializes the result to JSON, writes it to a Redis key and notifies the client.

In this implementation, the client is in Ruby, and uses the mlanett-redis-lock gem to implement the locking mechanism. Once the client has written the R code it publishes and awaits the response by taking out a lock. The R server clears the lock once the evaluation has completed.

A Out of process interface to R using Redis

Out of process interface to R using Redis

Gemfile

source 'https://rubygems.org'
gem 'redis'
gem 'mlanett-redis-lock', :require => 'redis-lock'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    mlanett-redis-lock (0.2.6)
      redis
    redis (3.2.0)

PLATFORMS
  ruby

DEPENDENCIES
  mlanett-redis-lock
  redis

r-server.r

library("rredis")
library("jsonlite")

# TODO: find a way to variablize the callback function name (which is also the channel key name)

poll <- TRUE

# create a new environment to use as the context
# for evaluating the received code, so that it is preserved
# between calls. this also serves as a sandbox for the evaluated code
env <- new.env(parent=baseenv())

channel <- function(data) {
  cat("Message received from channel:", data, "\n")

  if identical(data, "DONE") {
    poll <<- FALSE  # NOTE: assign global with "<<-" operator
  }
  else {
    # need to set the connection context for the read connection
    redisSetContext(read_context)

    # retrieve the R code
    code <- redisGet(paste(data, ":request", sep=""))
    # TODO: validate the code string

    tryCatch(
      {
        # evaluate the R code
        result <- eval(parse(text=code), env)

        # push the response as a JSON string
        redisSet(paste(data, ":response", sep=""), serializeJSON(result, digits=30))
      },
      error=function(condition) {
        # push the response as a JSON string
        redisSet(paste(data, ":response", sep=""), serializeJSON(condition))
      }
    )

    # notify the client
    #redisMulti()
    redisDelete(paste("lock:owner:", data, ":lock", sep=""))
    redisDelete(paste("lock:expire:", data, ":lock", sep=""))
    # redisPublish(paste(data, ":response", sep=""), "ready")  # for async
    #redisExec()
  }
}

# create connection for reads
read_context <- redisConnect(returnRef=TRUE)

# create connection for subscription
subscribe_context <- redisConnect(returnRef=TRUE)
redisSetContext(subscribe_context)

# this looks for a variable of the same name
# which is also the key of the subscription
# FIXME: would be nice to associate the channel key with the callback function
redisSubscribe(c("channel"))

while(poll)
{
  # need to set the connection context for redis
  # as the subscription callback sets it to the read context
  # since the same connection cannot be used for subscriptions
  # FIXME: the redis* methods should (optionally) take a connection as an argument
  redisSetContext(subscribe_context)

  redisMonitorChannels()
  Sys.sleep(0.05)
}

# clean up
redisSetContext(subscribe_context)
redisUnsubscribe(c("channel"))
redisClose()

# clean up
redisSetContext(read_context)
redisClose()

ruby-client.rb

require 'redis'
require 'redis-lock'

def exec(redis, command)
  key = SecureRandom.hex

  # create the lock, which the R server will clear
  redis.lock("#{key}:lock")

  # issue command
  redis.multi do
    redis.set "#{key}:request", command
    redis.publish 'channel', key
  end

  # wait for the R server to clear the lock
  # so the response can be retrieved
  redis.lock("#{key}:lock", :acquire => 20) do |lock|
    redis.get "#{key}:response"
  end
end

redis = Redis.new

puts exec(redis, 'x <- 1')

puts exec(redis, 'x')

puts exec(redis, <<-R)

  # Sample code from http://www.mayin.org/ajayshah/KB/R/html/b1.html
  # Goals: A first look at R objects - vectors, lists, matrices, data frames.

  # To make vectors "x" "y" "year" and "names"
  x <- c(2,3,7,9)
  y <- c(9,7,3,2)
  year <- 1990:1993
  names <- c("payal", "shraddha", "kritika", "itida")
  # Accessing the 1st and last elements of y --
  y[1]
  y[length(y)]

  # To make a list "person" --
  person <- list(name="payal", x=2, y=9, year=1990)
  person
  # Accessing things inside a list --
  person$name
  person$x

  # To make a matrix, pasting together the columns "year" "x" and "y"
  # The verb cbind() stands for "column bind"
  cbind(year, x, y)

  # To make a "data frame", which is a list of vectors of the same length --
  D <- data.frame(names, year, x, y)
  nrow(D)
  # Accessing one of these vectors
  D$names
  # Accessing the last element of this vector
  D$names[nrow(D)]
  # Or equally,
  D$names[length(D$names)]

  c(x, y, year, names, person, D)

R

# notify server to end...
# redis.publish 'channel', 'DONE'