services.md

Services are a standard feature of transient. a service is a transient program with a diferent codebase than the one of the calling program. the services are initiated stand alone or by the monitor service. Although they are transient haskell programs, they can execute internally wathever :they can do FFI or execute a shell. They can also be packaged into docker images.

monitorService:

-is an executable included in transient -it should run in each node of the cloud:

     monitorService -p start/thishost/3000    # first node 
     monitorService -p start/otherhost/3000/add/thishost/3000/y   # next node connected to the first
     ...

So all the monitors of all the nodes are connected.

The monitor is called form a transient program with

     requestInstance :: PIN -> Service -> NumInstances -> Cloud [Node]

     data Service= [("String","String")]
     
     
     node <- requestInstance "KEY"  servicedesc  1
     result <- callService' node param
     donext result
     ...

This assoc list contains the service description. Below there are an example of service description

requestInstance provision the service. it sends the specification of a service to the monitor. This install (if necessary) and run it (if necessary) if the service is running, it return the node immediately. if the monitor service executable is not running, requestInstace initiates it. the returned nodes are added to the list of known nodes. Instances are provisioned among the available nodes

callService' is the primitive that call the service with the param and return the result (or results if it returns an stream of events)

also:

   initService :: initService :: String -> Service -> Cloud Node

initService search for a node in the local list of nodes, instead of calling the monitor. if there is no such node, it call requestInstance. initService is used by callService

The monitor also can be called with

    callService :: PIN -> Service  ->  a  -> Cloud b

it is a direct form of call to the service. Internally it uses the two previous primitives

Services are independent programs. This is the code of a service that echo his parameter each second:

streamEchoService= 
                [("service","streamEchoService")
                ,("executable", "streamEchoService")
                ,("package","https://github.com/blabla/streamecho")]

if we put this code with the corresponding cabal file:

main=  keep . runCloud $ 
   runService streamEchoService 4000 $ serv 
   where
   serv param =  do 
       waitEvents $ threadDelay 1000000 >> return param
       

The body of the service can invoke anything, for example FFI to anything, invoke a shell etc.

The port 4000 is the default port but the monitor can assign any other port.

Besides installing from GIT it can also provision and execute docker images

[("service","other") ,("image", "http://hub.docker.com/.....")]

Now the differences between transient services and cloudshell

Functionality	transient services	cloudShell
config file	no	yes
install dependencies	no	yes
run non transient programs	within serivices	via configuration files
install before run	git-cabal/docker	script in configuration file
monitoring execution	no	yes
checkpoint/recovery	no	yes (not fully implemented)
web monitoring	not yet	not yet
transient services	yes	it is a particular case
clearance/respawn on fail	need some rework	need rework

CloudShell status: Need a redesign to embed some of his functionalities in transient services others as utility libraries

Couple questions:

• What scope does checkpoint/recovery encompass? The initialization of "services" or more general checkpointing?
  - It is storing the execution state and the resumption from this point on if the process is interrupted by an error. sorth of workflow. funflow like.
• With respect to monitoring execution, I know it's possible to collect stdout from a shell, does this mean something more like monitoring the hardware on which a service is run?
  • It is more like showing the execution log in the console/web page at run time. think for example on spawning a process in other node like cabal or stack that produce messages in the standard output.
• Since axiom can render arbitrary data from transient processes, what does web monitoring mean? A dedicated, standard UI?
  • Yes ,a general interface that monitor nodes, jobs in nodes and execution logs.
• clearance/respawn on fail: does this refer to errors in during execution or failover of the nodes themselves incase of hardwarefailure, etc?
  • All of these things.

funflow like.

• What is the scalability of this? If we have a workflow that processes 500gb of data, through 3 transformations across 3 nodes, will the checkpointing hold up? Are the "Checkpoints" stored on node-local disks?
  • theoretically that should be possible.
  • Yes. the checkpoints are stored locally