daemon-kit.log

<joelr1> kennethkalmer: hi
<kennethkalmer> no that is speed :)
 how goes?
 joelr1: ping
<joelr1> back
 sorry about that
<kennethkalmer> no worries
 so, let's see how I can help you figure this one out
 i need to understand your problem domain a bit better first
<joelr1> ok
 it's basic
 i have a server that i need to test by launching thousands or hundreds of thousands of clients
 there are two kinds of clients: socket and http
 think of them as emulating flash and javascript in the browser
 because the browser connects to the server normally but i cannot launch hundreds of thousands of browsers
<kennethkalmer> understandably
<joelr1> the simpler client, the socket one, connects and just sits there waiting for requests. occasionally, it receives a ping and responds with a pong (binary)
<kennethkalmer> ok, tcp sockets
 not unix sockets
<joelr1> yeah
 now, the http client establishes a connection and sits there in a long poll
 once the request ends it issues another one
 i'm strong with erlang but not with ruby
 :(
 so i can't quite figure out the best way to do this
<kennethkalmer> no worries, we'll get you there
<joelr1> ruby is good for dsls which is why i want to use it. easier for my client to script tests
<kennethkalmer> ok, can I make an assumption at this point?
<joelr1> hmm
 go ahead, of course
<kennethkalmer> cool
 you're hoping to use an army of nanites to stress test your setup using a clean DSL
 each nanite would perform differently depending on the resource called
<joelr1> yes
<kennethkalmer> ah
<joelr1> and each nanite would be a demon
 need to be a demon
<kennethkalmer> nanite is a no go then
<joelr1> oops
<kennethkalmer> no worries
 you're very close
<joelr1> why so?
<kennethkalmer> gonna be tough to really define
 luhm
 nanites are workers, meant for hard working
 hard == big, long running tasks
 however
 nanite uses AMQP as it's core
 which is what you're looking for
 how about a quick coffee ?
<joelr1> yes, i'm using rabbitmq
 what?
<kennethkalmer> a coffee?
<joelr1> coffee?
* kennethkalmer has this antic of offering guest coffee over IM, twitter, IRC, whatver is convenient
<joelr1> haha
<kennethkalmer> :)
 i'm gonna brew a cup quickly, back in two...
 but
 don't despair
 i know exactly how we're gonna solve your problem
<joelr1> ok
<kennethkalmer> daemon-kit + raw AMQP
<joelr1> thanks
 hmm
<kennethkalmer> right
 armed and ready
 you familiar with AMQP right?
<joelr1> well
 we are using rabbitmq with erlang
<kennethkalmer> i'm just starting to think out aloud here, we'll have to refine as we go along
<joelr1> but i still don't see how nanites don't fit. a demon task of the kind i envision is a long-running task, no?
<kennethkalmer> i don't think nanite will give you the throughput you need
 and nanite's dogpile effect means that an agent will be bombed with work for 15 seconds at a time
 this might, or might not work for you
<joelr1> i think it's fine
<kennethkalmer> ok
<joelr1> what throughput did you have in mid?
 mind
 when you said nanite is not gonna give me the throughput
<kennethkalmer> of message processing
 from the mapper's point of view
<joelr1> what kind of message processing? the only messages i have for nanite is connect to this server/port as a client
 then it should switch into
 demon mode and that's where fun starts
 then again, what i'm doing is fanning out packets from the server to an amqp channel
 and all those clients are really listening for those fanned out packets, either via sockets or http
 and the idea is to measure delivery latency to hundreds of thousands of clients
 so it doesn't have to be nanite per se
<kennethkalmer> well, let's step one back (for my sake at least)
 you have the service you wanna stress test
 you have workers, ruby daemons, that do the stress testing
<joelr1> i'm not stress-testing, i'm measuring latency
<kennethkalmer> sorry
<joelr1> the workers don't really stress, they just listen
<kennethkalmer> i meant the service gets stressed
<joelr1> and they are connected to a server that uses amqp internally
 they don't connect directly to amqp
 rather, they connect to a port either via http or sockets
<kennethkalmer> yes
 the workers mimic browsers
 thousands of browsers
<joelr1> right
 100%
<kennethkalmer> how do those workers know what to do?
 instruction point of view
<joelr1> i send them a request
<kennethkalmer> yes
<joelr1> connect to this port of this host, disconnect and listen to this packet
 to make sure they match what they are expecting with what they receive
<kennethkalmer> ok, and you're betting on the semantics of nanite for the command/response
 mapper sends work, gets response from agents
<joelr1> once they do receive, they are supposed to send receipt time back so that i can calculate latency
 not quite
 i mean
 yes for the 3 requests: connect, disconnect, request
 but i don't have any time requirements for those
 they don't need to be lightning quick
<kennethkalmer> ok, gotcha
<joelr1> what needs to be quick is the "demon mode"
<kennethkalmer> we were talking the same thing, just different words
<joelr1> i.e. what happens after the request is received
 nanite just triggers this
<kennethkalmer> ok, so that is your nanite agent
 in nanite speak it would be "calling a resource on an agent"
<joelr1> i guess
<kennethkalmer> my guess is you need two actors
 a socket actor, and a http actor
<joelr1> definitely
 and the nanite one, no?
<kennethkalmer> a nanite agent encapsulates one or more actors
<joelr1> what i don't understand is how those should join the nanite event machine loop
 the http and socket actors cannot talk the nanite protocol, though, theirs is a completely different protocol that has nothing to do with nantie
 right?
<kennethkalmer> thats for nanite to worry about, all you need is to have the actors "expose" methods that are callable
<joelr1> i cannot expose the socket and http actors via nanite, though
 meaning
 let me listen through
<kennethkalmer> ok
 you have the daemon-kit gem installed?
<joelr1> yes i do
<kennethkalmer> cool
<joelr1> an easier model for me to think of is that a nanite request "launches" a demon
 that talks its own protocol
 either http or socket
<kennethkalmer> oh no, nanite is the protocol so to speak.. i'll show you now
<joelr1> but i don't want to fork another ruby process
<kennethkalmer> you're not going to
<joelr1> don't get the "nanite is the protocol" part because my socket protocol has nothing to do with nanite and my server doesn't know what nanite is
<kennethkalmer> it doesn't need to
<joelr1> ok
<kennethkalmer> nanite becomes the controlling protocol
<joelr1> right
<kennethkalmer> nanite sends the instructions to your agents, and your agents send responses back
 nanite handles that, so you don't have to
 run this: daemon_kit joel -i nanite_agent
<joelr1> did
<kennethkalmer> cat lib/actors/sample.rb
<joelr1> where do i set up my client event loop, though, http or socket
 that's nanite, that part i understand
 i have it open in textmate
 the generated dir
<kennethkalmer> the sample actor exposes an "echo" method, callable via "/sample/echo"
<joelr1> where do i connect to the server, though, and set up my own event loop
<kennethkalmer> you don't setup your own event loop
 only one is needed
<joelr1> yes, echo would be my connect, disconnect and expect requests
<kennethkalmer> you setup your client each time echo is called
 inside the echo method
<joelr1> allright, then where do i join the eventmachine loop once i connect to my server?
 how do i add my socket to the nanite event machine loop?
 or should i do it differently?
<kennethkalmer> you following the normal eventmachine semantics for opening and closing sockets
 you just skip the EM.run {} portion
 since a reactor is already running
<joelr1> oh...
 so eventmachine is a singleton of sorts?
<kennethkalmer> definitely
 only one reactor can run at a time
<joelr1> cool
 so we are getting closer and closer
 :D
<kennethkalmer> you just wanna jump ahead too quickly ;)
 and forget nanite's involvement in the whole thing
 it handles communications
 end of story
 you need to focus on what happens when the resource is called
 i'm going to take a stab at quickly making an actor that pulls the google homepage via eventmachine
 hopefully that helps clear things up for you a lot
<joelr1> :D
 thanks kenneth!
<kennethkalmer> np :)
 love this kinda stuff
<joelr1> hmm... i wonder if eventmachine can replace my server erlang code
<kennethkalmer> em is amazing
 just testing quickly
 EUREKA!
 (no, I'm not running around naked)
 prepping gist for you
<joelr1> :D
 gist is like pastie, right?
<kennethkalmer> indeed
 http://gist.github.com/123675
 there you go
<joelr1> wow
 awesome
 how do you make lib/actors/google.rb appear above?
<kennethkalmer> i don't, first file added to the gist :)
 i think it sorts it alphabetically, not use
 *sure
<joelr1> i mean, how does gist learn about the path?
<kennethkalmer> oh, you enter a filename (optionally)
<joelr1> ah!
<kennethkalmer> you on twitter?
<joelr1> yes, wagerlabs
<kennethkalmer> and in Africa (well, almost)
 awesome
 I'm from South Africa btw
<joelr1> also, this is a request and disconnect, right? how would i respond to packets coming over an EM socket connection, though?
 cool
 several people i work with are from SA
 i think i understand
<kennethkalmer> brilliant
<joelr1> it's a request/response system!
<kennethkalmer> i hve not used em sockets directly yet, but the principle remains the same
<joelr1> so i set up a callback in a new thread or something like that
<kennethkalmer> step back quickly
 em is async
 but async doesn't help inside the nanite call
<joelr1> right
<kennethkalmer> thats why I block the current thread and wake it up once the async call is done
 so then the response can be built up and sent back to the mapper
 the response is simple the return value of your exposed method
 *simply
<joelr1> gotcha
 let me try to adapt this to long-running connections and socket bits
<kennethkalmer> please
 jump into #eventmachine as well
 the guys are very helpful
<joelr1> great
 thanks kenneth! i appreciate your help!
<kennethkalmer> i have to leave unfortunately, but I'll be online again in about 3 hours
<joelr1> me too. cheers!
<kennethkalmer> only a pleasure Joel
 drop in again, would like to follow your progress !
<joelr1> over and out
<kennethkalmer> ttyl !

google.rb

class Google
  include Nanite::Actor

  expose :search, :home

  def home(*args)
     start = Time.now
     response = {}

     conn = EM::Protocols::HttpClient2.connect 'google.com', 80

     th = Thread.current

     req = conn.get('/')
     req.callback{ |resp|
       response[:status] = resp.status
       response[:time] = Time.now - start
       th.wakeup
     }

     Thread.stop

     response
  end

  def search(query)
  end
end

REAME

INSTRUCTIONS FOR USE
====================

Generate the nanite agent

  $ daemon_kit /path/to/daemon -i nanite_agent

Copy the above google.rb into lib/actors/

Open libexec/daemon-daemon.rb and register the actor:

  agent.register Sample.new
  +agent.register Google.new

Run the actor

  ./bin/daemon run

Open another console, start nanite mapper in console mode

  $ nanite-mapper -i

Wait at least 15 seconds for the agent to be register, you should see something like this in your console:

INFO: registered: nanite-315476e6df9df98402a614aba9d751e7, {:tags=>[nil], :status=>1.20333333333333, :services=>["/sample/echo", "/google/search", "/google/home"]}

Now you can call your exposed resource:

  >> request('/google/home') { |r| puts r.inspect }

You should see this:

  {"nanite-315476e6df9df98402a614aba9d751e7"=>{:status=>301, :time=>0.02}}

Enjoy :)