Skip to content

Instantly share code, notes, and snippets.

@gleicon

gleicon/restmq.html

Created August 18, 2010 12:11
Show Gist options
  • Save gleicon/534494 to your computer and use it in GitHub Desktop.
Save gleicon/534494 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
restmq.html
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>RestMQ - Redis based message queue</title>
<link rel="stylesheet" href="style.css" type="text/css" media="screen">
</head>
<body>
<div id="wrapper">
<div id="mainheader">
<div class="inner">
<ul id="projects">
<a href="http://www.7co.cc" class="home" target="_blank">7co.cc</a>
<li><a href="http://7co.cc" target="_blank">URL Shortener</a></li>
<li><a href="http://pasteme.7co.cc" target="_blank">Pasteboard</a></li>
<li><a href="http://vuvuzelr.7co.cc" target="_blank">VuvuZelR</a></li>
<li><a href="http://github.com/gleicon" target="_blank">Source code</a></li>
<li><a href="http://zenmachine.wordpress.com/" target="_blank">Blog</a></li>
</ul>
</div>
</div>
<div id="header">
<h1 id="logo">RestMQ</h1>
<h2 class="desc">Redis based message queue</h2>
</div>
<div id="content">
<div class="columns">
<div>
<h3>About</h3>
<p>RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.</p>
<p>Redis is more than just a key/value db, and its data types provided support for this project.</p>
<p>The queues are created on the fly, as a message is sent to them. They are simple to use as a curl request can be.</p>
<h3>Example</h3>
<p>A http client (curl) post to /queue:</p>
<p>Point your browser to <a href="#">http://localhost:8888/c/test</a></p>
<p>Run:</p>
<pre><code>$ curl -X POST -d "queue=test&value=foobar" http://localhost:8888/</code></pre>
<p>Your browser is acting as a consumer to the queue. Using json encoded data it's easy to fit the data into a js based app.</p>
<p>Aside from the COMET consumer, there are xmlrpc methods, rest routes and the JSON protocol to manipulate queue items.</p>
<h3>COMET consumer</h3>
<p>There is a COMET based consumer, which will bind even if the queue doesn't already exists.</p>
<p>The main route is thru <span class="highlight">/c/&#60;queuename&#62;</span>, It can be tested using curl: <span class="highlight">$ curl http://localhost:8888/c/test</span></p>
<p>In another terminal, run:</p>
<pre><code>$ curl -X POST -d "queue=test&value=foobar" http://localhost:8888/</code></pre>
<p>This is the basic usage pattern for map/reduce (see examples).</p>
<p>See below on how to purge and disconnect all consumers from a queue, using DELETE.</p>
<h3>WebSocket consumer</h3>
<p>Now that cyclone has websockets support, check README.websocket to test it.</p>
<p>If you are using a browser or library which already supports websockets, you may take advantage of this interface</p>
<h3>REST services</h3>
<p>A queue can be accessed as <span class="highlight">/q/&#60;queuename&#62;</span>.</p>
<p>GET requests will dequeue an object.</p>
<p>POST requests inserts an object in the queue</p>
<p>DELETE requests will purgue the queue.</p>
<p>The usual pattern is listen in the COMET consumer (<span class="highlight">/c/&#60;queuename&#62;</span>) and insert new stuff at the REST route (POST <span class="highlight">/q/&#60;queuename&#62;</span>).</p>
<h3>JSON Protocol</h3>
<p>The HTTP route <code>/queue/&#60;queuename&#62;</code> uses the JSON protocol. Its the same protocol I've implemented for</p>
<pre>
<code>{
"cmd": "add",
"queue": "genesis",
"value": "abacab"
}
</code></pre>
<p>Creates the queue named "genesis" and inserts the string "abacab" as the message.</p>
<p>If we want to take that out of the queue, the payload would be like that:</p>
<pre>
<code>{
"cmd": "take",
"queue": "genesis"
}
</code></pre>
<p>The message can be formatted as a json object, so more complex data can be sent. It really mimics some of <a href="http://aws.amazon.com/sqs/" target="_blank">Amazon SQS</a> workings, because it's a simple queue.</p>
<p>For the first release it has:</p>
<ul>
<li>Select, EPoll or KQueue concurrency (depends on twisted)</li>
<li>Persistent queueing using Redis</li>
<li>Can work on pools, N daemons consuming from the same queues.</li>
<li>Cute ?</li>
<li>Small codebase</li>
</ul>
<h3>Dependencies</h3>
<ul>
<li>python, twisted</li>
<li><a href="http://github.com/fiorix/txredisapi" target="_blank">redis client</a>: git clone git://github.com/fiorix/txredisapi.git</li>
<li><a href="http://github.com/fiorix/cyclone" target="_blank">cyclone</a>: git clone git://github.com/fiorix/cyclone.git</li>
</ul>
<h3>Running</h3>
<p>Development environment:</p>
<pre><code>twistd -ny restmq_server.tac</code></pre>
<p>Production environment (needs epoll and proper user/group):</p>
<pre><code>twistd --pidfile=/var/run/restmq.pid --logfile=/var/log/restmq.log \
--reactor=epoll --uid=www-data --gid=www-data --python=restmq_server.tac</code></pre>
<h3>Tests</h3>
<pre><code>examples/test_rest.sh
examples/test_xmlrpc.py
python examples/test_comet.py
python examples/twitter_trends.py
python examples/test_comet_curl.py
python restmq_engine.py -h
</code></pre>
<h3>Files</h3>
<ul>
<li>restmq/dispatch.py: - a simple command dispatcher</li>
<li>restmq_engine.py: the redis abstraction layer to the queue algorithm</li>
<li>restmq_server.tac - the main app (a web service)</li>
</ul>
<h3>Credits</h3>
<p>
<strong>Thanks to (in no particular order)</strong>:<br>
Salvatore Sanfilippo for redis and for NoSQL patterns discussion. Alexandre Fiori for the redis client enhancement and patches.
</p>
</div>
</div>
</div>
<div id="footer">
<p id="copy">RestMQ - Redis based message queue</p>
</div>
</div>
</body>
</html>
Raw
style.css
* {
padding:0;
margin:0;
}
body {
background:#FDFDFD;
font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
font-size: 13px;
line-height: 1.5;
}
a {
color:#006FBB;
}
a:hover {
color:#002E4E;
}
h2 {
font-size:20px;
font-weight:normal;
margin:0;
}
p {
margin:1em 0;
}
h3 {
font-size:20px;
font-weight:bold;
border-bottom:1px solid #DDDDDD;
margin:40px 0 10px 0;
padding-bottom:5px;
}
pre {
background:#F1F1F1;
border:1px solid #DDD;
color:#000;
font-size:12px;
margin:10px 2%;
overflow:auto;
padding:5px 2%;
width:80%;
}
code, .code {
background:#F1F1F1;
border:1px solid #DDD;
padding:0 5px;
}
pre code {
border:none;
font-family:"Monaco","Consolas","DejaVu Sans Mono",monospace;
padding:0;
}
ul {
margin: 1em 0 1em 2em;
}
#header, #content, #footer, .inner {
margin:0 auto;
width:800px;
}
#mainheader {
background:#006FBB;
border-bottom:6px solid #005188;
color:#DDDDDD;
font-size:11px;
padding:8px;
}
#mainheader ul#projects {
line-height:1.2;
margin:2px 0 0 0;
list-style: none;
}
#mainheader ul#projects .home {
color:#FFF;
font-size:16px;
font-weight:bold;
padding: 0 10px 0 0;
text-decoration:none;
}
#mainheader ul#projects .home:hover {
text-decoration:underline;
}
#mainheader ul#projects li {
display:inline;
border-left:1px solid #005188;
}
#mainheader ul#projects li a {
color:#FFFFFF;
font-size:11px;
font-weight:normal;
padding:0 7px 0 8px;
text-decoration:none;
}
#mainheader ul#projects li a:hover {
color:#FFF;
text-decoration:underline;
}
#header {
margin:20px auto;
}
#header #logo {
color:#005188;
font-size:32px;
font-weight:bold;
}
#header h2.desc {
color:#777777;
font-size:20px;
margin-top:-6px;
}
#footer {
border-top:1px solid #DDD;
color:#777;
font-size:11px;
height:50px;
margin:18px auto 0;
padding:5px 0;
}
#footer #copy {
margin:0;
}
.desc {
color:#777;
font-size:16px;
}
.article {
border-bottom:1px solid #CCC;
padding:0 0 4px 0;
}
.article h2 {
font-size:26px;
}
.highlight {
background:#EDEDED;
}
.rule {
border-bottom:1px solid #EEE;
height:0;
margin:18px 0;
overflow:hidden;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment