gistfile1.txt

15:51 <fujin> yo guys 

15:52 <fujin> is there a way to automatically cluster Riak via similar method to rabbitmq_cluster.config?
A list of node@sname terms, if I'm not mistaken. 

15:52 <fujin> I'm levelling up benblack's shit. 

15:54 <fujin> oh 

15:54 <fujin> disregard 

15:54 <fujin> BRB testing 

16:04 <skeptomai> fujin: we've got something like that based on Ben's work. Let me 
know if you want it. Still alpha quality, used internally 

16:05 <fujin> skeptomai: please!!!! 

16:06 <fujin> skeptomai: I was haxing Ben's, trying to get alpha quality auto clustering going ;) 

16:12 <skeptomai> cool. I'll have mark anderson send it off to you. we didn't want to publish
yet because of rough edges 

16:52 <markanderson> fujiin: skeptomai said you were looking for a riak cookbook. 

16:54 <markanderson> fujin: I have a somewhat hacked version of the Benjamin Black's 
riak book that I developed for internal use, but 

16:55 <markanderson> fujin: I'd call it alpha quality. Glad to send it to you if you are
interested. 

16:56 <markanderson> fujin: email me (mark@opscode.com) if you'd like. 

17:17 <fujin> skeptomai: fear not, sekrets are safe with me 

17:17 <fujin> markanderson: thanks, will do 

17:17 <skeptomai> fujin: stellar. and of course, let us know of improvements. 

17:18 <fujin> cheers guys ++opscode 

17:56 <fujin> the init script uses chmod instead of chown 

18:03 <benblack> actually 

18:03 <benblack> i wrote a patch to riak 

18:03 <benblack> that arg was too lazy to push 

18:04 <fujin> hahaha 

18:04 <fujin> ;) 

18:04 <benblack> to let you specify a seed node for automatic clustering 

18:04 <fujin> ohh. 

18:04 <benblack> on which my cookbook depends in order to do that trivially 

18:04 <fujin> fuck me. 

18:04 <fujin> that's ridiculous? 

18:05 <benblack> https://gist.github.com/d096feb679a18a2d4481 

18:05 <benblack> (from april) 

18:05 <benblack> guessing it would not apply cleanly now. 

18:06 <fujin> Makes me want to punch myself 

18:06 <fujin> I thought this was a ring mode Internet-scale datastore 

18:06 <benblack> but, imo, easier to do that than hack up the cookbooks 

18:08 <markanderson> The gossip_seed parameter? Aha, I wondered what that was about. 
Of course, I thought it was easier to hack the cookbook than patch riak :-) 

18:08 <benblack> turns out... 

18:08 <benblack> ok 

18:09 <benblack> you have convinced me 

18:09 <benblack> i will make new patch for 0.12 

18:09 <benblack> and verbally abuse andy 

18:10 <fujin> Thanks Ben 

18:10 <fujin> <3 

18:10 <fujin> I can upvote or whatever. 

18:10 <fujin> Critical requirement for automation 

18:10 <seancribbs1> benblack: we verbally abuse andy regularly 

18:10 <benblack> seancribbs: i bet you find it as productive as i do 

18:11 <seancribbs> he laughs 

18:11 <benblack> NYHIIIII 

18:11 <seancribbs> so… you don't really need that 

18:11 <benblack> seancribbs: should i patch riak repo or riak_core repo? 

18:11 <seancribbs> all "join" does is send the ring to the other node 

18:11 <fujin> I don't really need a gossip seed? 

18:12 <benblack> seancribbs: how can it be done entirely from riak.config? 

18:12 <benblack> no running commands 

18:12 <seancribbs> benblack: you can't start riak without a command 

18:13 <drev1> benblack: patch riak_core 

18:13 <benblack> i mean joining new nodes to a ring 

18:13 <markanderson> I wouldn't mind seeing the gossip_seed config option either. 

18:13 <seancribbs> works great until the selected node is down 

18:13 <benblack> also true 

18:13 <markanderson> For our situation, it a separate join wasn't a big deal, as I wanted
to be able to check before joining automatically. 

18:13 <benblack> and the solution would be an array of them but this is better than nothing 

18:14 <benblack> and adding a new node when the seed node is unavailable is generally unlikely 

18:14 <fujin> gossip_seed is what I need ? 

18:14 <fujin> please advise 

18:14 <seancribbs> gossip_seed = not implemented in released version 

18:14 <fujin> is there some way to seed cluster members in advance 

18:15 <benblack> seancribbs and i are painting the bikeshed, fujin, please wait a moment 

18:15 <drev1> fujin: for now just run join after the node has started 

18:15 <seancribbs> fujin: use roles, pick one that's not you and join to i 

18:15 <seancribbs> it 

18:15 <seancribbs> yeah 

18:15 <seancribbs> what drev1 said 

18:15 <benblack> seancribbs: if i submit a patch against current code will you guys accept it? 

18:15 <seancribbs> that's not up to me 

18:15 <fujin> I'm not a retard guys, I understand I can manually cluster 'em. 

18:15 <markanderson> fujin: in the cookbook I sent you, the autoconf.rb attempts to sort this
out for you. 

18:15 <benblack> FUJIN NOT RETARD 

18:15 <fujin> markanderson: got ya. reviewing it now 

18:15 <benblack> FUJIN SMASH 

18:15 <fujin> It's a lot of code to go throuhg. 

18:15 <seancribbs> fujin: but i don't see how doing it with chef is not any different 

18:16 <seancribbs> system "/usr/sbin/riak-admin join #{other_node}" 

18:16 <seancribbs> boom. done. 

18:16 <fujin> (vs. the stuff I just spent hours doing) 

18:16 <fujin> is it a no-op if it's already joined? 

18:16 <seancribbs> basically 

18:16 <fujin> ? 

18:16 <fujin> like it wont' error 

18:16 <seancribbs> but you shouldn't be joining them after they are already joined 

18:16 <fujin> exactly the problem I don't want to have, and one that would be solved with a 
gossip seed 

18:16 <fujin> k thx 

18:17 <benblack> fujin: you can do the same check i do in that erlang patch 

18:17 <benblack> fujin: check for ring file 

18:17 <benblack> if it exists, don't join 

18:17 <fujin> cool 

18:17 <benblack> so, yes, this can be 2 lines of chef 

18:17 <fujin> bleh 

18:17 <benblack> point of the patch was to make it trivial regardless of automation 

18:17 <fujin> sure 

18:18 <fujin> ^ this 

18:18 <fujin> automatic localhost riak clusters like rabbitmq-multi is another win from gossip seed, no? 

18:18 <* fujin> not understanding potentially 

18:18 <benblack> if question is how in chef: what seancribbs said + a not_if on the ring file 

18:18 <fujin> quite confident with how to do it in Chef, just didn't understand the implications 
from an app level 

18:18 <fujin> and hadn't tested joining an already joined system 

18:19 <fujin> cheers guys 

18:19 <fujin> I 

18:19 <benblack> if question is how to generally do this regardless, the patch 

18:19 <fujin> I'd still upvote a gossip seed patch 

18:19 <fujin> personally 

18:19 <seancribbs> fujin: join == gossip my ring to this other node 

18:19 <seancribbs> if it's not the same, that'll invoke resolution/changing the claim 

18:19 <seancribbs> if it's the same, the other node says "yeah, that's what i have" 

18:22 <benblack> seancribbs: do you disagree that just checking for the ring file should 
be sufficient? 

18:23 <fujin> should I be building this from source? having continuous issues with the 
latest Ubuntu package 

18:24 <seancribbs> i do - but let me get some hard facts first ;) 

18:24 <seancribbs> fujin: do explain your ubuntu issues 

18:24 <benblack> fujin: i never trust any ubuntu things related to erlang 

18:24 <benblack> (nothing personal) 

18:24 <seancribbs> even though we package our own... 

18:24 <benblack> seancribbs: my paranoia runs deep 

18:25 <seancribbs> you must have been a sysadmin at some point ;) 

18:25 <fujin> hmm 

18:25 <fujin> trying to find logs 

18:25 <fujin> hold up 

18:27 <markanderson> Just FYI, we also had some troubles with the EE debian pkg, but building 
from source worked so I just did that. Can't remember what it was now. 

18:29 <seancribbs> ok, so if you're a customer you should put these issues in help.basho.com 

18:29 <seancribbs> then we'll get them attended to 

18:29 <benblack> what if you're just a frequent nuisance? 

18:29 <seancribbs> we have a special place for you, ben 

18:29 <benblack> hahaha 

18:30 <seancribbs> markanderson: if you don't have an account, email sean@basho.com 

18:32 <markanderson> Building from source had other advantages for our configuration anyways, 
and I didn't want to bug 
anyone over what wasn't a big deal for us. 

18:33 <seancribbs> don't consider it bugging us. consider it improving the product! 

18:33 <benblack> the way releases get packaged makes it soooo easy to roll your own 

18:33 <benblack> i *heart* it 

18:33 <seancribbs> and then maybe next time, you won't have to build from source because something 
is borked 

18:33 <seancribbs> it *hearts* you back, benblack 

18:34 <benblack> i got pulled into internal rabbit discussion about dep management/packaging 

18:34 <dysinger> benblack: I agree - re: not trusting pre-packaged debianisms of stuff I use 

18:34 <markanderson> gotta go; fujin feel free to email me with any questions about the riak 
cookbook hacks we did. 

18:35 <seancribbs> we break debian convention anyway. we're not sliced-and-diced 

18:36 <benblack> fujin: i think the simplicity of checking for a ring file and calling join 
if not is the way to go 

18:36 <benblack> sounds like markanderson made something a lot more complicated than that 

18:36 <dysinger> I have only just built and the tar.gz the rel/riak and distribute 

18:36 <seancribbs> only if you don't want to join again. doing so won't hurt anything 

18:36 <benblack> seancribbs: and won't give an error? 

18:36 <seancribbs> no, all it does is invoke gossip with that node 

18:36 <seancribbs> so… if you're running your chef recipe every minute, don't do it 

18:37 <benblack> ok, then cool 

18:37 <seancribbs> otherwise, no biggie 

18:37 <benblack> no need to check first, perfecto 

18:37 <benblack> i don't know what more complicated thing markanderson is doing in that 
cookbook 

18:37 <dysinger> yeah you can tell it to join 20 times 

18:38 <seancribbs> we have vclocks on rings now too 

18:38 <seancribbs> iirc 

18:38 <benblack> fujin: seancribbs with another 1-liner win. 

18:38 <fujin> ya 

18:38 <fujin> looks good 

18:38 <dysinger> when we were evaluating riak last year, I had a script that looked up 
ec2 metadata in the 'riak' security group and just joined everything 

18:38 <dysinger> promiscuous as a hoe 

18:38 <benblack> brute force 

18:39 <fujin> aww yea 

18:39 <seancribbs> Promiscuous join is promiscuous. 

18:39 <fujin> The troubles I was having related to the /etc/init.d/riak script (using chmod 
instead of chown) and then something related to enviroment / sudo / su. 

18:39 <fujin> Changing the riak users shell to /bin/bash, sudo -u riak -i 

18:39 <fujin> then riak start 

18:39 <fujin> works as expected 

18:40 <seancribbs> fujin: yeah the init script is not 100% 

18:40 <fujin> until I change the shell and login to the user, no combination 
of /etc/init.d/riak / /usr/sbin/riak workz 

18:40 <fujin> ah 

18:40 <fujin> well, that explains it I suppose 

18:40 <fujin> I see opscode are using Runit 

18:41 <seancribbs> fujin: you don't really need a separate daemon for riak 

18:41 <fujin> I'm going to try a source build 

18:41 <seancribbs> it daemonizes on its own 

18:41 <fujin> right 

18:41 <fujin> I launch loads of self-daemonizing stuff with bluepill 
(instead of runit, which is odd with it) 

18:42 <fujin> to waste 'em if they go mad 

18:42 <seancribbs> rtilder: we should OSS our packaging scripts at some point 

18:42 <seancribbs> at least the ones for riak OSS 

18:42 <seancribbs> then the debian nerds can fix it for us! 

18:44 <benblack> think i will make that change to the cookbook now, actually 

18:48 <fujin> benblack: which? 

18:48 <fujin> oh, the joining one? 

18:48 <fujin> that'd be nice, saves me doing it. heh 

18:52 <dysinger> fujin riak also can monitor itself somewhat 

18:52 <dysinger> AFAIK there is heartbeat monitoring in riak 

18:52 <fujin> sure. It's the somewhat I'm concerned about, and why I'll monitor 
it externally :) 

18:52 <fujin> can only rely so much 

18:52 <fujin> that was something down the track anyway, not immediate 

18:53 <dysinger> ruby (bluepill) monitoring an erlang node sounds upside-down to me 

18:54 <fujin> haha 

18:54 <fujin> I guess :) 

18:55 <fujin> I don't think monitoring particular responses from Riak's HTTP api is a bad idea. 

18:55 <fujin> Especially if it can give me an overall cluster state response or 
something like that 

18:55 <fujin> More than enough to consider automaticaly recycling a node 

18:57 <fujin> or repair, or similar 

18:57 <fujin> paging my ass 

18:57 <fujin> lol 

18:57 <fujin> BBZZZzzt 

18:58 <seancribbs> fujin/dysinger: we have heart disabled 

18:58 <seancribbs> it is bad juju 

18:58 <seancribbs> that is, it tries TOO hard to keep the node up 

18:58 <fujin> ah yes 

18:59 <fujin> then I'm sure some gentle external monitoring couldn't hurt at all, 
especially with heart disabled. 

18:59 <seancribbs> any of monit and friends would be fine 

19:00 <dysinger> aye - I've never used heart w/ riak 

19:00 <dysinger> it says "warning: there be dragons" in the config 

19:01 <fujin> ha 

19:01 <dysinger> it just feels wrong to have ruby monitoring riak 

19:01 <dysinger> (bluepill I mean)