gistfile1.txt

 
[4:20pm] evan: like the S3 keys
[4:20pm] claco: I prolly misunderstood
[4:20pm] claco: that's what I get for disappearing for a day
[4:20pm] evan: i'd like us to come up with something that works now
[4:20pm] evan: and we can change it later
[4:21pm] vertis: evan: yeah
[4:21pm] vertis: where are the S3 keys at this point
[4:21pm] claco: regardless, this process is always awesome given the end result
[4:21pm] evan: i've got them on my machine.
[4:21pm] vertis: okay
[4:21pm] vertis: so they're not on the app server yet
[4:21pm] claco: starting the same thing tomorrow for a new company. nerds love automation.
[4:21pm] evan: no
[4:22pm] evan: we had used a secret.rb before
[4:22pm] evan: that was an initializer
[4:22pm] evan: that injected the values into ENV vars
[4:23pm] jtimberman: so many native extensions.
[4:24pm] GitHub5 joined the chat room.
[4:24pm] GitHub5 left the chat room.
[4:24pm] GitHub5: [rubygems-aws] stevendanna opened pull request #91: Use secrets databag to store postgresql password. (master...data-bag-secrets)  http://git.io/4BHv0g
[4:24pm] ssd7: ^- lets start there and see where we get
[4:25pm] ssd7: you'll need to create a databag item like the one jtimberman linked to called "postgresql.json" (change the id to "postgresql") in the data_bags/secrets folder.
[4:25pm] vertis: great just need the host too
[4:26pm] ssd7: OK, so the host is currently hard coded here as localhost: https://github.com/stevendanna/rubygems-aws/blob/5bec1832f63907dcd15261abaf41a1b8afa9a5b9/chef/site-cookbooks/rubygems/recipes/rails.rb#L64
[4:26pm] ssd7: but you said earlier you were seeing something different on disk?
[4:27pm] evan: whats the way to tweak the nginx worker_process value?
[4:27pm] jtimberman: \m/-(><)-\m/ : 3 commands, 3 ec2 instances provisioned and dynamically discovered.
[4:27pm] evan: I see it defined in the nginx/attributes/default.rb
[4:27pm] jtimberman: and curl works.
[4:27pm] evan: where do I put my code to change it?
[4:27pm] evan: jtimberman: SO BOSS
[4:27pm] jtimberman: now to see about the changes made across the repo
[4:27pm] ssd7: evan: You can override the attribute in the relevant role.
[4:27pm] ben_h joined the chat room.
[4:28pm] vertis: ssd7: no only in the staging one
[4:28pm] ssd7: jtimberman: You still need the custom nginx work yes?
[4:28pm] jtimberman: i had to rerun chef to get the deploy + migrations to run, will investigate that too.
[4:28pm] jtimberman: ssd7: yes, that'd be rad.
[4:28pm] evan: ssd7: ah, I see
[4:28pm] evan: ok
[4:28pm] jtimberman: i'd like to dig into the app server not running migration
[4:28pm] jtimberman: the first time
[4:28pm] ssd7: vertis: OK, can you run with how you would add a "host" entry to that databag item and then use it and I'll work on getting jtimberman the new nginx
[4:28pm] vertis: yep
[4:28pm] vertis: will do
[4:29pm] evan: side topic: has anyone written a sudo that integrated with ssh-agent?
[4:29pm] evan: that would be so money.
[4:29pm] jtimberman: if you want to see, i've got it running on 54.234.125.32 (lb)
[4:30pm] jtimberman: evan: i've used sudo integrated with kerberos+ssh keys on AIX. it was pretty hawt. 
[4:30pm] evan: mmmm
[4:30pm] evan: sounds delicious.
[4:31pm] jtimberman: I mean, people bag on AIX, but from a "well, if we're going to manage this shit manually, this is kind of nice" perspective, I liked it 
[4:31pm] GitHub143 joined the chat room.
[4:31pm] GitHub143 left the chat room.
[4:31pm] GitHub143: [rubygems-aws] vertis pushed 2 new commits to master: http://git.io/_vpWHg
[4:31pm] GitHub143: rubygems-aws/master 5bec183 Steven Danna: Use secrets databag to store postgresql password.
[4:31pm] GitHub143: rubygems-aws/master 949cefd Luke Chadwick: Merge pull request #91 from stevendanna/data-bag-secrets...
[4:31pm] evan: just seems like with how we used sudo these days
[4:32pm] evan: having it use ssh-agent to validate the user would be better than passwords.
[4:32pm] evan: i'd have a few sudo keys loaded into my agent then
[4:32pm] jtimberman: evan: normally i use sudo with nopasswd.
[4:32pm] evan: ditto.
[4:32pm] jtimberman: some people think its bad, but that's what we did at SANS and they teach this stuff, so 
[4:32pm] evan: i'll probably make that change to the cookbooks
[4:32pm] jtimberman: it's already in my chef-server-refactor branch.
[4:32pm] postmodern joined the chat room.
[4:32pm] evan: nice.
[4:32pm] jtimberman: \m/
[4:35pm] teancom: evan: I use pam_ssh_agent_auth at one of my non-$DAYJOBS. It's pretty sweet, and super-simple to setup.
[4:35pm] vertis: okay
[4:35pm] vertis: just finishing off the app server password/host thing
[4:35pm] vertis: then will push straight to master
[4:35pm] vertis: cos thats the thing we're doing
[4:36pm] vertis:
[4:36pm] teancom: evan: libpam-sshauth on debian/ubuntu
[4:36pm] evan: sounds awesome
[4:38pm] teancom: truly the only downside is it doesn't work with mosh, because mosh doesn't forward keys (yet)
[4:39pm] evan: ok! it looks like the LB is ready to go.
[4:39pm] evan: all the configs are up
[4:39pm] vertis: one sec
[4:39pm] evan: and it's handles all the paths properly.
[4:39pm] vertis: almost ready to do app
[4:39pm] evan: k
[4:39pm] lsegal left the chat room.
[4:39pm] ssd7: jtimberman: https://github.com/jtimberman/rubygems-aws/pull/3
[4:39pm] ssd7: jtimberman: Can't really test it at the moment, but that should be the bulk of it
[4:40pm] jtimberman: ssd7: k
[4:41pm] jtimberman: ssd7: can you check for changes in the nginx config too?
[4:41pm] jtimberman: i want to make sure i can redeploy with the same bits they're using on prod now
[4:42pm] ssd7: ahh, yeah. Forgot about those bits as I didn't do them.  For those I think you just need to pull in the new templates but I'll check.
[4:42pm] jtimberman: those arei n the rubygems cookbook?
[4:42pm] envygeeks left the chat room. (Quit: Bye)
[4:43pm] ssd7: yup
[4:43pm] vertis: hmmm
[4:43pm] jtimberman: k
[4:43pm] jtimberman: that's simple enough 
[4:43pm] vertis: okay the data bag id doesn't matter right?
[4:44pm] jtimberman: for secrets?
[4:44pm] jtimberman: or ?
[4:44pm] vertis: for the way we're loading it
[4:44pm] vertis: yeah
[4:44pm] jtimberman: data bag id is how it looks for the item.
[4:44pm] ssd7: vertis: the choice doesn't matter but the id needs to match the file name without the .json
[4:44pm] vertis: okay
[4:44pm] ssd7: so if you file name is foobar.json, your id should be foobar
[4:44pm] jtimberman: the data_bag_item() method needs the bag name and the ID to load.
[4:44pm] vertis: thanks
[4:44pm] evan: quick question
[4:45pm] jtimberman: i don't know if the filename matters for solo, it might.
[4:45pm] jtimberman: evan: quick answer!
[4:45pm] evan: it looks like changing attributes in a role that cascade into a recipe
[4:45pm] evan: doesn't cause the recipe to perform a restart
[4:45pm] vertis: i can redeploy the app now
[4:45pm] jtimberman: evan: depends on the priority
[4:45pm] vertis: evan you can get the data bag of the server after I've done it once
[4:45pm] evan: cool.
[4:46pm] evan: how do I add the S3 keys do it?
[4:46pm] vertis: we can do the same way
[4:46pm] evan: vertis: actually, I'll just give them to ya
[4:46pm] evan: you've got access to the data_bag anyway
[4:46pm] evan: so it's the same diff
[4:46pm] vertis: hmmm
[4:46pm] vertis: we're going to need to change the that secret.rb right?
[4:46pm] evan: we're going to change the S3 keys next week too.
[4:47pm] evan: are you writing the secret.rb file?
[4:48pm] vertis: we can
[4:48pm] evan: that would be the easiest way to get the app up and going
[4:48pm] evan: because it expects to find those ENV vars
[4:48pm] vertis: yeah I can see that
[4:48pm] vertis: in s3.rb
[4:48pm] vertis: so I can get that working
[4:49pm] GitHub148 joined the chat room.
[4:49pm] GitHub148 left the chat room.
[4:49pm] GitHub148: [rubygems-aws] evanphx pushed 3 new commits to master: http://git.io/VZEzcg
[4:49pm] GitHub148: rubygems-aws/master 3f5adb0 Evan Phoenix: Remove stats calculation for now
[4:49pm] GitHub148: rubygems-aws/master 17a5704 Evan Phoenix: Bump worker_processes up to 8 for nginx
[4:49pm] GitHub148: rubygems-aws/master 12d8036 Evan Phoenix: Disable the max body size because of large gems
[4:49pm] vertis: just shoot me the keys somehow
[4:49pm] evan: k
[4:49pm] jtimberman: yall are using an EBS volume for the data base right?
[4:50pm] evan: yeah
[4:50pm] jtimberman: word.
[4:51pm] evan: vertis: ok, check rubygems-s3.txt in my home on the LB
[4:51pm] evan: that was the quickest way to get them to ya securely
[4:51pm] vertis: will do
[4:51pm] vertis: 5 mins
[4:51pm] ben_h: evan: vertis: any db replication planned? i've been working with it postgres streaming repl fair bit recently, would be happy to take care of it.
[4:51pm] vertis: just writing the code to put down that file
[4:52pm] evan: ben_h: that would be great
[4:52pm] evan: yes, I want to do it in the next couple weeks at least
[4:52pm] evan: i'm not going to add it as a wrinkle tonight though
[4:52pm] ben_h: i agree, best done in a week or two once everything has calmed down
[4:53pm] ben_h: it's fantastic tech though. http://theconversation.edu.au/health reload that a couple of times -- that's the lag from london to dallas 
[4:53pm] evan: very nice
[4:53pm] vertis: ben_h: awesome
[4:54pm] evan: I'd actually love to setup a separate rubygems.org in another zone entirely too.
[4:55pm] evan: another instance of the app and LBs
[4:55pm] evan: with replicated DB
[4:55pm] vertis: okay, I'm ready to run it
[4:56pm] vertis: just going to stop that extra postgres on the app server first
[4:56pm] ben_h: mm. that's what we have going at theconversation. a dedicated box that runs the lot in london, with the same thing in dallas. because the apps are local to the db, there's no app-level failover: just trigger the standby db and update DNS (and if the old master is still available, proxy to the new master for the DNS cache interval)
[4:57pm] ben_h: anyhow, i'll look over the existing configs and have a think about how best to incorporate it
[4:57pm] evan: ben_h: cool
[4:57pm] ben_h: existing rubygems configs that is
[4:57pm] evan: vertis: ok
[4:57pm] vertis: ready
[4:57pm] evan: i guess run it?
[4:57pm] evan: not sure what you're doing 
[4:57pm] vertis: fingers crossed
[4:57pm] vertis: database.yml being right after the run
[4:58pm] evan: cool 
[4:58pm] vertis: so we can actually deploy to the app server if you need to make changes
[5:00pm] evan: how'd it go?
[5:02pm] vertis: forgot something
[5:02pm] vertis: running again now
[5:02pm] vertis: (yes it was the actual keys)
[5:02pm] evan:
[5:03pm] vertis: failed
[5:03pm] vertis: hmm
[5:04pm] evan: vertis: are you doing the normal cap deploy from the rubygems.org repo to put code into /applications/rubygems/production ?
[5:04pm] ssd7: vertis: error?
[5:04pm] vertis: yeah
[5:04pm] vertis: it's all good
[5:04pm] vertis: just a variable that I changed
[5:05pm] vertis: has a create database, create user section
[5:05pm] vertis: which we don't want right now anyway
[5:06pm] jtimberman: hey so
[5:06pm] vertis: though that will be worth working out how we will do vagrant
[5:06pm] bhenerey left the chat room. (Quit: Leaving.)
[5:06pm] jtimberman: rubygems::rails does handle deploying
[5:06pm] ssd7: vertis: Yeah, in my vagrant branch I removed that and made it only create those on the server.
[5:06pm] evan: jtimberman: where at?
[5:06pm] evan: jtimberman: in the application block?
[5:06pm] jtimberman: ya
[5:06pm] jtimberman: that application block essentially does a capistrano style deploy
[5:07pm] jtimberman: it wraps up the deploy resource in chef
[5:07pm] jtimberman: looks like the recipe doesn't have a revision set, so it may just deploy master.
[5:07pm] jtimberman: so as long as the sha doesn't change :), it will be the same
[5:07pm] jtimberman: the chef-server-refactor uses a data bag item
[5:08pm] jtimberman: with the reivision set. so you just update the revision there and it will deploy that.
[5:08pm] jtimberman: as i understand it you're using a branch called production, so whenever that gets a push, it would redeploy next time chef runs.
[5:08pm] evan: ok, well, i'm a little confused
[5:08pm] vertis: just testing the hard way
[5:08pm] evan: but i'll let vertis finish his work before we get into it.
[5:08pm] jtimberman: ok 
[5:08pm] evan: I'll need to deploy rubygems.org soon
[5:08pm] vertis: yeah tried to write secrets.rb into a did that doesn't exist
[5:08pm] evan: do I have to do that through chef?
[5:09pm] evan: vertis: oops
[5:09pm] jtimberman: evan: should just run chef and it will deploy
[5:09pm] vertis: misread what the rails_root was
[5:09pm] vertis: well not really
[5:09pm] vertis: it's badly named
[5:09pm] vertis: rails_root/current
[5:09pm] evan: we need a couple more values in there
[5:09pm] evan: let me just give them to you now
[5:10pm] vertis: run successful
[5:10pm] vertis: lets hope that actually worked
[5:10pm] evan: check the secret.rb in my home on app01
[5:10pm] evan: we need to inject those values into secret.rb too
[5:10pm] vertis: evan: ?
[5:10pm] evan: yes?
[5:11pm] vertis: oh
[5:11pm] vertis: more values?
[5:11pm] evan: yeah
[5:11pm] evan: 3 other values
[5:11pm] • jtimberman should do a talk on this deployment style
[5:11pm] evan: vertis: if it's going to be an issue, maybe we should just put the secret.rb on the machine by hand for now.
[5:12pm] vertis: sorry was confused
[5:12pm] vertis: not a issue
[5:12pm] vertis: an*
[5:13pm] vertis: not a problem
[5:13pm] evan: vertis: you must have changed your rubygems.org repo you're using to deploy from
[5:13pm] evan: either that or it's not going through the normal cap deploy
[5:13pm] evan: since we've got the various tasks in lib/deploy/misc.rb that copy files from shared/config into current/config
[5:13pm] vertis: it doesn't do cap deploy no
[5:14pm] vertis: it does what the opscode guys setup
[5:14pm] vertis: what does it copy?
[5:14pm] evan: config/secret.rb
[5:14pm] evan: config/newrelic.yml
[5:14pm] evan: config/database.yml
[5:15pm] evan: jtimberman: so, everyone that deploys to the machine has to do it via chef now?
[5:15pm] jtimberman: evan: ya, just run chef-client
[5:15pm] evan: that makes it sound like a lot, it's not, it's 4 ppl
[5:15pm] evan: well, we're not on chef-server on this branch atm.
[5:16pm] jtimberman: its okay, chef server isn't required for that.
[5:16pm] evan: I don't see where the github repo is configured for where to pull the code from
[5:16pm] vertis: this will be taken care of
[5:16pm] vertis: 2 minute and those other variables are ready
[5:17pm] evan: ok
[5:17pm] evan: sorry, i'm getting ahead of things.
[5:17pm] claco left the chat room.
[5:18pm] ssd7: OK, so it sounds like we need to make sure that lib/deploy/misc.rb stuff gets run?
[5:18pm] gazoombo: evan: https://github.com/rubygems/rubygems-aws/blob/master/chef/nodes/app.rubygems.org.json#L10 ?
[5:18pm] vertis: okay
[5:18pm] evan: gazoombo: ah!
[5:18pm] vertis: running again
[5:18pm] evan: ack was ignoring .json files
[5:18pm] evan: silly ack.
[5:18pm] gazoombo:
[5:19pm] vertis: newrelic
[5:19pm] vertis: we will deal with shortly
[5:19pm] evan: k
[5:19pm] evan: i'll dump the info in my home dir again
[5:19pm] vertis: sure
[5:20pm] evan: ok
[5:20pm] evan: newrelic-key.txt
[5:20pm] vertis: okay
[5:20pm] vertis: that deployed
[5:21pm] vertis: how do you test that
[5:21pm] evan: so, what deployed?
[5:21pm] evan: the app with all those values set?
[5:21pm] vertis: wait
[5:21pm] vertis: not quoted
[5:21pm] evan: eh?
[5:22pm] vertis: the file I wrote
[5:23pm] vertis: there are no quotes around the strings
[5:23pm] evan: I don't see the file at all
[5:23pm] evan: where is it?
[5:23pm] evan: or are you testing this not on app01
[5:23pm] evan: ?
[5:23pm] vertis: yeah app01
[5:23pm] vertis: one sec I'll show
[5:23pm] vertis: sudo cat /applications/rubygems/production/current/config/initializers/secret.rb
[5:24pm] vertis: just rerunning now
[5:24pm] vertis: then newrelic
[5:25pm] vertis: i feel like that isn't restarting the rails app after a deploy
[5:25pm] vertis: opscode people?
[5:25pm] evan: vertis: so
[5:25pm] evan: the issue here is the initializer order
[5:25pm] evan: secret is running AFTER s3
[5:25pm] evan: so the values aren't seen
[5:25pm] vertis: ah
[5:26pm] vertis: thought it was in secret.rb before
[5:26pm] evan: just write secret.rb to config/
[5:26pm] evan: we have the various config/environment scripts to load it early
[5:26pm] vertis: just call the file 'attributes'
[5:26pm] vertis: or env
[5:26pm] vertis: okay
[5:26pm] vertis: seems hacky
[5:27pm] evan: it is.
[5:27pm] evan: it's very ungreat.
[5:27pm] vertis: config/secret.rb
[5:27pm] evan: but I don't want to change it right now.
[5:27pm] vertis: I'll help fix that later
[5:27pm] vertis: note it down
[5:27pm] evan: k
[5:28pm] vertis: config/secret.rb coming right up
[5:28pm] evan: ok, so our next thing to do is the delayed_job process
[5:28pm] vertis: i am so going to make the sudo passwordless
[5:28pm] vertis: already sick of my password
[5:29pm] evan:
[5:29pm] vertis: i stupidly made it complex
[5:29pm] evan: jtimberman: for running a delayed_job rake task
[5:29pm] evan: jtimberman: in the background
[5:29pm] vertis: that's run now
[5:29pm] evan: is there a stock way?
[5:29pm] vertis: should be in config/secret.rb
[5:30pm] evan: yep
[5:30pm] evan: looks great
[5:30pm] vertis: okay
[5:30pm] vertis: newrelic
[5:30pm] ssd7: evan: Can you elaborate on the delayed_job?  How did  it run before?
[5:30pm] evan: the last part is the delayed_job task
[5:30pm] evan: it updates the indexes
[5:30pm] evan: ssd7: sure
[5:30pm] evan: basically, we run "rake jobs:work"
[5:31pm] evan: out of the deploy dir
[5:31pm] jtimberman: does it run forever or once?
[5:31pm] evan: forever
[5:31pm] evan: it's managed by bluepill now.
[5:31pm] jtimberman: i manage services in runit 
[5:31pm] jtimberman: it's what your app server should be running under
[5:31pm] evan: and needs to be restarted on each deploy
[5:31pm] jtimberman: the unicorns
[5:31pm] vertis: foreman?
[5:31pm] jtimberman: what command starts it
[5:32pm] gazoombo: https://github.com/rubygems/rubygems.org/blob/master/config/pill.rb#L5-L19
[5:32pm] vertis: jtimberman: can you take care of that
[5:32pm] vertis: i will get new relic key in
[5:32pm] jtimberman: sure
[5:32pm] evan: i'd be happier to use runit
[5:32pm] jtimberman: give me a minute, my repo is mid-change 
[5:32pm] evan: than bluepill
[5:32pm] jtimberman: ^5!
[5:32pm] evan: it would just be much faster if you guys could drop it in there than me.
[5:33pm] ssd7: jtimberman: I'm guessing you could bang out the runit stuff faster than me if you are still up for it.
[5:33pm] jtimberman: ya its cool
[5:33pm] ssd7: sounds like we'll also need a notification on the application resource to restart it on deploy
[5:33pm] GitHub120 joined the chat room.
[5:33pm] GitHub120 left the chat room.
[5:33pm] GitHub120: [rubygems-aws] vertis pushed 1 new commit to master: http://git.io/s1R3pw
[5:33pm] GitHub120: rubygems-aws/master 80dc888 Luke Chadwick: Lots of secret settings, no create database
[5:33pm] ssd7: or use one of the callbacks
[5:33pm] evan: that part isn't critical right now
[5:34pm] vertis: the secret values are obviously not in that commit
[5:34pm] evan: i'd like to finish in the next 30 mins
[5:34pm] evan: finish being have push access reenabled.
[5:34pm] gazoombo: jtimberman: when you deploy with an application resource, what's the best way to symlink in your configs?
[5:34pm] jtimberman: i don't have a system really fit for testing, but hey, http://7.asset.soup.io/asset/2891/0263_438a_390.jpeg
[5:34pm] jtimberman: gazoombo: it handles the symlinks
[5:34pm] jtimberman: gazoombo: there's a callback for symlinking
[5:34pm] jtimberman: i'm going to fix this worker bit
[5:35pm] gazoombo: ah
[5:35pm] ssd7: gazoombo: check out: https://github.com/opscode-cookbooks/application
[5:35pm] ssd7: you can create a list of files for it to symlink essentially
[5:35pm] vertis: evan: where can I find a newrelic.yml
[5:35pm] vertis: to get this key into
[5:35pm] jtimberman: evan: this worker runs on *every* app server?
[5:35pm] jtimberman: like, if we had more than one
[5:35pm] jtimberman: (in fantasy webscale land)
[5:36pm] vertis: I mean I can find a stock on
[5:36pm] vertis: one*
[5:36pm] evan: vertis: let me grab it.
[5:36pm] evan: the whole thing
[5:36pm] vertis: yeah
[5:36pm] vertis: we can do that with an environment variable too
[5:36pm] vertis: actually just commit it to rubygems.org
[5:37pm] evan: vertis: ok, newrelic.yml in my home on the LB
[5:37pm] vertis: with <%= ENV['NEWRELIC_KEY'] %>
[5:38pm] evan: given the timeframe, i'd prefer to just drop it into shared and link it
[5:38pm] evan: i know thats not the clean way
[5:38pm] evan: but we need to get this going
[5:38pm] evan: i'm close to just running "rake jobs:work" in a screen session
[5:38pm] evan: just to get stuff going.
[5:38pm] ben_h left the chat room. (Quit: ben_h)
[5:39pm] vertis: will do
[5:40pm] evan: i'm not able to use rails console
[5:40pm] evan: should I be able to?
[5:40pm] vertis: you do the link
[5:40pm] evan: I get a database error
[5:40pm] evan:  /applications/rubygems/production/releases/fdf2195fdd506c0cbdf02fbf48c2b9e80535020f/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.11/lib/active_record/connection_adapters/postgresql_adapter.rb:1208:in `initialize': FATAL:  no pg_hba.conf entry for host "10.249.31.114", user "postgres", database "postgres", SSL off (PG::Error)
[5:40pm] vertis: hmmm
[5:40pm] vertis: borked
[5:40pm] vertis: one sec
[5:40pm] vertis: database missing
[5:40pm] evan: oh, the database.yml file has no.. eyah.
[5:41pm] vertis: in consistent naming
[5:41pm] vertis: one more sec
[5:41pm] vertis: actually
[5:41pm] vertis: just hand code it
[5:41pm] evan: just edit database.yml? ok
[5:42pm] vertis: done
[5:42pm] vertis: sorry
[5:42pm] evan: no prob
[5:42pm] vertis: i've fixed that in the chef recipe
[5:42pm] evan: k
[5:43pm] vertis: so link newrelic.yml
[5:43pm] vertis: and restart
[5:43pm] vertis: then we can test
[5:43pm] evan: did you move it over?
[5:43pm] evan: or should I?
[5:44pm] vertis: I haven't moved it
[5:44pm] vertis: where did you put it
[5:44pm] vertis: ah LB
[5:44pm] evan: it's in my home on app01
[5:44pm] vertis: why LB
[5:44pm] evan: i'm lazy.
[5:44pm] evan: just moved it to app01
[5:44pm] vertis: okay
[5:44pm] evan: in my home
[5:45pm] vertis: linked newrelic
[5:46pm] ben_h joined the chat room.
[5:46pm] evan: ok, I started up "rake jobs:work" in a screen session
[5:46pm] evan: i'm going to try to push a gem now.
[5:47pm] jtimberman: https://github.com/rubygems/rubygems-aws/pull/92
[5:47pm] GitHub80 joined the chat room.
[5:47pm] GitHub80 left the chat room.
[5:47pm] GitHub80: [rubygems-aws] jtimberman opened pull request #92: add a runit service for delayed job (master...runit-delayed-job)  http://git.io/Gnhm7Q
[5:47pm] jtimberman: ohrite, notification 
[5:48pm] vertis: I need to deal with newrelic being manual before we can use that
[5:48pm] vertis: thanks
[5:48pm] vertis: evan: how goes it?
[5:48pm] evan: building a test gem now.
[5:48pm] vertis: we still have the push notification up
[5:50pm] vertis: announcement
[5:51pm] evan: ok, got a server error.
[5:51pm] evan: from the rails app
[5:51pm] evan: ok, redis is busted.
[5:51pm] evan: Redis::CommandError (OOM command not allowed when used memory > 'maxmemory'.):
[5:52pm] evan: maxmemory was left set
[5:52pm] evan: i'm going to just comment it out and restart redis
[5:52pm] vertis: okay
[5:52pm] vertis: *shrug*
[5:52pm] ssd7: default["redis"]["maxmemory"]                  = "64mb"
[5:52pm] evan: yeah
[5:52pm] evan: change the cookbook to unset that value entirely
[5:54pm] jtimberman: the template has it set, will an empty value be okay for redis?
[5:54pm] evan: i doubt it.
[5:54pm] jtimberman: we can totally calculate the value to use based on the memory in the machine
[5:54pm] evan: thats fine
[5:55pm] jtimberman: how much do you want?
[5:55pm] evan: actually
[5:55pm] evan: set it to 8g
[5:55pm] evan: thats good enough for now.
[5:55pm] jtimberman: ha 
[5:56pm] evan: ah shit.
[5:56pm] evan: redis restarted empty
[5:56pm] ssd7: So, just as a sanity check, where is that EBS mounted and is redis started on the same machine?
[5:57pm] evan: it's mounted on app01
[5:57pm] ben_h left the chat room. (Quit: ben_h)
[5:57pm] evan: and redis is setup to run there
[5:57pm] ssd7: yup, cool.
[5:57pm] evan: but the redis config isn't pointed to the mount
[5:57pm] jtimberman: which role is that in?
[5:58pm] jtimberman: 'app'?
[5:58pm] evan: yep
[5:58pm] jtimberman: k
[5:58pm] evan: i'm not sure how it was running before
[5:58pm] evan: perhaps sam had changed the config by hand
[5:58pm] evan: and it has been overriden
[5:59pm] ssd7: default["redis"]["dir"]
[5:59pm] ssd7: evan: Is  /var/lib/redis_data still the correct mount point?
[6:00pm] evan: yeah
[6:00pm] vertis: i would say that's likely
[6:00pm] vertis: seems there were quite a few manual changes
[6:01pm] evan: ok, can you change it in the cookbook and run chef on app01
[6:01pm] ssd7: pull request incoming unless someone else has it
[6:02pm] evan: ssd7: do it!
[6:02pm] GitHub198 joined the chat room.
[6:02pm] GitHub198 left the chat room.
[6:02pm] GitHub198: [rubygems-aws] stevendanna opened pull request #93: Redis configuration changes for app role (master...redis-fixup-1)  http://git.io/qaoAhA
[6:02pm] ssd7: bah, typo, one sec
[6:03pm] evan: oops merged
[6:03pm] evan: had
[6:03pm] GitHub9 joined the chat room.
[6:03pm] GitHub9 left the chat room.
[6:03pm] GitHub9: [rubygems-aws] evanphx pushed 2 new commits to master: https://github.com/rubygems/rubygems-aws/compare/80dc888f06af...e89ef51ce950
[6:03pm] GitHub9: rubygems-aws/master 4a2e930 Steven Danna: Redis configuration changes for app role
[6:03pm] GitHub9: rubygems-aws/master e89ef51 Evan Phoenix: Merge pull request #93 from stevendanna/redis-fixup-1...
[6:03pm] vertis: wait
[6:03pm] evan: waiting.
[6:03pm] vertis: we'll have to relink newrelic
[6:03pm] GitHub60 joined the chat room.
[6:03pm] GitHub60 left the chat room.
[6:03pm] GitHub60: [rubygems-aws] jtimberman opened pull request #94: give redis lots of memory (master...redis-memory)  http://git.io/bkfbWg
[6:03pm] jtimberman: bah, think ssd was faster 
[6:04pm] evan: vertis: ok
[6:04pm] evan: thats fine
[6:04pm] evan: we can run without newrelic for now.
[6:04pm] vertis: yep
[6:04pm] GitHub35 joined the chat room.
[6:04pm] GitHub35 left the chat room.
[6:04pm] GitHub35: [rubygems-aws] stevendanna opened pull request #95: Fix typo in redis config. (master...redis-fixup-1)  http://git.io/owv6Cg
[6:04pm] ssd7: ok, that fixes the typo ^
[6:04pm] vertis: which ones am I mergine
[6:04pm] ssd7: g -> gb
[6:04pm] ssd7: vertis: my first one is already merged I think
[6:04pm] vertis: merging*
[6:04pm] evan: ssd7: I merged them.
[6:05pm] ssd7: so merge the second to fix typo.
[6:05pm] GitHub43 joined the chat room.
[6:05pm] GitHub43 left the chat room.
[6:05pm] GitHub43: [rubygems-aws] evanphx pushed 2 new commits to master: http://git.io/27sUWg
[6:05pm] GitHub43: rubygems-aws/master 1ea13de Steven Danna: Fix typo in redis config.
[6:05pm] GitHub43: rubygems-aws/master 9cc2a91 Evan Phoenix: Merge pull request #95 from stevendanna/redis-fixup-1...
[6:05pm] ssd7: cool
[6:05pm] vertis: nm
[6:05pm] evan: vertis: can you deploy there
[6:05pm] evan: since you just did it.
[6:05pm] vertis: yeah will do
[6:05pm] ssd7: I wonder how many hours of my life I've lost to the typos of myself and others.
[6:05pm] vertis: 2 mins
[6:07pm] evan: vertis: actually, i'll just run it
[6:08pm] evan: oh, you are
[6:08pm] evan: ok
[6:08pm] jtimberman: ssd7: missing commas and semi-colons
[6:08pm] vertis: hrmm
[6:08pm] vertis: i mean two mins it's running
[6:08pm] evan: ah
[6:08pm] ssd7: jtimberman: ?
[6:08pm] ssd7: ahh
[6:08pm] ssd7: yes.
[6:09pm] jtimberman: what?
[6:09pm] jtimberman: oh
[6:09pm] evan: ok, redis is reloading
[6:10pm] vertis: i relinked newrelic.yml
[6:11pm] evan: ok, blew up again on uploading to S3
[6:11pm] vertis: hmmm
[6:12pm] ssd7: credential problem?
[6:12pm] evan: checking now.
[6:12pm] vertis: S3_KEY and S3_SECRET are definitely set in config/secret.rb
[6:12pm] fromonesrc left the chat room. (Quit: fromonesrc)
[6:12pm] evan: it's something else
[6:12pm] evan: oh, right qrush rekeyed
[6:12pm] evan: one sec.
[6:13pm] ssd7: Yeah, cookbooks look good there.
[6:13pm] vertis: ssd7: they need cleaning up
[6:13pm] vertis: but they are functional
[6:13pm] vertis: they'll break for vagrant*
[6:14pm] ssd7: vertis: Yeah, in my mind having vagrant working can be a "next week after people can push gems" goal
[6:14pm] vertis: sure
[6:14pm] vertis: we needed it early on
[6:14pm] vertis: to get the app working
[6:14pm] vertis: but we're past that now
[6:15pm] vertis: evan: I can rerun if you get me the new keys
[6:15pm] vertis: or are you going to paste them in directly?
[6:16pm] evan: i'll give them to ya
[6:16pm] evan: one sec.
[6:16pm] gazoombo: vertis: I've just discovered that they're broken for vagrant now
[6:16pm] gazoombo:  oh well, fixable.
[6:18pm] • ssd7 wishes chef was wired up to push updates to irc.
[6:18pm] evan: vertis: ok, ~/rubygems-s3.txt on app01
[6:18pm] jtimberman: ssd7: there's a report handler for that
[6:18pm] DonOtreply joined the chat room.
[6:19pm] vertis: okay
[6:19pm] gazoombo: ssd7: https://github.com/portertech/chef-irc-snitch
[6:20pm] jtimberman: thats the one
[6:20pm] vertis: running
[6:22pm] evan: vertis: k
[6:22pm] gazoombo: wow, there are quite a few handy looking handlers: http://docs.opscode.com/essentials_handlers_available_handlers.html
[6:22pm] gazoombo: no pun intended
[6:23pm] evan: vertis: are we sure that the unicorns are being restarted?
[6:23pm] vertis: evan: no
[6:23pm] vertis: not sure
[6:23pm] evan: ok, i'll restart manually
[6:23pm] evan: to be sure.
[6:23pm] vertis: have to restart them to link
[6:23pm] vertis: wait
[6:23pm] evan: k
[6:23pm] vertis: they're restarted
[6:24pm] evan: k
[6:24pm] vertis: i linked the newrelic.yml again
[6:25pm] evan: well fuck yeah.
[6:25pm] evan: push works.
[6:25pm] ssd7: Nice.
[6:25pm] evan: ok, i'm repointing DNS to the LB
[6:26pm] evan: here we go people.
[6:26pm] gazoombo:
[6:26pm] ssd7: Let's make sure we get issues on github re the rough edges so we can clean them up over the next few days.
[6:26pm] vertis: woot
[6:27pm] evan: ok, DNS repointed.
[6:27pm] • ssd7 updates his hosts file
[6:28pm] gazoombo: ;; ANSWER SECTION:
[6:28pm] gazoombo: rubygems.org.          60      IN      A      54.245.255.174
[6:28pm] jtimberman: 1,259,533,358 downloads
[6:28pm] jtimberman: \m/
[6:28pm] evan: the download counters are busted atm.
[6:28pm] evan: which is fine.
[6:29pm] ssd7: evan: Do we know why?  Happy to dig in some more, got a bit more energy here.
[6:29pm] evan: the access log on the LB is a constant stream of happy
[6:30pm] evan: ssd7: I turned it off
[6:30pm] vertis: the homepage says hosted by bluebox still
[6:30pm] ssd7: ahh
[6:30pm] ssd7: cool
[6:30pm] evan: it requires the stat-update service
[6:30pm] evan: which we haven't setup yet.
[6:30pm] jtimberman: the sponsor stuff is missing images
[6:30pm] vertis: jtimberman: not for me
[6:30pm] evan: not for me
[6:31pm] jtimberman: ok cool
[6:31pm] ssd7: looks good here
[6:31pm] ssd7: in other words, "Fix your shit jtimberman"
[6:31pm] vertis: the message saying you can't push is still up
[6:31pm] evan: ok, pushed another gem
[6:32pm] evan: everything looking good.
[6:32pm] ssd7: This is a beautiful sight: http://status.rubygems.org/
[6:32pm] evan: we need to take the banner down.
[6:32pm] jtimberman: ssd7: clealry.
[6:32pm] evan: vertis: how safe are we to deploy again?
[6:32pm] jtimberman:
[6:32pm] vertis: evan: safe why?
[6:32pm] evan: not going to break the site 
[6:32pm] vertis: new relic just hast to be relinked every time
[6:32pm] vertis: no
[6:32pm] vertis: don't think so
[6:33pm] ssd7: Want to work that into the cookbook so we can deploy without manual steps.  We should also verify the unicorn restart.
[6:33pm] vertis: ssd7: just need to get it into an environment variable instead
[6:33pm] vertis: ssd7: work on the unicorn start
[6:33pm] vertis: restart rather
[6:34pm] evan:
[6:34pm] evan: where is that "Pushing new gems" banner coming from...
[6:34pm] vertis: qrush said
[6:34pm] jtimberman: evan: do you need delayed_job?
[6:34pm] vertis: Announcement.first
[6:34pm] evan: i will
[6:34pm] vertis: i think
[6:34pm] jtimberman: I sent a PR for the runit service
[6:34pm] vertis: want me to check
[6:34pm] evan: I'm running it in a screen session atm.
[6:34pm] evan: jtimberman: I'd like to vet it first
[6:34pm] evan: we can do that this weekend
[6:34pm] jtimberman: evan: ok
[6:35pm] evan: screen session is a fine shortterm mechanism
[6:35pm] evan: besides, it helps me debug if things are working with it.
[6:35pm] jtimberman: haha 
[6:35pm] jtimberman: http://jtimberman.housepub.org/blog/2012/12/29/process-supervision-solved-problem/
[6:35pm] jtimberman: i ranted! 
[6:35pm] ssd7: fwiw, I review jtimberman's pull looks sane to me.
[6:35pm] evan: hehehe
[6:35pm] jtimberman: like a month ago 
[6:35pm] jtimberman: anyway its cool
[6:35pm] jtimberman: it'll go into this "chef server" stack
[6:36pm] evan: ssd7: if you vet it, go ahead and merge it.
[6:36pm] vertis: evan
[6:36pm] vertis: Announcement.first
[6:36pm] vertis:   Announcement Load (0.8ms)  SELECT "announcements".* FROM "announcements" LIMIT 1
[6:36pm] vertis: => #<Announcement id: 8, body: "Pushing new gems is temporarily disabled due to sec...", created_at: "2013-01-30 15:55:36", updated_at: "2013-01-30 15:55:36">
[6:36pm] jtimberman: which is pretty rockin' at the moment soon as i fix the load balancer nginx config
[6:36pm] vertis: want me to delete?
[6:36pm] evan: yep
[6:36pm] evan: I was about to do that 
[6:36pm] vertis: done
[6:36pm] evan: bingo.
[6:36pm] evan: ok, who was a gem they can push
[6:36pm] evan: ?
[6:36pm] evan: I need secondary confirmation.
[6:37pm] vertis: actually the sponsor logos have disappeared for me now
[6:37pm] vertis: I can probably find one
[6:37pm] evan: I'd like one of y'all to push a gem to verify
[6:37pm] evan: before I tweet about it.
[6:37pm] gazoombo: I still see the logos
[6:37pm] vertis: can push in a minute
[6:38pm] vertis: evan: building one now
[6:38pm] evan: oh
[6:38pm] evan: someone pushed
[6:38pm] vertis: https://rubygems.org/gems/offline
[6:38pm] vertis: lol
[6:38pm] vertis: or that
[6:38pm] evan: no, you go.
[6:38pm] evan: continue
[6:38pm] vertis: okay
[6:39pm] jtimberman: Successfully registered gem: ubuntu_ami (0.4.1)
[6:40pm] evan: boom.
[6:40pm] vertis: pushing
[6:40pm] vertis: damn
[6:40pm] vertis: rubygems creds
[6:41pm] evan: ok, the word is out.
[6:41pm] jtimberman: woo
[6:41pm] evan: thanks so much for all your hard work over the past 2+ days everyone
[6:41pm] evan: I can't thank you enough
[6:42pm] evan: we've still got a few things to get sewn up
[6:42pm] evan: but everything is looking great
[6:42pm] vertis:
[6:42pm] jtimberman:
[6:42pm] evan: i'll be in and out this weekend
[6:42pm] vertis: trying to push again
[6:42pm] jtimberman: \o/
[6:42pm] vertis: pushed
[6:42pm] ssd7: Re unicorn restart.  It will send a HUP by default
[6:42pm] evan: we can start to clean up those rough edges
[6:42pm] evan: we just bought ourselves a lot of time
[6:42pm] evan: a great base to build on
[6:43pm] vertis: ssd7: cool
[6:43pm] evan: if y'all want to look at your handy work
[6:43pm] evan: follow @rubygems
[6:43pm] evan: it tweets on every push
[6:43pm] evan:
[6:44pm] jtimberman: hahahaha
[6:45pm] vertis: picked up quite a few followers over the last few days
[6:45pm] jtimberman:
[6:46pm] evan: I've added dns for the lb and app
[6:46pm] evan: lb01-aws.rubygems.org
[6:46pm] evan: and app01-aws.rubygems.org
[6:46pm] evan: probably easier to remember than the IPs
[6:46pm] ssd7: vertis: Speeking of that...twitter handle?
[6:46pm] jtimberman: i fixed the error where I wasn't following evan 
[6:46pm] vertis: vertis
[6:47pm] ssd7: lol
[6:47pm] vertis: yours?
[6:47pm] • jtimberman <- everywhere 
[6:47pm] ssd7: SteveDanna
[6:48pm] gazoombo: gazoombo
[6:48pm] vertis: jtimberman: me too
[6:48pm] ben_h joined the chat room.
[6:48pm] vertis: hey ben_h
[6:49pm] ben_h: o/
[6:49pm] vertis: \o
[6:49pm] ben_h: how's it all going?
[6:49pm] vertis: we're live
[6:49pm] ben_h: Boom
[6:50pm] ben_h: hadn't checked the twitters  top work all.
[6:50pm] evan: ok everyone, I'm headed to bed
[6:50pm] evan: great job!
[6:50pm] vertis: have a good sleep evan
[6:50pm] evan: see y'all on the flip side
[6:50pm] ben_h: sleep well sir
[6:50pm] vertis: do you do alerting for rubygems.org?
[6:51pm] evan: there is some, yes
[6:51pm] evan: but I need to get it sorted out
[6:51pm] vertis: you can add me if you want
[6:51pm] evan: it doesn't alert enough
[6:51pm] jtimberman: nagios all the things!
[6:51pm] vertis: for australian hours
[6:51pm] evan: in other words, it won't alert if push breaks
[6:51pm] jtimberman: "There's a cookbook for that." 
[6:51pm] • vertis shoots jtimberman
[6:51pm] evan: vertis: that would be amazing
[6:51pm] vertis: evan set it up tomorrow
[6:51pm] jtimberman: nagios is less bad when its fully automated with chef 
[6:51pm] evan: vertis: next week, we should go through and get all the monitoring straight
[6:51pm] vertis: I'll watch it for now
[6:52pm] evan: thanks bud
[6:52pm] vertis: sleep well
[6:53pm] evan: nite
[6:54pm] ssd7: Hrmm. I can't seem to file issues.  I was going to record some of the todo's we've still got.
[6:55pm] vertis: ssd7: chrome?
[6:56pm] coderanger: evan: Pingdom was nice enough to donate us a free 30-check account for PyPI et al, might want to ask them 
[6:56pm] coderanger: just nice as an offsite thang
[6:56pm] ssd7: vertis: firefox.
[6:56pm] gcoderre joined the chat room.
[6:58pm] scoutcamper is now known as scoutcamper|away.
[6:59pm] gazoombo: does github still support firefox?
[6:59pm] gazoombo: kidding
[6:59pm] jtimberman: ssd7: I was able to create an issue.
[6:59pm] gazoombo: kind of
[6:59pm] jtimberman: gazoombo: only if your laptop is a MBPr
[6:59pm] jtimberman:
[7:02pm] ssd7: oh, derp. nvm, my issue got posted.
[7:02pm] jtimberman: four times?
[7:03pm] ssd7: no, just once 
[7:03pm] vertis: i wasn't able to post a gist before
[7:03pm] vertis: it was being very strange
[7:03pm] vertis: button greyed out
[7:04pm] jtimberman: reload the page and the buttons *sometimes* are available
[7:04pm] jtimberman: i've seen that happen too
[7:04pm] jtimberman: don't know why
[7:04pm] vertis: it's all ajax
[7:05pm] ssd7: Anyway, I'll be around, but I think I'm done for the nice unless there is something we definitely don't want to leave hanging at the moment.
[7:05pm] ssd7: HUGE thanks to all these people and more: https://github.com/jtimberman/rubygems-aws/graphs/contributors
[7:05pm] vertis: having dinner
[7:05pm] ssd7: s/nice/night/
[7:06pm] ssd7: opps, wrong link https://github.com/rubygems/rubygems-aws/graphs/contributors
[7:07pm] jtimberman: heh - coderanger deleted more code than he added. efficient!
[7:07pm] ssd7: By my code metric, he wins 
[7:07pm] jtimberman:
[7:07pm] jtimberman: I lose, added 243 lines, removed 96 
[7:07pm] ssd7: but not as much as Mike English
[7:08pm] jtimberman: oh - that's my fork... heh
[7:12pm] GitHub152 left the chat room.
[7:12pm] GitHub152: [rubygems-aws] stevendanna opened pull request #98: Remove user data from base role.  This now lives in the users data bag. (master...issue-83)  http://git.io/imCs-Q
[7:12pm] GitHub152 joined the chat room.
[7:12pm] vertis: okay
[7:14pm] GitHub112 joined the chat room.
[7:14pm] GitHub112 left the chat room.
[7:14pm] GitHub112: [rubygems-aws] vertis pushed 2 new commits to master: http://git.io/alcweg
[7:14pm] GitHub112: rubygems-aws/master cc527ac Steven Danna: Remove user data from base role.  This now lives in the users data bag....
[7:14pm] GitHub112: rubygems-aws/master 9375965 Luke Chadwick: Merge pull request #98 from stevendanna/issue-83...
[7:14pm] vertis: ssd7: you're just trying to catch up
[7:15pm] envygeeks joined the chat room.
[7:16pm] ssd7: vertis:  I'm just glad people can post gems.  Also, as someone who writes Chef eveyday lots of these cookbooks still set off my "O_o"-feelings.  But, we'll get it all into shape.
[7:17pm] ssd7: vertis: Also, great work pulling people together hear and pushing this over the finish line.
[7:17pm] vertis: ssd7: should show you our chef repos at work
[7:17pm] ssd7: s/hear/here/
[7:17pm] vertis: they make people depressed
[7:17pm] vertis: learned a lot about that over the last few days
[7:17pm] vertis: librarian / berkshelf
[7:18pm] vertis: might play with them at work this week
[7:18pm] jtimberman: vertis: to go along with the chef serve refactoring bit
[7:18pm] jtimberman: vertis: i'm planning to also have a hangout
[7:18pm] jtimberman: walk through how shit's gonna work 
[7:18pm] vertis: sure
[7:19pm] vertis: gonna wait for evan?
[7:19pm] jtimberman: will record for the youtubes too
[7:19pm] jtimberman: yeah
[7:19pm] jtimberman: it's going to be next week 
[7:19pm] vertis: okay
[7:19pm] jtimberman: at least, thats my plan.
[7:19pm] jtimberman: because i'm  close to having the stack ready
[7:20pm] jtimberman: doesn't have EBS yet, but the other stuff is working.
[7:20pm] vertis: i didn't realise that rubygems.org was in such a bad state ops wise
[7:20pm] vertis: before this
[7:20pm] jtimberman: see also the issue i opened
[7:20pm] envygeeks: come again say what?
[7:20pm] envygeeks: they built the instance without ebs from the get go?
[7:20pm] vertis: envygeeks: no
[7:20pm] jtimberman: envygeeks: no
[7:20pm] jtimberman: envygeeks: *I* didn't
[7:20pm] vertis: they manually mounted
[7:20pm] jtimberman: ya
[7:20pm] vertis: ebs volumes
[7:20pm] gcoderre left the chat room. (Quit: gcoderre)
[7:21pm] jtimberman: we have piles of code for automating it
[7:21pm] vertis: jtimberman: I'm going to spend time fixing up the architecture
[7:21pm] envygeeks: well that's not too bad, I was just scared they used the other crappy storage AWS has that would have made it hard to do the nightly snapshots
[7:21pm] vertis: looks like with ben_h's help
[7:21pm] vertis: replication for db
[7:22pm] vertis: multiple app servers behind an ELB
[7:22pm] jtimberman: ELB isn't necessary 
[7:22pm] jtimberman: chef-server-refactor automatically adds new web servers.
[7:22pm] vertis: *shrug*
[7:22pm] ben_h left the chat room. (Read error: Connection reset by peer)
[7:23pm] ben_h joined the chat room.
[7:34pm] jtimberman: i'm gonna sleep, got the elastic_ip bit working
[7:34pm] zerstorer joined the chat room.
[7:37pm] gazoombo: 'night jtimberman
[7:47pm] ben_h_ joined the chat room.
[7:47pm] ben_h left the chat room. (Read error: Connection reset by peer)
[7:47pm] ben_h_ is now known as ben_h.
[7:47pm] ben_h: yep i've got replication covered
[7:51pm] vertis: awesome
[7:55pm] sferik left the chat room. (Quit: ["Textual IRC Client: www.textualapp.com"])
[7:56pm] johndouthat: hey, nice
[8:03pm] workmad3 joined the chat room.
[8:05pm] envygeeks left the chat room. (Quit: Bye)
[8:05pm] DonOtreply left the chat room. (Quit: Computer has gone to sleep.)
[8:14pm] ben_h left the chat room. (Quit: ben_h)
[8:17pm] ben_h joined the chat room.
[8:24pm] Boxcar21 left the chat room. (Quit: Leaving...)
[8:26pm] vertis: johndouthat: 
[8:39pm] johndouthat left the chat room. (Quit: johndouthat)
[8:47pm] seanhorn left the chat room. (Quit: Leaving.)
[8:52pm] workmad3 left the chat room. (Ping timeout: 244 seconds)
[9:10pm] zerstorer left the chat room. (Remote host closed the connection)
[9:37pm] ben_h left the chat room. (Remote host closed the connection)
[9:38pm] ben_h joined the chat room.
[9:56pm] paulmooring left the chat room. (Ping timeout: 245 seconds)
[10:07pm] ben_h left the chat room. (Quit: ben_h)
[11:06pm] ben_h joined the chat room.
[11:41pm] teancom left the chat room. (Remote host closed the connection)
[11:58pm] vertis: anyone around?
[11:59pm] workmad3 joined the chat room.
[12:03am] stevenhaddox|afk: morning vertis
[12:03am] stevenhaddox|afk: just got caught up on backchat
[12:06am] • stevenhaddox|afk has to shower and get kids fed
[12:06am] stevenhaddox|afk is now known as stevenhaddox.
[12:07am] stevenhaddox: Anything you want me to look at from the issues in particular? BTW, still no images for me either.
[12:08am] • stevenhaddox goes to take care of kids
[12:09am] workmad3 left the chat room. (Ping timeout: 264 seconds)
[12:24am] stevenhaddox: Also, huge thanks to everyone who's stayed up countless hours to tackle this! You're all amazing!
[12:28am] teancom joined the chat room.
[12:41am] teancom left the chat room. (Remote host closed the connection)
[12:56am] workmad3 joined the chat room.
[01:04am] stevenhaddox left the chat room. (Ping timeout: 264 seconds)
[01:04am] btm left the chat room. (Ping timeout: 248 seconds)
[01:07am] bhenerey joined the chat room.
[01:11am] btm joined the chat room.
[01:14am] postmodern left the chat room. (Quit: Leaving)
[01:17am] sjaq joined the chat room.
[01:20am] sjaq left the chat room. (Remote host closed the connection)
[01:22am] teancom joined the chat room.
[01:30am] workmad3 left the chat room. (Ping timeout: 264 seconds)
[01:34am] ben_h left the chat room. (Quit: ben_h)
[01:51am] workmad3 joined the chat room.
[01:59am] bhenerey left the chat room. (Quit: Leaving.)
[02:01am] teancom left the chat room. (Remote host closed the connection)
[02:24am] workmad3 left the chat room. (Ping timeout: 252 seconds)
[03:00am] mr_ndrsn joined the chat room.
[03:01am] mr_ndrsn: hey
[03:01am] teancom joined the chat room.
[03:05am] teancom left the chat room. (Remote host closed the connection)
[03:06am] teancom joined the chat room.
[03:14am] teancom left the chat room. (Remote host closed the connection)
[03:20am] paulmooring joined the chat room.
[03:20am] bhenerey joined the chat room.
[03:23am] workmad3 joined the chat room.
[03:26am] fromonesrc joined the chat room.
[03:26am] Mab879 left the chat room. (Quit: quit)
[03:27am] teancom joined the chat room.
[03:29am] gcoderre joined the chat room.
[03:43am] gcoderre left the chat room. (Quit: gcoderre)
[03:53am] dontbecold_ joined the chat room.
[03:59am] teancom left the chat room. (Remote host closed the connection)
[04:00am] teancom joined the chat room.
[04:02am] pearkes left the chat room. (Quit: IRCRelay - http://ircrelay.com)
[04:03am] pearkes joined the chat room.
[04:07am] teancom left the chat room. (Remote host closed the connection)
[04:08am] bhenerey left the chat room. (Quit: Leaving.)
[04:10am] teancom joined the chat room.
[04:10am] workmad3 left the chat room. (Ping timeout: 245 seconds)
[04:21am] maledale joined the chat room.
[04:29am] teancom left the chat room. (Remote host closed the connection)
[04:40am] seanhorn joined the chat room.
[04:40am] Boxcar21 joined the chat room.
[04:52am] calmyournerves joined the chat room.
[04:53am] justincampbell joined the chat room.
[04:56am] lnxchk left the chat room. (Quit: lnxchk)
[04:57am] lnxchk joined the chat room.
[05:13am] DonOtreply joined the chat room.
[05:21am] scoutcamper|away is now known as scoutcamper.
[05:21am] johndouthat joined the chat room.
[05:21am] stevenhaddox joined the chat room.
[05:24am] stevenhaddox: Meh. Internt outage is finally over...
[05:24am] teancom joined the chat room.
[05:24am] stevenhaddox: Anything I can do to pitch in?
[05:24am] qrush: time to start improving it 
[05:39am] DonOtreply left the chat room. (Quit: Computer has gone to sleep.)
[05:47am] dontbecold_ left the chat room. (Quit: dontbecold_)
[05:48am] dontbecold_ joined the chat room.
[06:00am] teancom left the chat room. (Remote host closed the connection)
[06:13am] phlipper_: good day all!
[06:13am] phlipper_ is now known as phlipper.
[06:19am] andrewhubbs joined the chat room.
[06:19am] andrewhubbs left the chat room. (Client Quit)
[06:21am] scoutcamper is now known as scoutcamper|away.
[06:24am] seanhorn left the chat room. (Quit: Leaving.)
[06:24am] maledale left the chat room. (Quit: maledale)
[06:26am] andrewhubbs joined the chat room.
[06:29am] maledale joined the chat room.
[06:30am] snooc joined the chat room.
[06:30am] seanhorn joined the chat room.
[06:38am] teancom joined the chat room.
[06:38am] DonOtreply joined the chat room.
[06:46am] DonOtreply left the chat room. (Quit: Computer has gone to sleep.)
[06:52am] workmad3 joined the chat room.
[06:53am] seanhorn left the chat room. (Quit: Leaving.)
[06:53am] seanhorn joined the chat room.
[06:53am] DonOtreply joined the chat room.
[06:58am] teancom left the chat room. (Remote host closed the connection)
[07:06am] bhenerey joined the chat room.
[07:06am] gazoombo: ssd7, vertis: not sure if you're around, but one thing that was mentioned needing to fixed was placing newrelic.yml - any reason that can't be done by the app resource?
[07:07am] mr_ndrsn left the chat room. (Quit: mr_ndrsn)
[07:07am] qrush: ssd7: gazoombo vertis how can we do that without publishing the api key to github?
[07:07am] vertis: hey good morning everyone
[07:08am] vertis: qrush: how did you do that
[07:08am] mr_ndrsn joined the chat room.
[07:08am] qrush: vertis: do what? 
[07:08am] vertis: I woke up 2 mins ago
[07:08am] mr_ndrsn: Hey all
[07:09am] vertis: qrush, gazoombo: we have a secrets.rb + data bag
[07:09am] gazoombo: I thought there was a solution for some of the other secrets involving .gitignored data bags
[07:09am] vertis: newrelic.yml can take ENV['BLAH']
[07:09am] vertis: in place of the key
[07:09am] gazoombo: yeah - any reason NR id different?
[07:09am] gazoombo: *is
[07:10am] vertis: the yaml file should be put in the rubygems.org repo
[07:10am] qrush: OK lets do that
[07:10am] vertis: I can create a pull request if you want?
[07:10am] mr_ndrsn: YAML.  It's hard not to hate it at this point.  
[07:10am] gazoombo: vertis: that and runit for the delayed jobs worker were the two big things done by hand right?
[07:11am] qrush: Let's just use XML instead
[07:11am] qrush: trolololoo
[07:11am] mr_ndrsn:
[07:11am] gazoombo: jtimberman has a pull req. waiting for feedback re: runit
[07:12am] mr_ndrsn: Does anyone have logs for 2/1?
[07:14am] vertis: mr_ndrsn: I can have a look
[07:14am] mr_ndrsn: It's not a big deal, keep your momentum.
[07:16am] gazoombo: mr_ndrsn: irc logs? I have them in irccloud - not sure if they have a way to export them...
[07:17am] mr_ndrsn: gazoombo: yes, I see them for rubygems-exploit at the link in status, but none for was for 2/1, and I was hoping to catch up.
[07:23am] stevenhaddox: mr_ndrsn: jas, I've got a gist from someone that has a good portion of 2/1 I think
[07:23am] stevenhaddox: https://gist.github.com/036ef634c92902b8c680
[07:23am] mr_ndrsn: stevenhaddox: That would be great.  Thank you!
[07:23am] vertis: qrush: just cloning rubygems.org now, and I'll create a pull request
[07:23am] qrush: thanks
[07:23am] stevenhaddox: mr_ndrsn: only covers midnight to 07:43 apparently.
[07:23am] stevenhaddox: I may have logs for most of the rest of the day though
[07:24am] mr_ndrsn: stevenhaddox: Well, since I slept thru that, it's pretty relevant then.  
[07:24am] stevenhaddox: mr_ndrsn: 1637-2359 (EST): https://gist.github.com/d95736d8d4b346924652
[07:25am] stevenhaddox: or EDT, whatever it is now
[07:25am] mr_ndrsn: Thanks