englishm · February 2, 2013 20:27
diff --git a/gistfile1.txt b/gistfile1.txt
 07:41:00  → jasoares, btucker, benchMark, bradland and stevenhaddox joined  ← yasho99 left  ⇐ teancom_ and kek quit  ↔ technicalpickles, bhenerey and jcaudle popped in  ↔ dontbecold_, kek (was kek_), zerstorer and yfeldblum nipped out  •  vertis → vertis-sleeping, dwradcliffe → dwradcliffe  
 09:37:33 <stevenhaddox> I have to run a few errands this morning and then I'm free to help out for several hours today. Anything I can pitch in on?
 09:38:05  → mbulat and teancom joined  
 09:41:09 <samkottler> morning
 09:41:22 <dwradcliffe> morning guys
 09:41:42 <dwradcliffe> stevenhaddox: https://github.com/rubygems/rubygems-aws/issues
 09:42:41 <dwradcliffe> samkottler: vertis has been setting up some jmeter tests to help with performance testing; will finish when he wakes up
 09:42:43 <stevenhaddox> dwradcliffe: duh. I should've remembered to check there. Has everything that's actively being worked been assigned out already?
 09:43:00 <bradland> excellent suggestion on the licence [sic] in #72
 09:43:18 <stevenhaddox> Also, it looks like the logs turned off around midnight UTC so I'm not sure if anything important happened in between then and now we should be aware of from backchat?
 09:43:21 <samkottler> hey
 09:43:39 <dwradcliffe> bradland: yep. I've got a commit for that ready as soon as I find out who the copyright holder is
 09:43:40  ⇐ kek quit  ↔ stevenhaddox nipped out  
 09:44:43 <samkottler> dwradcliffe: we are gonna have to wait for evan to find that out, but I suspect it's rubycentral
 09:44:58 <dwradcliffe> stevenhaddox: don't think much happened overnight, although without the log I can't prove it :)
 09:45:11 <samkottler> stevenhaddox: I can give you a log - one second
 09:45:13 <stevenhaddox> dwradcliffe: touché
 09:45:17 <stevenhaddox> samkottler: thanks
 09:45:56 <stevenhaddox> With the verification stuff completed it feels like the last big piece of urgency is obviously the new site launch, API, etc.
 09:46:13 <samkottler> stevenhaddox: times are in EST - https://gist.github.com/036ef634c92902b8c680
 09:46:15 <stevenhaddox> So I'll do whatever I can to help out here. I was kind of watching all three rooms yesterday to see where I could pitch in.
 09:46:19 <stevenhaddox> thanks samkottler.
 09:46:35 <samkottler> stevenhaddox: no problem
 09:46:39  → blasterpal and kek joined  
 09:49:16 <bradland> you guys are probably already aware of this, but it looks like the gem metadata parsing vuln fix hasn't been merged yet: https://github.com/rubygems/rubygems.org/pull/516
 09:50:18 <dwradcliffe> bradland: I've had it in the back of my mind. good to remember though
 09:52:59  → Boxcar21 joined  ⇐ btucker quit  
 09:54:17 — dwradcliffe reading the transcript
 09:54:31 <dwradcliffe> things got a little OT last night :)
 09:55:12  → btucker, technicalpickles, eggmaster, bhenerey and andrewhubbs joined  ↔ dwradcliffe nipped out  
 10:10:34 <stevenhaddox> dwradcliffe: at least it was interesting OT stuff ;)
 10:11:25 <stevenhaddox> okay, about to head out and run errands, I'll catch up when I return. Do we know if someara's work has been discussed to be merged in yet / what work needs to happen to update his readme?
 10:11:38 <blasterpal> morning folks
 10:11:40 <dwradcliffe> stevenhaddox: idk
 10:12:04 <dwradcliffe> stevenhaddox: last I heard, we were waiting until post-ship
 10:12:12 <stevenhaddox> seems like the Vagrant stuff may be falling farther and farther behind too. Do we want to try to make sure someone starts working on making it more properly mirror the aws stuff or do we want to consider it on hold until after aws is live?
 10:12:40 <stevenhaddox> Main concerns I saw out of backchat were those really. Nothing too major.
 10:15:39 <stevenhaddox> also, do we need to figure out how to get logging working again for when people start getting more active?
 10:16:37 <samkottler> stevenhaddox: I think most people who care about the convo will have a bouncer in here anyhow
 10:16:41 <samkottler> but I might be wrong about that
 10:19:01 <dwradcliffe> I'm an IRC noob. Need to figure that out.
 10:20:39 — stevenhaddox isn't an irc noob, but certainly haven't ever bothered figuring out a bouncer as I'm not nearly as often as I should be...
 10:21:15 <bradland> the challenge with IRC is keeping anything you have running up and connected :\
 10:21:16 ⇐ technicalpickles quit ([email protected]) Quit: Computer has gone to sleep.
 10:21:21 <bradland> so obnoxious
 10:22:12 <dwradcliffe> anyone: what do you use?
 10:22:45 <bradland> i've not found the occasion to run a bouncer, but i ran a bot (that also did logging) from the ruby library named autumn
 10:24:08 <bradland> there's this: http://irclogger.com
 10:24:12 <bradland> source is on github
 10:24:19  → stevenhaddox_ joined  ↔ zerstorer nipped out  
 10:28:58 <samkottler> dwradcliffe: quassel
 10:29:27 <samkottler> dwradcliffe: mitchellh runs an awesome service, too - https://www.ircrelay.com/
 10:30:01 ⇐ stevenhaddox_ quit ([email protected]) Read error: Connection reset by peer
 10:33:14 → technicalpickles joined ([email protected])
 10:36:13 <dwradcliffe> cool, thx
 10:38:08  → andrewhubbs1, thatRD and snowshoe joined  ⇐ bhenerey, postmodern, andrewhubbs and Boxcar21 quit  ↔ dwradcliffe_ popped in  ↔ zerstorer and dwradcliffe nipped out  
 10:49:19 <ssd7> Anything I can help with?  Looks like holoway got us a geoip packaging script https://github.com/rubygems/rubygems-aws/pull/70/files
 10:50:44 ⇐ andrewhubbs1 quit ([email protected]) Quit: Leaving.
 10:50:58 <ssd7> If someone wants to merge that I could try working on an nginx-from-source recipe for us.
 10:51:08 → Boxcar21 joined ([email protected])
 10:51:10 <dwradcliffe> ssd7: sure
 10:53:22 <dwradcliffe> ssd7: done
 10:54:10 → maledale joined ([email protected])
 10:54:12 <samkottler> I'm gonna build the package and then push it out
 10:54:21  → stevenhaddox_ joined  ↔ jgraichen nipped out  •  scoutcamper|away → scoutcamper|away  
 10:59:12 <ssd7> samkottler: Push it out to where?
 10:59:22 <samkottler> ssd7: the load balancer
 10:59:58 <ssd7> kk, I thought someone might have had a magical asset mirror that was going to host this instead of building it on each machine when needed.
 11:00:35 <samkottler> ssd7: there was talk of setting one up or putting the packages on s3, but for now I think git is unfortunately the best spot
 11:01:53 <ssd7> Roger.
 11:02:34  → andrewhubbs joined  ⇐ stevenhaddox_ quit  
 11:07:15 <dwradcliffe> I'm going to merge the rest of the nginx lb config
 11:08:05 <samkottler> dwradcliffe: okay great, I'll cook that too
 11:08:23  → snowshoe (was snowshoe_), bhenerey, isaacsanders and fbernier joined  ⇐ paulmoor1ng and snowshoe quit  
 11:19:07 <dwradcliffe> any objections to leaving the chef files in place after a chef run?
 11:19:32 <samkottler> dwradcliffe: fine by me - we don't have any passwords or sensitive stuff in the repo
 11:19:57 <dwradcliffe> it's public anyway
 11:20:01 <samkottler> indeed
 11:20:18 <bradland> would be a great service to the ruby community as well
 11:20:32 <bradland> a nice example for those getting started with chef
 11:20:33 <samkottler> bradland: it's already public :)
 11:20:54 <bradland> to leave in place as public i mean
 11:21:01 ⇐ bhenderson quit ([email protected]) Quit: ZNC - http://znc.in
 11:21:16 <dwradcliffe> bradland: I was referring to leaving the files on the server after chef runs
 11:21:25 <bradland> oh, duh. sry
 11:21:52 <samkottler> we should totally leave it public, though because doing so means no one will ever intentionally check in sensitive info
 11:21:57  ⇐ mbulat and kek quit  
 11:22:09 <ssd7> samkottler: Once you have a package you're happy with and commit it to git, let me know.  The cookbook to install it is like 4 lines so once it is uploaded I'll test it and open a pull request.
 11:23:43 <samkottler> ssd7: how about I just give you a link to the deb?
 11:24:28 <ssd7> that is fine too
 11:24:37 <dwradcliffe> ssd7: feel free to enable geoip on that node once that's in
 11:24:59 <samkottler> ah yeah, make sure that attribute is set to true
 11:26:26 <samkottler> ssd7: http://5.9.167.115/nginx_1.2.6-1_amd64.deb
 11:27:39  → seanhorn, brotspinne and DonOtreply joined  ⇐ sc_raptor quit  ↔ Boxcar21 nipped out  
 11:33:35 <ssd7> Given the size of this binary, I might consider doing do git-filter-tree once we have a place to put it outside of git.  People will need to re-clone after, but AFAIK it is really the only way to stop the big file from killing your git speeds even after it is deleted.
 11:36:08 <samkottler> yep
 11:36:44 ⇐ dontbecold_ quit ([email protected]) Quit: dontbecold_
 11:37:10 <dwradcliffe> ha! http://monosnap.com/image/9XogooFzKTuxjlSzFoDqnz44D
 11:37:45 → fromonesrc joined ([email protected])
 11:39:22 <samkottler> HA that's awesome
 11:39:33 <dwradcliffe> did we try that first? ;)
 11:39:59 → dontbecold_ joined ([email protected])
 11:44:19 <dwradcliffe> samkottler: is that new script what you used to build?
 11:44:21  ⇐ dontbecold_ and isaacsanders quit  
 11:45:07 <samkottler> dwradcliffe: holoway wrote it, that patch is just come modifications so it can run without any user input
 11:46:19 → dontbecold_ joined ([email protected])
 11:46:29 <dwradcliffe> right, was just curious if you needed those changes to get it to work
 11:47:08 <samkottler> dwradcliffe: yes, the new packages are needed for openssl support and nokogiri's extensions to build
 11:47:16 ⇐ brotspinne quit ([email protected]) Quit: brotspinne
 11:47:28 <dwradcliffe> samkottler: cool - merged
 11:47:32 <samkottler> and the tmp dir should exist, as well as the right gems via bundler
 11:47:34 <samkottler> thanks dwradcliffe
 11:50:32  ⇐ Boxcar21, maledale, jfelchner1 and mephux quit  ↔ zerstorer nipped out  
 11:55:07 <evan> morning everyone!
 11:55:14 <evan> so today we need to restore push
 11:55:32 <dwradcliffe> evan: good morning!
 11:55:38 <evan> hiya!
 11:55:40 → mephux joined (~mephux@unaffiliated/mephux)
 11:56:50 <samkottler> evan: hey
 11:56:56 <evan> hi
 11:57:07 <samkottler> I'm tuning postgres now
 12:02:47 <blasterpal> any trivial, fetching coffee tasks you guys need help with?
 12:03:40 <dwradcliffe> license added: https://github.com/rubygems/rubygems-aws/pull/72
 12:05:08 <jtimberman> mornin' folks
 12:06:42 <samkottler> ssd7: any update on the nginx cookbook?
 12:07:59 <ssd7> samkottler: Have what I think should work, trying to verify in vagrant
 12:10:35 <samkottler> k
 12:11:45  → dannyb and Boxcar21 joined  
 12:15:07 — samkottler is doing some maintence that is going to take the db offline for a few mins
 12:15:08 <evan> so, what we need to store push is: nginx LB's redirecting, rails app, postgress
 12:15:28 <evan> since it sounds like redis is up, it too.
 12:15:38 <evan> so, I guess thats most things :)
 12:15:50 <evan> we could get away with no stat-update to start
 12:16:01 <dwradcliffe> evan: https://github.com/rubygems/rubygems-aws/issues?milestone=1
 12:18:20 ⇐ eggmaster quit ([email protected]) Remote host closed the connection
 12:18:37 <phlipper> good day all
 12:18:40 <phlipper> awesome progress!
 12:18:56 <phlipper> anything i can help out with regarding testing/tweaking/tuning/final bits?
 12:19:56 <samkottler> evan: can we cut over to the new stuff with push disabled and the enable it once things stabilize?
 12:20:08 <evan> sure.
 12:22:29 <samkottler> evan: should the whole site sit behind SSL?
 12:22:46 <evan> going forward, yeah.
 12:22:57 <evan> we allowed normal HTTP access previously
 12:23:01 <evan> but given everything that has occured
 12:23:06 <evan> switching to SSL only seems prudent.
 12:23:19 <qrush> +1
 12:23:30 <qrush> HTTP should redirect to HTTPS
 12:23:38 <jtimberman> +12
 12:24:20 <phlipper> +1000 https everywhere... should terminate SSL at the ELB and use rails config.force_ssl for redirects and headers
 12:25:25 <dwradcliffe> I beleve that's how we have it setup
 12:25:40 <dwradcliffe> although we don't have a real SSL cert yet
 12:25:52 <phlipper> excellent
 12:25:53 <samkottler> that's the way it's configured right now :)
 12:26:01 <phlipper> good job guys ;)
 12:26:57 <samkottler> wtf unicorn
 12:27:13 ⇐ dontbecold_ quit ([email protected]) Quit: dontbecold_
 12:30:31 <mephux> dwradcliffe: http://freessl.com/
 12:31:18 <mephux> unless you need a wildcard cert - mad money :-\
 12:31:41 ⇐ btucker quit ([email protected]) Quit: Leaving.
 12:31:56 <dwradcliffe> evan: thoughts? ^^
 12:34:20  → joerocklin and btucker joined  
 12:34:58 <phlipper> not sure what the budget is for that sort of thing, but fwiw, we;ve had excellent luck w/ reasonable prices both single and wildcard at https://www.servertastic.com/rapidssl/
 12:35:32 <mephux> phlipper: feessl is signed by rapidssl.. its just free
 12:36:09 <phlipper> mephux: neat, i don't remember that from last time i looked, i will check it out again!
 12:36:46 <dwradcliffe> mephux: only free for 30 days though, right?
 12:37:19 <phlipper> free 30 days doesn't seem that great to me ..
 12:37:26 <mephux> click the try button
 12:37:39 <phlipper> i had that url confused with this: https://www.startssl.com/?app=1 which i've had weird luck w/ in the past
 12:38:24 <dwradcliffe> mephux: Validity Period: 1 month
 12:38:32 <phlipper> mephux: doesn't look free to me: http://cloud.phlippers.net/image/2C2W2J2X0f1H/Notification%20Center-2.jpg
 12:38:53 <fromonesrc> feel really dirty posting this but https://www.godaddy.com/ssl/ssl-open-source.aspx
 12:38:59 <fromonesrc> free ssl for open source for 1 year
 12:38:59 <mephux> phlipper: thats for the upgrade from free
 12:39:11 <fromonesrc> "Free first year exclusively for qualified Open Source projects.*"
 12:39:14 <fromonesrc> whatever qualified means
 12:39:16 ⇐ dannyb quit (c648e798@gateway/web/freenode/ip.198.72.231.152) Ping timeout: 245 seconds
 12:39:28 <phlipper> mephux: i don't think so: http://cloud.phlippers.net/image/3E3A0l1e3L2c
 12:39:50 <phlipper> fromonesrc: after 1 year, they are more costly than servertastic.
 12:40:33 <mephux> phlipper: ah, they changed it recently.. odd - anyway, i know people there
 12:40:41 <mephux> i can get to work on getting you a 3 year one
 12:41:17 <phlipper> mephux: if you can, i'm sure everyone here would be tremendously grateful
 12:42:21  ↔ btucker nipped out  
 12:45:06 <samkottler> this has some security implications - review please - https://github.com/rubygems/rubygems-aws/pull/77/files
 12:45:12 <samkottler> phlipper, mephux ^^
 12:45:46 <mephux> ok
 12:47:12 → gabceb joined ([email protected])
 12:47:19 <ssd7> So, this packages doesn't set up a lot of the default config we were depending on, so I'm currently working on getting that into place.
 12:47:52 <qrush> we have an SSL cert for rubygems.org
 12:48:01 <qrush> not sure what the arguing here is about - but we def have one
 12:49:03 <dwradcliffe> qrush: if you want to use the same one that's great
 12:49:14 <qrush> oh is that compromised too? not sure :(
 12:49:48 <dwradcliffe> qrush: most providers let you re-issue free
 12:50:38 <phlipper> qrush: is there any chance that the ssl private key was "compromised" at all? if we can verify that it has hasn't, you should be able to reuse the same cert. if there's any question, it would be worth issuing a new cert
 12:50:55 <qrush> according to the RH forensics no commands were run like that
 12:50:59 <qrush> but we should reissue anyway
 12:51:08 <phlipper> ok
 12:51:13 <qrush> evan or tcopeland will need to do that - not sure how they got it
 12:51:40 <dwradcliffe> also, who knows how to deploy it to the server securely?
 12:51:58 <phlipper> dwradcliffe: someone w/ AWS creds will install it to the ELB
 12:52:19 <jtimberman> yeah, reissue the cert.
 12:52:36 <jtimberman> any secrets that are on that system should be assumed compromised.
 12:52:42 <dwradcliffe> phlipper: didn't know we were using ELB
 12:52:56 <jtimberman> even if its forensically "clean" (no evidence of tampering)
 12:53:31 <phlipper> jtimberman: agreed totally
 12:53:37 <phlipper> dwradcliffe: lol :) http://cloud.phlippers.net/image/3x1b2m2g1D1l
 12:54:44 <dwradcliffe> phlipper: lol. I swapped ELB/LB as I read that
 12:54:55 <phlipper> haha no worries ;)
 13:06:29  ⇐ bhenerey quit  •  scoutcamper|away → scoutcamper  
 13:10:40 <samkottler> I just updated the postgres PR with some params for tuning
 13:10:56 <holoway> Amorning
 13:13:59 <holoway> samkottler: on tunings
 13:14:09 <holoway> is postgresql the only thing on the box?
 13:14:40 <holoway> for example, shared_buffers is often 1/4 ram
 13:14:54 → bhenerey joined ([email protected])
 13:15:02 <samkottler> holoway: example the kernel won't allow shared buffers over 32mb
 13:15:12 <samkottler> so you can set it that high, but it won't do anything
 13:15:39 <holoway> samkottler: that's tunable
 13:15:39 <samkottler> but yes it is the only thing right now
 13:15:59 <benchMark> How big is the rubygems database?
 13:16:02 <samkottler> holoway: yep
 13:16:08 <samkottler> benchMark: tiny, like 5GB
 13:16:14 <samkottler> we could store the whole thing in RAM
 13:16:17 <holoway> samkottler: we always tweak the sysctl stuff to go with it
 13:16:25 ⇐ bhenerey quit ([email protected]) Client Quit
 13:16:36 <benchMark> samkottler: Makes sense. I couldn't imagine it was pretty big. :)
 13:16:51 scoutcamper → scoutcamper|away
 13:17:11 <holoway> samkottler: on work_mem, do we do sorts that are large often, or cross table merge sorts?
 13:17:35 <holoway> samkottler: if they do crazy stuff like 8 way merge sorts, that's 800mb per query of sorting slab
 13:17:36 <samkottler> holoway: I have no idea...just as new to the way the DB is utilized as you :)
 13:17:41 → bhenerey joined ([email protected])
 13:17:41 <holoway> good times
 13:17:42 <holoway> :)
 13:17:44 <benchMark> The very nice thing about Postgresql is that you can tune live a lot better than MySQL.
 13:17:48 <holoway> yes
 13:17:51 <samkottler> true
 13:17:54 <benchMark> Since it uses the system buffers for disk access.
 13:17:58 <benchMark> You can restart it on a whim.
 13:17:58 <holoway> and several people, benchMark included, know lots about doing it
 13:18:12 <benchMark> Unlike MySQL where you blow out your innodb buffer pool.
 13:18:20 <samkottler> holoway: setting kernel.shmall = 1310720
 13:18:20 <samkottler> kernel.shmmax = 5368709120
 13:18:39 <samkottler> benchMark: yeah good times
 13:18:50 — samkottler used to run a 2.2TB unshared mysql cluster
 13:18:53 <holoway> samkottler: one convenient thing oyu can do - set the shared_buffers where you want it, then start postgres, and watch it fail and tell you the exact tuning you want :)
 13:18:53 <samkottler> unsharded**
 13:19:06 <samkottler> holoway: that's what I've been doing
 13:19:11 — holoway smiles
 13:19:18 <benchMark> Yep, we had a 5 or 6TB instance at LivingSocial.
 13:19:19 <holoway> that's waht I get for reviewing your commits, man
 13:19:19 <benchMark> Never again.
 13:19:51 <samkottler> benchMark: we had 5 different completely replicated DC's around the world
 13:19:59 <samkottler> tungsten =~ the devil
 13:20:24 <benchMark> heh
 13:20:53 <benchMark> I'm pretty tied up today, but let me know if there's anything I can help out with.
 13:21:13 <benchMark> If it were me, I probably wouldn't spend a great deal of time tuning right now and optimize for getting everything back online like Evan wants.
 13:21:22 <evan> yeah
 13:21:26 <evan> we can tune a little
 13:21:29 <benchMark> But that's just my worthless opinion since I'm not actually helping much.
 13:21:31 <evan> but we don't need to get crazy for now.
 13:21:46 <evan> remember that the current instance is largely untuned.
 13:21:51 <evan> and handled it fine
 13:22:14 <evan> I already put some stuff in place to help with that
 13:22:25 <evan> ie, we were using COUNT() to generate the top 5 boxs on the front
 13:22:29 <evan> which hammered things.
 13:22:30 scoutcamper|away → scoutcamper
 13:22:49 <evan> but that is frontended with memcache now.
 13:23:57 <qrush> +1 for not tuning  - this is why we have newrelic
 13:24:19 <evan> the focus today needs to be restoring push
 13:24:23 <qrush> and now an army of volunteers who want to work on ops :D
 13:24:34 <evan> we can even turn off parts of the site to make that happen sooner
 13:24:45 <evan> if we're worried about perf of features
 13:24:51 <qrush> +1
 13:24:59 <qrush> doubtful though. :)
 13:25:04 <evan> :)
 13:25:58 <samkottler> postgres is at the point where I feel comfortable taking some traffic
 13:26:06 <samkottler> can someone else try logging in? I'm getting an error
 13:26:21 <samkottler> on my profile, not the actual login action
 13:28:04 <evan> dwradcliffe: could you tell samkottler about your primary key issue?
 13:28:15 <evan> sounds like maybe something didn't get setup right
 13:28:40 ← thatRD left ([email protected])
 13:28:44 <samkottler> we talked about it
 13:28:58 <dwradcliffe> samkottler: more than just one table
 13:28:58 <samkottler> I said in the issue that it'd be better to fix it in the app - evan do you think that makes sense?
 13:29:04 <evan> i don't.
 13:29:12 <evan> because it sounds like all the tables lost their primary key
 13:29:57 <dwradcliffe> some, not all. checking them all
 13:29:58 — someara is back. stats?
 13:30:08 <samkottler> ah so it happens on tables other than rubygems?
 13:31:09 <samkottler> should I try importing from the dump again?
 13:31:49 <dwradcliffe> all but one table
 13:32:32 <dwradcliffe> WebHook is the only one that has a primary key set
 13:33:49 scoutcamper → scoutcamper|away
 13:35:34 <samkottler> what's the best way forward then?
 13:35:45 <someara> where are we now? =)
 13:35:48 <someara> sorry just got here
 13:36:17 <samkottler> ssd7: where does nginx stand?
 13:36:19 <someara> I see database things in the back scroll
 13:36:21 <samkottler> we need that to launch
 13:36:27 <samkottler> someara: fixing some missing primary keys
 13:36:40 <someara> are we using the opscode awes branch? or the vagrant branch
 13:36:44 <someara> there's about a day divergence
 13:36:52 <someara> I posted an update ~3am
 13:37:09 <ssd7> I've cargo culted some init scripts from the comm. cookbook, trying to get them to work with this package.
 13:37:18 <someara> currently nuke/testing the repo
 13:37:34 <evan> samkottler: yeah, you might want to try to import it again
 13:37:34 <someara> how do we get all the nginx/database stuff from yesterday reconciled?
 13:40:42 <samkottler> evan: gonna do that now
 13:40:55 ← locks left (uid130@gateway/web/irccloud.com/x-xbwhzxnaqxdolyth)
 13:41:23 <evan> also, check the schema in the dump to make sure the primary keys are properly declared.
 13:43:56 ← btucker left ([email protected])
 13:44:24 <samkottler> https://gist.github.com/bfe6fb36714dff7fefb1
 13:45:33 <dwradcliffe> I assume unicorn has been restarted since the initial import?
 13:45:39 <samkottler> also it's worth noting that I'm going to be completely offline from around 6pm tonight until noon on Sunday
 13:45:54 <samkottler> dwradcliffe: yep
 13:46:00 — samkottler dropped the db and is reimporting it now
 13:46:14 <phlipper> samkottler: not sure how you plan to hand-off, but i'd be happy to coordinate if you need
 13:46:34 <dwradcliffe> ha, I'll be off the grid for that same window
 13:46:47 — phlipper missed the big party invite ..
 13:46:48 <phlipper> :)
 13:47:01 — samkottler is going skiing
 13:47:21 → lsegal joined (uid1565@gateway/web/irccloud.com/x-yzbdcxuhefajsunk)
 13:47:22 — dwradcliffe is going to a cabin with friends
 13:47:51 <phlipper> samkottler: where are you located?
 13:47:52 <samkottler> phlipper: we should give evan and qrush access to the machines
 13:47:59 <samkottler> phlipper: Boston
 13:48:15 <samkottler> but also new york, it's confusing...
 13:50:05 <samkottler> alright, the drop and import fixed the primary key issue
 13:50:09 <samkottler> dwradcliffe, evan ^^
 13:50:09 <phlipper> nice
 13:50:17 <dwradcliffe> awesome!
 13:50:30 <evan> samkottler: vunderbar
 13:50:42 — samkottler closes the ticket
 13:50:53 <dwradcliffe> site seems to be functioning now!
 13:51:08 <samkottler> w00t!
 13:52:24 <ReinH> pa!
 13:52:27 <ReinH> er
 13:52:28 <ReinH> yay!
 13:52:48 <samkottler> alright, nginx
 13:54:54 <samkottler> evan: do you have the SSL certs for production?
 13:55:00 <evan> yep
 13:55:04 <evan> just purchased them
 13:55:06 <samkottler> evan: want to OTR them to me?
 13:55:10 <evan> sure
 13:55:22 <dwradcliffe> samkottler: are you going to manually install?
 13:55:51 <samkottler> dwradcliffe: I'm gonna have chef do it from .chef, but not include them in the repo (durh)
 13:56:00 → rondale_sc joined ([email protected])
 13:56:12 <dwradcliffe> samkottler: cool
 13:56:23 ← rondale_sc left ([email protected])
 13:56:40 <ReinH> samkottler: not include them in the repo? what is this, security through obscurity? :D
 13:57:17 <samkottler> ReinH: lmao if you've never inteviewed someone and asked for their SSH pubkey and gotten a private
 13:57:21 <ssd7> OK, my status:  ** [out :: 33.33.33.11] STDERR: nginx: [emerg] unknown directive "stub_status" in /opt/nginx/conf/conf.d/nginx_status.conf:6
 13:57:29 <ReinH> samkottler: ;)
 13:58:00 <phlipper> ssd7: did holoway add that to his custom build?
 13:59:09 <ssd7> From the script it looks like no
 13:59:30 <ssd7> Do we definitely want that? It is easy enough to nuke that config if we don't.
 14:00:26 <phlipper> ssd7: not sure if monit uses that directly or if the original template parts that use that got stripped out
 14:00:49 <dwradcliffe> phlipper: can you add an attribute to enable/disable that?
 14:00:55 <phlipper> i have it configured to only respond locally for stats for munin, etc.
 14:01:03 <phlipper> dwradcliffe: sure thing, that should be easy
 14:01:04 → tkramer joined ([email protected])
 14:01:18 <phlipper> give me ~ 5 mins to wrap something up
 14:01:59 → ssklar joined ([email protected])
 14:04:25 <samkottler> the real certs are in place
 14:04:43  → maxamillion and bfleischer joined  
 14:04:57 <dwradcliffe> samkottler: ++
 14:05:03 <ssd7> I've pushed here with what I was using so far: https://github.com/stevendanna/rubygems-aws/compare/nginx-package
 14:13:05 <samkottler> so maybe apt-get source is easier?
 14:13:07 <samkottler> :/
 14:13:30 ⇐ bhenerey quit ([email protected]) Quit: Leaving.
 14:13:33 <phlipper> dwradcliffe: just pushed an update to the nginx cookbook, set `node["nginx"]["enable_stub_status"] = false` to prevent the stub status module
 14:13:38 <holoway> samkottler: easy to tweak
 14:13:57 <dwradcliffe> ssd7: can you try that ^
 14:13:58 <phlipper> or ssd7 ^^
 14:14:01 <phlipper> sorry :)
 14:14:09 <dwradcliffe> phlipper: np
 14:14:11 — samkottler is going to get lunch
 14:14:15 <samkottler> I can do it when I get back
 14:16:05 ⇐ stevenhaddox quit ([email protected]) Remote host closed the connection
 14:16:08 <ssd7> getting the new recipe and trying, thanks
 14:16:11 <gazoombo> thanks for your continued efforts, everyone
 14:16:26 <holoway> adding stub status in to the build/nginx.sh script, since thats super handy
 14:16:36 <gazoombo> I haven't had any more time to spare, but I've been peaking in here to see how things are going
 14:16:37 <dwradcliffe> race!
 14:16:40 <gazoombo> so thanks
 14:17:01 <ssd7> holoway: +1 once that is in we can turn it back on, no harm having both :)
 14:17:35 <holoway> here is the list of other modules
 14:17:36 <holoway> https://gist.github.com/4693402
 14:17:49 <holoway> ssl, geoip, and stub_status are on
 14:18:20 <jtimberman> i know its a design goal from the standpoint of performance, but man do i wish nginx had dynamic modules sometimes :)
 14:18:25 <dwradcliffe> what about gzip_static?
 14:18:36 <holoway> that was my first thought, too
 14:19:10 <holoway> probably realip too
 14:20:36 <holoway> adding 'em both
 14:22:09 <phlipper> ssd7: just realized i forgot to push the tags w/ that last nginx cookbook update. if you had issues, try again. sorry
 14:24:24 <ssd7> cool, librarian-chef clean; librarian-chef install; going now
 14:24:32 <phlipper> great
 14:25:09 <phlipper> git-achievements told me that last tag earned me "Master Gipsy Level 8" .. heh, going to be a good weekend :P
 14:25:31 — samkottler is back
 14:27:07 <samkottler> so is the nginx init script working now?
 14:27:31 <holoway> https://github.com/rubygems/rubygems-aws/pull/79
 14:27:44 <holoway> adds stub_status, realip, and gzip_static
 14:28:09 <phlipper> holoway: good and merged
 14:28:14 <phlipper> thank you!
 14:28:19 <holoway> np
 14:29:39 <ssd7> nice
 14:29:50 <ssd7> holoway: was done before vagrant even had a chance
 14:30:27 <phlipper> holoway is quad core :)
 14:32:38 <ssd7> If someone wants to build that new package and try my branch on it, I'm gonna keep going with the package I have and this option to see if there are other bumps further down the road
 14:36:27  → bhenerey joined  ← felipecvo left  ⇐ grzywacz quit  
 14:45:22 <ssd7> OK, I just rebased and force pushed to the above branch that uses the old build and phlipper's new options.
 14:45:29  ⇐ DonOtreply and joerocklin quit  
 14:45:39 <ssd7> seems to be working in vagrant.  Should work with the new build, just flip that option back.
 14:48:47 → DonOtreply joined ([email protected])
 14:49:39 <samkottler> ssd7: can that get merged?
 14:50:32  → johndouthat joined  ⇐ DonOtreply quit  
 14:53:40 <ssd7> samkottler: I believe it can.  It works in vagrant for me.  Although, my vagrant was a clean setup, you might need to do something to nuke the current nginx that is running in your actualy EC instance.
 14:53:48 <ssd7> s/actualy/actual/
 14:54:09 <samkottler> 1.2.6 is more recent so hopefully the upgrade is seemless
 14:54:17 <samkottler> ssd7: can you want me to rebuild the package with the additional modules?
 14:54:42 <samkottler> s/do/do
 14:54:45 <samkottler> words...
 14:55:06 <dwradcliffe> samkottler: it's almost the weekend...
 14:55:31 <ssd7> Might as well if you can.  You can merge this now, then once that build is done you can update it in git and flip the status endpoint back on.
 14:56:44  → stevenhaddox, ereslibre and justincampbell joined  ⇐ zerstorer quit  •  dwradcliffe → dwradcliffe_brb  
 15:21:25 <vertis-sleeping> morning
 15:21:30 vertis-sleeping → vertis
 15:23:48 <someara> moin
 15:24:03  ⇐ seanhorn and andrewhubbs quit  •  dwradcliffe_brb → dwradcliffe  
 15:31:21 <phlipper> good day vertis
 15:31:30 <vertis> hey
 15:31:38 → dontbecold_ joined ([email protected])
 15:32:02 <vertis> i'll hop back onto working on performance stuff in a while
 15:32:33 <vertis> need to get breakfast and a coffee first
 15:33:58 <holoway> what's the status right now, btw? still working on a single box ec2?
 15:34:34 <vertis> me?
 15:34:47 <samkottler> alright people
 15:34:47 <holoway> with jtimberman and someara working on future stack stuff?
 15:34:54 <samkottler> the nginx stuff is out
 15:35:12 <dwradcliffe> holoway: staging? 3 ec2 boxes
 15:35:44 <dwradcliffe> samkottler: woot!
 15:36:04 <samkottler> are um...are we ready to do the switch?
 15:36:17 <samkottler> evan: qrush: ^^
 15:36:24 <someara> yep
 15:36:37 <evan> i'm in a meeting
 15:36:51 <evan> I'll circle back around shortly
 15:36:55 <samkottler> great
 15:36:58 <evan> I'd like to go through and validate the setup
 15:37:01 <samkottler> all the initial items are now done
 15:37:05 <evan> if you guys can give me access to it
 15:37:07 <ReinH> yay!
 15:37:11 → sferik joined ([email protected])
 15:37:14 <samkottler> evan: send me your pubkey
 15:37:30 <dwradcliffe> probably should add qrush keys too
 15:37:38 <samkottler> qrush: please send me your key
 15:37:48  → seanhorn joined  •  scoutcamper|away → scoutcamper  
 15:38:00 <qrush> isnt this on github now ?
 15:38:01 <qrush> :)
 15:38:26 <samkottler> qrush: evan is in a meeting so I'm trying to make his life easy :)
 15:38:29 <samkottler> qrush: PR is great, ofc
 15:38:42 scoutcamper → scoutcamper|away
 15:40:12 <dwradcliffe> samkottler: want to document the instances/ips on the wiki?
 15:41:42 <qrush> samkottler: i meant this - https://github.com/qrush.keys
 15:42:05 <samkottler> qrush: ah right
 15:55:15  → andrewhubbs joined  ⇐ timhin quit  ↔ mephux nipped out  •  dwradcliffe → dwradcliffe_away  
 16:06:16 <evan> ok, I'm out of my meeting
 16:07:08 <samkottler> evan: I'm on a call now :(
 16:07:12 <samkottler> 15 minutes
 16:07:15 <evan> ha!
 16:07:18 <evan> no prob.
 16:07:27 <evan> people are being very understanding about push access thus far
 16:07:43 <evan> so I may just do the switch this evening
 16:07:52 <evan> I have another thing at 1:30
 16:08:54 <samkottler> evan: okay sounds good - everything is basically just sitting at the ready
 16:10:06 <holoway> samkottler: do you feel like the total hero you are?
 16:10:17 <holoway> need a sticker?
 16:10:21 <elskwid> ^^ +1
 16:10:27 <gazoombo> +10
 16:10:59 <samkottler> heh thanks :) glad I was able to help out on this
 16:11:02 <samkottler> you are all awesome :D
 16:11:20  ↔ fromonesrc nipped out  
 16:15:30 <phlipper> samkottler, everyone, very excellent work! has been truly amazing to watch you all work
 16:22:30 ← ssklar left ([email protected])
 16:28:04 <evan> samkottler: can you send me an email with the IPs?
 16:28:29 <samkottler> evan: yeah I'm giving you access right now
 16:28:46 <samkottler> evan: can you generate an passwd via openssl and send it to me?
 16:29:43 <evan> sure
 16:31:11 <vertis> evan: I'm part way through setting up a meter script we can use to do some performance testing
 16:31:17 <vertis> jmeter*
 16:31:23 <evan> great
 16:32:01 <vertis> I have access to performance tools that can be used to throw production like load at it (whatever that is)
 16:33:55  ↔ teancom nipped out  
 16:35:52 <vertis> samkottler: Are the IPs documented somewhere?
 16:36:15 <samkottler> vertis: working on it
 16:36:37 <vertis> cool
 16:37:10  → alexmreis joined  ⇐ teancom quit  
 16:42:25 <stevenhaddox> When we go back online do we know what functionality will be ready to go versus still on hold?
 16:42:47 <stevenhaddox> API v1, push, etc.? Anything any of us can do to help with anything that may not be ready to go live just yet?
 16:43:00 <qrush> i'm back
 16:43:03 ← jasoares left ([email protected])
 16:43:15 <qrush> is the app up somewhere privately?
 16:43:35  → jessie joined  ⇐ bhenerey quit  
 16:46:43 <evan> yeah, samkottler is putting together info on accessing it
 16:46:51 <evan> qrush: I'll want you to go in and verify it
 16:46:57 <evan> the app, postgres, etc
 16:47:09 <qrush> sure
 16:47:11 → bhenerey joined ([email protected])
 16:47:20 <qrush> sferik is here too to help!
 16:47:26 <sferik> hi!
 16:47:31 <sferik> you rang?
 16:48:55 <stevenhaddox> Do we know if the vagrant install is up to date from the README in the github repo? I'm walking through it now/ a failure part way through the first `cap chef`. Second seems to be getting farther along so far.
 16:51:14 <qrush> samkottler: add sferik to this box too. ppppiiiing cmeiklejohn
 16:51:29 <cmeiklejohn> pong
 16:51:33 <qrush> any core contributor should have access
 16:51:44 <qrush> cmeiklejohn: sferik  can you get samkottler your ssh pubkey
 16:52:18 <cmeiklejohn> k
 16:52:49 <vertis> stevenhaddox: people have reported that
 16:52:54 → dblock joined ([email protected])
 16:53:03 <stevenhaddox> vertis: I remember seeing that in backchat, but wanted to verify it's expected
 16:53:11 <sferik> https://github.com/sferik.keys
 16:53:16 <vertis> i haven't tried recently
 16:53:18 <stevenhaddox> vertis: I'll update the readme to reflect they need to run it multiple times for now if you'd like
 16:53:28 <sferik> samkottler ^^
 16:53:39 <samkottler> sferik: thanks
 16:53:45 <vertis> samkottler: are the users still the ones in the data bags in the repo?
 16:53:46 <samkottler> sorry, stuck on a call
 16:53:52 <vertis> or are we putting them somewhere else
 16:54:06 <vertis> because if so I can create them
 16:54:58 <samkottler> please submit a PR to the repo
 16:55:05 <vertis> will do
 16:56:26 <vertis> rush / evan: can I get access too?
 16:56:32 <vertis> qrush*
 16:56:40 <vertis> stupid autocorrect
 16:56:45 ⇐ bfleischer quit ([email protected]) Quit: bfleischer
 16:57:26 <vertis> also just generating users for each of you
 16:57:48 ← dblock left ([email protected])
 16:58:25 <evan> samkottler: can you give vertis access?
 16:58:36 <vertis> it's in a databag
 16:58:40 <samkottler> just submit PR's
 16:58:41 <vertis> i can add myself
 16:58:48 <vertis> just asking permission
 16:59:07 <vertis> evan I'm adding you , qrush, sferik in PR now
 16:59:13 <evan> k
 16:59:16 <vertis> anyone else need to be added
 16:59:17 <evan> vertis: go right ahead
 16:59:28 <evan> the fewer the better for the time being.
 16:59:36 <vertis> yeah just checking
 16:59:37 → JohnHirbour joined ([email protected])
 16:59:49 <samkottler> let's try to limit access as much as possible :)
 17:00:08 ← JohnHirbour left ([email protected])
 17:01:07 <vertis> https://github.com/evanphx.keys
 17:01:08 <vertis> ?
 17:01:40 <evan> thats fine
 17:05:29 <jtimberman> oh right, i forgot about that feature :)
 17:06:31 <cmeiklejohn> vertis: me too?
 17:06:41 <cmeiklejohn> i send my key over samkottler
 17:06:47 ⇐ bradland quit ([email protected]) Quit: bradland
 17:06:52 <vertis> okay, hang on
 17:08:15 dwradcliffe_away → dwradcliffe
 17:09:20 <vertis> cmeiklejohn: can you send the public key to me as well
 17:09:56 <samkottler> vertis: can you want do the merges and then I'll deploy?
 17:09:59 <stevenhaddox> So I saw a staging IP in here earlier. Looks like that's working great for me (minus the footer images for credit to org's)
 17:10:14 <jtimberman> if a single chef run doesn't completely configure the node, imo, therer's a bug somewhere in the recipes
 17:10:22 <stevenhaddox> Am I supposed to see similar w/ Vagrant after `cap chef`? Do we hit the LB IP or the app IP in vagrant setup?
 17:10:22 <vertis> samkottler: just have to do cmeiklejohn then I'm ready, yeah
 17:10:45 <jtimberman> in other words, imo, if you have to run multiple times to be completely configured, it's broke
 17:10:45 <ssd7> stevenhaddox: Is there an issue for whatever you were seeing?
 17:11:22 <stevenhaddox> ssd7: the only issue on the 54.245.255.174 IP is the missing sponsor images at the bottom of the page
 17:11:37 <stevenhaddox> for vagrant I can't get anything besides Nginx welcome page on port 80 for the lb / app IPs
 17:11:56 <stevenhaddox> unless I'm doing something wrong. Pulled about 15-20 minutes ago, destroyed, re-up'd, cap chef twice
 17:12:36  → teancom joined  ← alexmreis left  
 17:14:13 <vertis> just creating pull request now
 17:16:33 <vertis> rather waiting for the git push, cos my for is out of date
 17:17:46 <samkottler> send me the PR when you're done and I'll merge it and push
 17:18:50 <stevenhaddox> ssd7: With the IP I listed (staging IP?) on port 80 I'm getting "Welcome to Nginx" currently.
 17:19:22 <stevenhaddox> ssd7: I'd assume it should be redirecting to SSL? It did the first time, but last few requests it's sticking to the welcome page.
 17:19:57 <stevenhaddox> weird... now I can't replicate. Was only two requests :-/
 17:20:04 <vertis> https://github.com/rubygems/rubygems-aws/pull/81
 17:20:10 <holoway> stevenhaddox: converging in the background?
 17:21:03 <stevenhaddox> holoway: no clue...
 17:21:27 <stevenhaddox> Okay... so port 443 on the lb on my vagrant instance renders the page, but not css styling. Seems a lot better than before.
 17:22:45 <stevenhaddox> looks like js / css are getting 404'd in my vagrant setup, but otherwise looking good.
 17:23:02 <stevenhaddox> Going to nuke it from scratch and try again. I'll update the readme once I verify the process.
 17:24:03 <samkottler> instance listing - https://github.com/rubygems/rubygems-aws/wiki/Machines
 17:24:15 <samkottler> evan: ^^
 17:26:16 <ssd7> stevenhaddox: So, I'm rebuilding this to check that I'm not out of date.  But it looks like something is definitely wrong with the nginx configs under vagrant.  Prob. a consequence of the various hard-coded IPs that are about.
 17:26:18 <samkottler> vertis: cmeiklejohn: can you try access the load balancer and sudo'ing with your password?
 17:26:59 <stevenhaddox> ssd7: looking similar to me. The lb doesn't load at all besides welcome page. The app works on 443 but doesn't get access to resources (I'm guessing those are served via nginx though possibly via lb?)
 17:27:42 <samkottler> vertis: what did you set as the sudo password for cmeiklejohn?
 17:27:50 <samkottler> (don't post it here obviously)
 17:28:20 <vertis> i set them all somewhat randomly
 17:28:54 <samkottler> vertis: okay well people will need them so they should set it themselves
 17:29:06 <vertis> i was under the impression we were passwordless sudo
 17:29:24 <vertis> hrmmmm
 17:29:30 <samkottler> nope
 17:29:36 <vertis> pull requests themselves?
 17:29:50 <vertis> or passing me a hash
 17:29:52 <samkottler> vertis: they just send you the hash
 17:29:58 <vertis> okay
 17:30:04 <vertis> hang on
 17:30:10 <samkottler> openssl passwd -1 <password to be hashed>
 17:30:12 <stevenhaddox> ssd7: starting from scratch with vagrant up again
 17:30:12 <ssd7> stevenhaddox: OK, so for one, you'r gonna want to make the vagrant role override the application_servers ip array. Currently it will be  "application" => { "application_servers" => [ "10.249.31.114" ]} which is wrong for vagrant.
 17:30:48 <ssd7> Once were more ready to entertain moving this over to using chef-client with search, alot of things like that go away.
 17:30:55 <ssd7> If that is still the plan.
 17:31:36 <samkottler> I just updated evan's since he gave it to me
 17:31:58 <cmeiklejohn> i just dm'ed you vertis
 17:31:59 <vertis> qrush, evan, sferik?
 17:32:06 <samkottler> vertis: evan is all set
 17:32:14 <stevenhaddox> ssd7: thanks. I'll try it soon as it's up again.
 17:32:14 <vertis> oops
 17:32:16 <vertis> yeah
 17:32:17 <qrush> sorry, what?
 17:32:26 <vertis> openssl passwd -1 <password to be hashed>
 17:34:02 <qrush> who am i sending this to? vertis ?
 17:34:10 <cmeiklejohn> yes
 17:34:15 <vertis> yep
 17:34:17 <sferik> I just sent him mine
 17:34:31 <sferik> vertis: did you get it?
 17:34:38 <vertis> yep
 17:34:45 <vertis> just waiting on qrush
 17:35:20 <qrush> sorry i am really in a million places
 17:35:41 <dwradcliffe> I'm heading out for the weekend. Thank you all for your work!!
 17:35:48 <stevenhaddox> dwradcliffe: thank you!
 17:35:48 <teancom> qrush needs a transmogrifier to make more qrush's...
 17:36:32 <vertis> just checking I put the right hashes in the right files
 17:36:54 <vertis> samkottler: do you want to send me evans so we only have to do one pull request?
 17:37:07 <samkottler> vertis: I just committed it to master
 17:37:12 <samkottler> sorry for being a bad person
 17:37:16 <vertis> lol
 17:37:19 <vertis> well fine
 17:37:27  → paulmooring joined  •  dwradcliffe → dwradcliffe_away  
 17:39:25 <vertis> https://github.com/rubygems/rubygems-aws/pull/82
 17:40:26 — samkottler is updating then will deploy
 17:40:31 <holoway> ssd7: you'll still need exceptions for vagrant, since you can't tie the rubygems dev process to hosted chef/private chef
 17:40:35 → GitHub180 joined ([email protected])
 17:40:35 <GitHub180> [rubygems-aws] none pushed 3 new commits to master: http://git.io/z1-u5w
 17:40:35 <GitHub180> rubygems-aws/master be0a939 Sam Kottler: Merge pull request #81 from vertis/master...
 17:40:35 <GitHub180> rubygems-aws/master e5f2787 Luke Chadwick: User generated passwords
 17:40:35 <GitHub180> rubygems-aws/master e67dc5b Nick Quaranto: Merge pull request #82 from vertis/master...
 17:40:35 ← GitHub180 left ([email protected])
 17:40:45 <samkottler> yay! gh notifications
 17:40:46 — qrush kicks the stupid github irc hook
 17:40:59 <vertis> lol
 17:41:02 <jtimberman> haha
 17:41:14 <qrush> i had SSL checked but it wasn't on the right port for SSL on freenode
 17:41:17 <samkottler> qrush, evan, cmeiklejohn: can you try logging into the balanacer and getting sudo
 17:41:26 <qrush> sorry what IP?
 17:41:38 <ssd7> holoway: Good point.
 17:41:42 <samkottler> https://github.com/rubygems/rubygems-aws/wiki/Machines
 17:41:50 <samkottler> qrush: 54.245.255.174
 17:42:14 <cmeiklejohn> i logged into the balancer
 17:42:23 <samkottler> cmeiklejohn: sudo via your password?
 17:42:25 <cmeiklejohn> sudo works
 17:42:31 <qrush> yep - root@ip-10-253-6-159
 17:42:51 <samkottler> qrush: w00t!
 17:42:55 — samkottler rolls it out everywhere
 17:43:09 <vertis> yeah mine works
 17:43:27 <qrush> please document how to "roll it out"
 17:43:34 <qrush> would be cool to know how to do shit like this.
 17:46:54 <ssd7> quick question, is the app config for the database server in the repo or was that fixed up by hand?  Looks like on master the app still points at localhost?
 17:48:56 ⇐ yfeldblum quit ([email protected]) Ping timeout: 245 seconds
 17:52:40 <vertis> did we end up getting newrelic back in?
 17:58:00 <samkottler> vertis, qrush, evan, cmeiklejohn: you guys have access now to everything
 17:58:21 <samkottler> I have to head out from now until sunday for skiing, but email me with questions - my email is on GH
 17:58:38 <phlipper> samkottler: thanks again for so much hard work, enjoy the ski trip!
 17:58:44 <samkottler> see ya!
 17:58:46 <samkottler> phlipper: :D
 17:58:48 <vertis> samkottler: hey
 17:58:51 <samkottler> nice destresser
 17:58:56 <vertis> the rolling out changes
 17:59:06 <vertis> who else has done it in *prod*
 17:59:22 <samkottler> phlipper has, vertis
 17:59:27 <vertis> cool
 17:59:35 <vertis> have good weekend
 17:59:42 <stevenhaddox> samkottler: +1. Thanks for everything.
 18:01:40 — teancom went to install a gem, couldn't figure out why it wasn't working. Then he remembered rubygems.org pointed to his vagrant instance. *doh*
 18:02:17 scoutcamper|away → scoutcamper
 18:02:26 <vertis> teancom: did that yesterday
 18:02:46 <teancom> Glad I'm not the only one :-P
 18:03:44 ⇐ Boxcar21 quit ([email protected]) Quit: Leaving...
 18:05:11 <vertis> qrush: is it possible to get access to the newrelic account?
 18:05:12 <sferik> I can't login to the balancer
 18:05:22 <sferik> are my keys on that machine?
 18:05:32 <vertis> they should be
 18:05:44 <vertis> everyones was done together
 18:05:45 <sferik> ssh [email protected]
 18:05:48 <vertis> no
 18:05:51 <sferik> or erik@
 18:05:53 <ssd7> stevenhaddox: So, with some attribute shuffling I think I've got the vagrant stuff working, cap cheffing' now so it'll be a bit before I know.
 18:05:53 <sferik> sferik@
 18:05:56 <vertis> ssh sferik
 18:06:11 <sferik> Permission denied (publickey).
 18:06:19 <sferik> let me make sure I gave you the right key
 18:06:23 <vertis> let me just check things
 18:06:31 <vertis> did I not get it from your github keys?
 18:06:34 <stevenhaddox> ssd7: awesome. Have a branch I can pull from?
 18:06:57 <sferik> yeah, that should be right
 18:07:06 <sferik> https://github.com/sferik.keys is the right one
 18:08:11 <sferik> get the same thing when I try sshing into app01 or dbmaster01
 18:08:13 <vertis> something is wrong with the key
 18:08:16 <ssd7> stevenhaddox: One sec, I really hate pushing code I haven't at least seen converge all the way :)
 18:08:22 <vertis> i'll readd it
 18:08:52 <vertis> okay, just creating pull request
 18:09:07 <stevenhaddox> ssd7: understandable.
 18:09:11 <vertis> then phlipper can run me though deploying it so we don't have a SPF
 18:09:37 <vertis> (i have a fair idea)
 18:09:43 <phlipper> vertis: definitely. i'm submitting a PR w/ updated keys for me too
 18:10:12 <sferik> vertis: looks like you gave me evan's public key the first time :)
 18:10:21 <vertis> yeah
 18:10:33 <vertis> sorry was doing a whole bunch of them at once
 18:10:56 <vertis> now I have to race phlipper
 18:11:02 <vertis> so I don't have to rebase
 18:11:05 <phlipper> :D
 18:11:13 <stevenhaddox> lol
 18:11:14 <evan> ok, I have some errands to run
 18:11:18 <evan> I'll be back later today
 18:11:29 <evan> I'm aiming to validate and switch things over this evening
 18:11:33 <evan> and activate push access
 18:11:34 <vertis> https://github.com/rubygems/rubygems-aws/pull/84
 18:11:34 → GitHub26 joined ([email protected])
 18:11:34 <GitHub26> [rubygems-aws] vertis opened pull request #84: sferik had the wrong public key (master...master)  http://git.io/VNvU6A
 18:11:34 ← GitHub26 left ([email protected])
 18:11:37 <qrush> i am in the middle of some other shit - can we live without NR for at least a little bit?
 18:11:43 <evan> qrush: yep
 18:11:45 <qrush> i can add everyone later tonight
 18:11:45 → GitHub120 joined ([email protected])
 18:11:45 <GitHub120> [rubygems-aws] phlipper opened pull request #85: update credentials for phlipper: (master...phlipper-credentials)  http://git.io/nAZ4cw
 18:11:45 ← GitHub120 left ([email protected])
 18:11:46 <qrush> ok
 18:11:52 <vertis> evan or qrush: newrelic?
 18:12:04 <stevenhaddox> ssd7: I only have another half hour or so before the family is going out to eat. I'll try and run through the vagrant stuff later to verify it from scratch. Appreciate you updating it.
 18:12:11 <evan> vertis: we don't need it for now.
 18:12:24 <vertis> i was going to use it for performance testing
 18:12:32 <stevenhaddox> ssd7: unless I can get it through it before we go obviously ;)
 18:13:09 <vertis> evan, I'll add my own temporarily
 18:13:20 <vertis> so I can get metrics while testing
 18:13:38 → GitHub93 joined ([email protected])
 18:13:38 <GitHub93> [rubygems-aws] sferik pushed 2 new commits to master: http://git.io/sHcmyA
 18:13:38 <GitHub93> rubygems-aws/master d5517d9 Luke Chadwick: sferik had the wrong key
 18:13:38 <GitHub93> rubygems-aws/master eba6280 Erik Michaels-Ober: Merge pull request #84 from vertis/master...
 18:13:38 ← GitHub93 left ([email protected])
 18:14:04 <ssd7> stevenhaddox: Yeah, I'll leave a branch in channel for you to check out.  We'll need someone familiar with the EC2 environment to verify it carefully before merge as it required a small change to how some config is managed.
 18:14:20 <stevenhaddox> ssd7: sounds good
 18:14:35 — stevenhaddox obviously won't be that person as I've only run one dev box in EC2 for less than an hour...
 18:14:44 <vertis> okay, phlipper help me deploy that change to the LB?
 18:14:49  → GitHub70 joined  ⇐ tkramer quit  
 18:14:50 <GitHub70> [rubygems-aws] sferik pushed 2 new commits to master: http://git.io/obpwYg
 18:14:50 <GitHub70> rubygems-aws/master 557ce92 Phil Cohen: update credentials for phlipper:...
 18:14:50 <GitHub70> rubygems-aws/master 4c191a4 Erik Michaels-Ober: Merge pull request #85 from phlipper/phlipper-credentials...
 18:14:50 ← GitHub70 left ([email protected])
 18:14:51 <sferik> okay, I merged both of those
 18:15:04 <sferik> let me know when I should try again
 18:15:41 <phlipper> vertis: 1s
 18:21:39  → yfeldblum joined  ⇐ snowshoe quit  
 18:31:44 <vertis> we're lacking details about how the production ssl certs get on the balancer
 18:35:06 <stevenhaddox> ssd7: off to dinner. I'll check in later. Thanks again.
 18:36:14  → workmad3 joined  ⇐ bhenerey quit  
 18:39:39 <ssd7> vertis: So, I can provide some details based solely what I see on the cookbooks; but I'm not in touch with what is happening on EC2.
 18:39:51 <vertis> shot
 18:39:54 <vertis> shoot*
 18:40:37 <vertis> I'm not sure anyone is in touch with EC2 besides samkottler
 18:40:43 <vertis> which is less than ideal
 18:41:15 <ssd7> There are two attribites node["application"]["ssl_key"], node["application"]["ssl_cert"]
 18:41:37 <ssd7> These should be set to the file name of the key and cert
 18:41:59 <ssd7> the key and cert then need to end up in #{node["nginx"]["dir"]}/certs/
 18:42:18 <ssd7> which should be /opt/nginx/conf/certs
 18:42:25 <ssd7> given the current cookbooks
 18:42:43 <ssd7> I don't see anything that would actually place those keys on disk for you, so that would still be manual
 18:43:15 <vertis> well that is useful
 18:43:30 <vertis> i.e. fixing sferik won't break things
 18:43:58 <vertis> alright I'm going to run this on the LB
 18:46:33  ↔ stevenhaddox nipped out  
 18:48:32 <vertis> hrrmm
 18:49:09 <sferik> ?
 18:49:13 <vertis> i need access the deploy ssh key
 18:49:55 <vertis> phlipper: do you have that
 18:51:02 <phlipper> vertis: yes, OTR IM?
 18:52:25  ⇐ workmad3, jessie and technicalpickles quit  
 19:15:33 <ssd7> stevenhaddox: Doubt I'll have that branch soonish, there seem to be some db permissions issues to sort out before a seperate database will be working in vagrant
 19:20:06  → technicalpickles joined  ← indirect left  ⇐ kyd, yfeldblum and benchMark quit  
 19:56:25 <vertis> sferik: still around?
 19:56:35 <vertis> can you try logging into the LB now
 19:56:51  → Boxcar21 and snowshoe joined  ⇐ dontbecold_ quit  
 20:03:10 <ssd7> Here is a branch with the changes I had to make to get a 3-machine vagrant setup working https://github.com/stevendanna/rubygems-aws/compare/fixup-vagrant
 20:03:37 <ssd7> How to manage those postgres user creds is an open question since ideally they wouldn't sit in the attributes like that.
 20:05:56 — vertis pokes sferik
 20:09:45  → GitHub110 joined  ← snowshoe left  
 20:12:36 <GitHub110> [rubygems-aws] vertis opened pull request #86: Add configuration sections to ignore vagrant, gitignore production keys (master...master)  http://git.io/WIAMWA
 20:12:36 ← GitHub110 left ([email protected])
 20:12:55 <vertis> phlipper: check and merge
 20:12:58 <vertis> ?
 20:13:09 <phlipper> vertis: yep, 1s
 20:13:50 <phlipper> all set
 20:13:53 → GitHub172 joined ([email protected])
 20:13:53 <GitHub172> [rubygems-aws] phlipper pushed 3 new commits to master: http://git.io/9hm7TA
 20:13:53 <GitHub172> rubygems-aws/master 22f8885 Luke Chadwick: Update node configs to have required ['sudo']['add_vagrant'] section
 20:13:53 <GitHub172> rubygems-aws/master 46a036f Luke Chadwick: Ignore production keys
 20:13:53 <GitHub172> rubygems-aws/master e9d51d5 Phil Cohen: Merge pull request #86 from vertis/master...
 20:13:53 ← GitHub172 left ([email protected])
 20:14:08 <vertis> cool
 20:14:26 <vertis> as soon as sferik checks his ssh key, will push to others
 20:14:37 ⇐ sferik quit ([email protected]) Quit: Computer has gone to sleep.
 20:14:42 <phlipper> my new key worked
 20:14:47 <phlipper> his should too
 20:14:47 <vertis> okay
 20:14:52 <vertis> i'll do the others
 20:14:55 <phlipper> great
 20:15:08 <vertis> and I'll make it use the cap method
 20:15:17 <phlipper> great
 20:15:18 <vertis> then create pull request for that
 20:19:10  → bhenerey joined  ⇐ andrewhubbs quit  
 20:23:48 <vertis> grrr
 20:24:06 <vertis> i think I understand why sam wasn't using the cap command
 20:24:35 <vertis> it's a pain modifying it
 20:28:33 ⇐ technicalpickles quit ([email protected]) Quit: Textual IRC Client: www.textualapp.com
 20:31:22 <vertis> trying to deploy app server
 20:31:28 <vertis> *fingers crossed*
 20:31:37 <vertis> hope he wasn't had modifying anything else it
 20:31:39 <vertis> in*
 20:33:01 <qrush> is there a branch at rubygems.org yet for configuring cap on the new setup?
 20:34:40 <vertis> well that was a bust
 20:34:54 <vertis> Error executing action `create` on resource 'template[nginx.conf]'
 20:35:00 <vertis> Parent directory /opt/nginx/conf does not exist.
 20:35:17 <vertis> qrush: ?
 20:35:34 <qrush> https://github.com/rubygems/rubygems.org/blob/master/config/deploy.rb
 20:35:51 <vertis> ah
 20:36:03 <phlipper> qrush: i think someone built application deployment in to the chef deploy for the app role
 20:36:07 <vertis> i don't think we'll be using that
 20:36:12 <vertis> it's all in chef now
 20:36:16 <qrush> thats really strange
 20:36:22 <qrush> i dont understand why we can't use cap
 20:36:47 <phlipper> https://github.com/rubygems/rubygems-aws/blob/master/chef/site-cookbooks/rubygems/recipes/rails.rb#L45-L72
 20:36:56 <vertis> ask the opscode guys … /cc ssd7
 20:37:03 <phlipper> you could definitely still use cap, not sure who wrote that
 20:37:19 <qrush> it's kind of the standard - this doesn't cover rolling back or anything else that cap does
 20:38:18 <vertis> yeah, I personally wouldn't do it either way
 20:38:22 <vertis> but there you go
 20:38:47 scoutcamper → scoutcamper|away
 20:38:57 <vertis> qrush: that cap could be modified though
 20:39:06 <qrush> i really want to keep the cap workflow
 20:39:23 <qrush> it seems kind of crazy to rewrite what it does
 20:39:32 <qrush> maybe evan has different opinions - but i trust cap.
 20:39:53 <phlipper> not sure if the chef `application` block is the same as this or not: http://docs.opscode.com/resource_deploy.html
 20:40:36 <vertis> qrush: pretty sure the chef rails app deploy is supposed to mirror the cap functionality
 20:40:42 <vertis> it creates releases etc
 20:41:03 <vertis> you really should talk to the opscode guys though
 20:41:09 <vertis> they're the experts
 20:42:11 <ssd7> heya
 20:44:14 <ssd7> I didn't write either the cap stuff or the deploy stuff, but I can probably help work through what we have right now as I just had to grok much of it to get that vagrant branch working.
 20:47:25 <ssd7> vertis: The problem you hit is probably because the custom nginx stuff is only only the loadbalancer and I didn't take into account the "fullstack" role when doing that edit.
 20:47:36 <ssd7> s/only only/only on/
 20:47:58 <vertis> it was coderanger that wrote the deploy stuff
 20:48:08 <coderanger> yo
 20:48:12 <ssd7> We probably don't want the fullstack role on the app sever since that will also deploy the database there and..well...everything
 20:48:20 <vertis> coderanger -> qrush
 20:48:46 <vertis> so where was samkottler getting the node.json from then
 20:48:52 <coderanger> qrush: Does cover rolling back et al, uses almost the same file structure in fact :)
 20:48:53 <vertis> grrr
 20:49:16 <vertis> why the fuck wasn't this documented
 20:49:34 <vertis> this server is no better than hand rolled
 20:49:42 <vertis> if you can't fucking reproduce shit
 20:50:08 <ssd7> Right, as I mentioned above, when I started looking at the branch to fix the vagrant stuff, I'm not sure what has been going on in EC2 as it isn't clear to me that those cookbooks would actually work in a splitdb setup without modification
 20:50:34 <vertis> okay
 20:50:40 <ssd7> I'm guessing people decided that 'up' was better than 'down' and forged ahead.
 20:50:53 <qrush> coderanger: ok...just please document it
 20:50:58 — vertis growls
 20:51:03 <qrush> kind of scary that it's not cap :(
 20:51:15 <qrush> i dont get why the deploy process needs to be coupled to provisioning
 20:51:23 <coderanger> qrush: Same process as any other deploy, just update whereever you are storing the tag to deploy, chef will take care of the rest :)
 20:51:46 <coderanger> qrush: Because having logic in two places is the enemy?
 20:52:29 <vertis> sam has manually commented stuff out when running chef-solo
 20:52:38 <vertis> -if node["sudo"]["add_vagrant"]
 20:52:39 <vertis> -  sysadmins << "vagrant"
 20:52:40 <vertis> -end
 20:52:41 <vertis> +#if node["sudo"]["add_vagrant"]
 20:52:42 <vertis> +  #sysadmins << "vagrant"
 20:52:43 <vertis> +#end
 20:53:37 <someara> the chef-server-revision repo was su[er close to working %100 last night ~4am
 20:53:47 <vertis> doesn't appear to have changed the app server though
 20:53:47 <someara> s/su[er/super/
 20:54:04 <vertis> still installs fullstack
 20:54:05 <vertis> k
 20:55:21 <vertis> looks like postgres is running on the app server
 20:55:48 <vertis> and has been for quite a while
 20:55:57 <ssd7> vertis: Right, and if you look at the configs the app is likely pointing at localhost.
 20:55:58 <coderanger> You would have to rebuild the node, removing the role won't make chef uninstall things
 20:56:13 <vertis> will check that now
 20:57:15 <someara> so here's a question for 9pm on a friday
 20:57:22 <vertis> database.yml
 20:57:39 <vertis> is pointed at 10.249.66.172
 20:57:46 <vertis> which is….?
 20:57:49 <vertis> checking
 20:57:57 <vertis> the new db master
 20:58:08 <ssd7> Hrmm. So maybe there were cookbook changes not on master or I misread.
 20:58:19 <vertis> i can't see any
 20:58:32 <vertis> i rsynced it back to my working dir
 20:58:35 <vertis> to check
 20:58:43 <vertis> and those don't appear to be changed
 20:58:46 <vertis> i'll check again
 20:59:23 ⇐ ReinH quit ([email protected]) Read error: Connection reset by peer
 21:00:12 <vertis> yeah, no changes there
 21:00:29 <vertis> it is possible he was copying that in manually after running chef
 21:00:31 <vertis> i guess
 21:01:12 <phlipper> :|
 21:01:18 <vertis> i found the signing_key
 21:01:24 <vertis> phlipper: this is all fixable
 21:01:37 <ssd7> Yes.
 21:01:46 <vertis> we'll just get it working with changes saved to master
 21:01:49 <ssd7> You really just need to sort out some roles and attributes
 21:01:51 <vertis> rather than hacking around
 21:02:03 <phlipper> off, just sounds out of sync compared to where i thought we were
 21:02:11 <phlipper> ofc*
 21:02:15 <vertis> and the README needs to document what HAS to be copied in manually for security reasons
 21:02:27 <phlipper> yes
 21:02:40 <vertis> okay
 21:02:46 <vertis> give me five mins
 21:02:49 <vertis> and I'll try again
 21:03:23 <ssd7> Here is the branch I used to get the vagrant stuff working again.  As you can see, it is mostly just attr changes: https://github.com/stevendanna/rubygems-aws/compare/fixup-vagrant
 21:03:40 <ssd7> Some not appropriate for prod (postgres password for instance)
 21:03:44 → sferik joined ([email protected])
 21:04:44 <elskwid> vertis: I haven't been able to be much help with the actual setup short of some bad pull requests in the beginning but if you need help documenting I love a good README
 21:04:59 <elskwid> vertis: I would be happy to take rough notes and spruce em up
 21:05:22 <sferik> vertis: sorry I missed your pings earlier
 21:05:31 <sferik> vertis: back online now
 21:06:10 <stevenhaddox> ssd7: thanks for the link, kid time and then I can take a peak in 30-60 minutes-ish
 21:07:12 <vertis> coderanger: are you taking qrush through how to do rollbacks and the like?
 21:07:20 <vertis> elskwid: thanks
 21:07:30 <vertis> sferik: check login on LB
 21:07:35 <qrush> i dont know how to deploy, so thats first before rollbacks :)
 21:07:37 <coderanger> vertis: Sorry no, need to pay attention to work at the moment
 21:07:39 <vertis> we're just working on the app server now
 21:07:47 <vertis> qrush: I mean in theory
 21:07:49 <coderanger> Also not up to speed on how things are setup now
 21:08:00 <vertis> coderanger: it's your rails app code
 21:08:08 <vertis> and fair enough
 21:08:28 <vertis> it keeps the same directory structure as capistrano, so should have rollback right?
 21:08:42 <coderanger> Yes
 21:08:44 scoutcamper|away → scoutcamper
 21:08:50 <coderanger> and right now it is just deploying from master
 21:08:53 <vertis> qrush: I'll work with you shortly on your concerns
 21:09:02 <coderanger> You'll want to add https://github.com/rubygems/rubygems-aws/blob/master/chef/site-cookbooks/rubygems/recipes/rails.rb#L47 "revision node['something']"
 21:09:08 <vertis> yeah
 21:09:10 <coderanger> which should be the tag/branch you want to deploy from
 21:09:21 <ssd7> vertis: So, the first thing I would do is make sure that you have three roles (lb, app, and db) that install what you need on each.
 21:09:29 <coderanger> rollback is like any other release, except that chef notices if it has that copy cached on disk
 21:09:34 <sferik> vertis: okay, I'm able to ssh into 54.245.255.174 and sudo
 21:09:51 <vertis> ssd7: the fullstack one is what's currently on app
 21:10:03 <vertis> so I'll just get it working, THEN start changing roles
 21:10:29 <vertis> thanks coderanger
 21:10:45 <vertis> sferik: you'll be able to get into the other servers shortly then
 21:10:55 <sferik> awesome!
 21:11:00 <sferik> thank you so much
 21:17:00 <vertis> okay, I've audited all the difference between the server version and my working copy
 21:17:21 <vertis> only thing of note I can see is chef/nginx_signing.key
 21:17:31 <vertis> which I assume gets used somewhere
 21:17:50 <vertis> or perhaps not
 21:18:16 <ssd7> That is likely the ssl certificate key which probably was moved into place on the load balancer earlier?
 21:18:34 <ssd7> Just a guess from previous IRC comments
 21:20:35 <vertis> okay
 21:20:41 <vertis> lets see where this fails
 21:20:46 <vertis> and go from there
 21:20:49 <ssd7> +1
 21:20:58 <vertis> maybe back to cutting out the fullstack stuff
 21:21:03 <vertis> but not sure
 21:21:32 <ssd7> Having a fullstack role is good, but eventually that app server should have a role that just deploys an app server.
 21:21:41 <ssd7> Right now it will try to be a loadbalancer and an app server
 21:21:44 <ssd7> which isn't gonna work
 21:22:45 <vertis> back to the same
 21:22:45 <vertis> .. /opt/nginx/conf
 21:22:45 <vertis> not existing
 21:22:45 <vertis> cool
 21:22:45 <vertis> i wonder how sam was getting it installed
 21:23:53 ⇐ Boxcar21 quit ([email protected]) Quit: Leaving...
 21:24:02 <ssd7> vertis: one second, I probably can tell you the fix for that
 21:24:40 <ssd7> can you paste the full error message and some context for it
 21:24:44 <ssd7> into a gist or something
 21:25:48 <vertis> sure
 21:25:51 <vertis> one sec
 21:27:36 <someara> hi guys... been watching...
 21:27:45 <ssd7> np, I need to go eat soon.  But I was gonna have someara and jtimberman jump in here and help bang out a plan of attack
 21:28:13 <someara> ssd7 whats not in the chef-server-refactor branch?
 21:28:23 → Boxcar21 joined ([email protected])
 21:28:24 <someara> database tweaks + nginx-geoip
 21:28:26 <someara> ?
 21:28:30 <someara> (and certs of course)
 21:28:49 <ssd7> someara: jtimberman would know better. But that sounds correct.
 21:28:53 <vertis> https://gist.github.com/4695805
 21:28:53 <someara> k
 21:29:13 <ssd7> someara: also some stuff related to the fact that the real cluster I guess has ebs volumes with the data mounted and such
 21:29:13 <someara> right.
 21:29:20 <someara> okay.
 21:29:22 <someara> so
 21:29:36 <someara> vertis you're the guy with access?
 21:29:42 <vertis> one of
 21:30:07 <someara> whats more important? turning it on for reads and getting it working or doing it right?
 21:30:09 <someara> also
 21:30:15 <someara> its just a db right
 21:30:28 <vertis> this is the app server
 21:30:31 <someara> so we could dump tables, and reimport into a properly done infrastructure in the future
 21:30:46 <vertis> and the db master should be fine
 21:30:52 <vertis> i haven't looked at that yet
 21:31:00 <vertis> but it has it's own role
 21:31:12 <vertis> not sure about the credentials
 21:31:36 <vertis> nginx config is failing to install as per that gist
 21:31:46 <someara> okay. if I were in charge, I'd say its ince we have it %99 automated, lets do the last mile by hand, then we can migrate to the %100 automated infrastructure when the dust settles
 21:32:07 <vertis> can't run chef at all currently
 21:32:12 <someara> cause sam's skiiing, I cant stay around all night, etc
 21:32:14 <vertis> but yeah that's my opinion too
 21:32:31 <someara> word. so whats next
 21:32:32 <vertis> it's not the manual stuff I have a problem with
 21:32:39 <vertis> just that it's undocumented
 21:32:57 <someara> aha. whats left to document?
 21:33:07 <vertis> https://gist.github.com/4695805 <- current error when I run on app server
 21:34:44 <ssd7> vertis: So can you give a few more lines above that.  Basically, /opt/nginx is where the custom nginx for the load balancer lives.
 21:34:57 <vertis> hrrmm
 21:35:01 <someara>  /opt/nginx/con... is this adams nginx build?
 21:35:10 <vertis> sorry
 21:35:15 <vertis> knew that would happen
 21:35:20 <phlipper> vertis: is it an order-of-operations issue? https://github.com/rubygems/rubygems-aws/blob/master/chef/site-cookbooks/rubygems/recipes/nginx_source.rb#L25
 21:35:28 <someara> sec leme pull the repo. whats the working branch?
 21:35:33 <vertis> master
 21:36:07 <vertis> if we shouldn't be installing the LB on this node at all
 21:36:10 <vertis> and we shouldn't
 21:36:22 <ssd7> phlipper: like 10 above it should create /opt/nginx if the package is the same as it was earlier
 21:36:24 <vertis> then lets just create a role that has what we need
 21:36:25 <ssd7> vertis: +1
 21:36:40 <phlipper> vertis: +1
 21:36:49 <vertis> i need to create a pull request for some stuff I've changed
 21:36:51 <ssd7> vertis: I would say make an app server role and put the stuff in there that we need rather than try to make this massive role work together
 21:37:01 <ssd7> anyway, dinner for me. I can be back on later
 21:37:17 <vertis> ssd7: yep
 21:37:22 <someara> we already did the component refactoring stuff in the chef-server branch
 21:37:37 <someara> problem is its using platform packaged nginx
 21:37:55 <ssd7> someara: that is fine for the app server
 21:38:02 <ssd7> only lb needs geoip afaik
 21:38:13 <someara> hrm okay.
 21:38:16 <vertis> yeah
 21:38:22 <vertis> bear with me
 21:38:34 <vertis> making changes now
 21:41:11 <phlipper> dinner with the family, back in a bit
 21:44:30 <vertis> where is memcache/redis currently going to sit?
 21:45:43 <vertis> anyone?
 21:46:08 <someara> on the database node
 21:46:21 <vertis> okay so we don't need that either
 21:46:37 <vertis> who configured THAT on the app server though
 21:46:39 <someara> as of last night, it was going to look like this according to sam
 21:46:39 <someara> https://gist.github.com/8495e57f013a20655e20
 21:47:09 <someara> if I had to guess, it would be redis configured from cookbook, with the database manually tweaked
 21:47:23 <someara> or tweaked with code from the master branch
 21:47:31 <vertis> redis anyway
 21:47:38 <vertis> memcache is still on app server
 21:48:08 <jtimberman> howdy
 21:48:10 <someara> yep
 21:48:15 <vertis> hey
 21:48:45 <vertis> i don't actually have the rubygems_redis mentioned in that gist
 21:49:16 <jtimberman> vertis: thats in the chef-server-refactor branch on my fork
 21:49:26 <vertis> sure
 21:49:30 <jtimberman> https://github.com/jtimberman/rubygems-aws/blob/chef-server-refactor/chef/roles/rubygems_redis.rb
 21:49:36 <vertis> but we're not using that till after
 21:49:46 <vertis> and he has it working in *prod*
 21:49:52 <vertis> so SOMEHOW
 21:49:59 <vertis> this was getting done
 21:50:18 <someara> how were the chef recipes being applied before? cap?
 21:50:37 <vertis> i think knife cook
 21:50:49 <vertis> before what though?
 21:50:58 <vertis> before sam left?
 21:51:06 <someara> yea
 21:51:12 <jtimberman> who has ssh access to the running instances?
 21:51:15 <vertis> yeah I THINK knife cook
 21:51:19 <vertis> jtimberman: i do
 21:51:24 <someara> knife-solo you mean?
 21:51:45 <someara> f only we had a way to use an API to get answers about the infrastructure.
 21:51:47 <jtimberman> someara: knife cook is from solo
 21:51:49 <jtimberman> from knife-solo
 21:51:54 <vertis> e.g. knife cook [email protected] nodes/balancer.rubygems.org.json
 21:52:06 <someara> the lengths people go out of their way to avoid server amazes me.
 21:52:19 <vertis> which doesn't make tooo much difference
 21:52:26 <vertis> someara: we do at work as well
 21:52:55 <vertis> ANYWAY
 21:52:58 <someara> so whats the high level stragegy? hand-band-aid, accept traffic, documenting our steps along the way, then migrating to a %100 automated setup?
 21:52:59 <someara> or
 21:53:03 <someara> continue iterating
 21:53:08 <someara> or
 21:53:25 <someara> i dunno
 21:53:41 <vertis> We're accepting a few unautomated things
 21:53:53 <vertis> such as the ssl certs
 21:54:16 <vertis> but putting down config for redis wasn't one of them
 21:54:29 <jtimberman> where is redis running?
 21:55:06 <someara> should be on the database
 21:55:23 <vertis> currently it's on the app server
 21:55:31 <vertis> it is not running on the db server
 21:55:44 <vertis> given only one app server currently
 21:55:48 <vertis> it can stay there
 21:55:54 <jtimberman> yeah thats fine.
 21:55:58 <vertis> question answered for now
 21:56:00 <vertis> okay
 21:56:01 <vertis> i
 21:56:08 <vertis> i've created the new role
 21:56:39 <someara> is redis running there and we cant figure out how it got there?
 21:56:47 <someara> what did the solo workflow look like?
 21:56:56 <jtimberman> it was baked into the "almost_fullstack" and "fullstack" roles.
 21:56:57 <someara> and there are long running nodes right?
 21:57:05 <someara> fullstack worked the other night
 21:57:15 <someara> minus db tweaks and geongix
 21:57:55 <samkottler> hey
 21:57:57 <someara> checking the vagrant file... where are the run_lists being injected?
 21:58:09 <vertis> jtimberman: almost_fullstack was mine
 21:58:14 <samkottler> I'm on my phone but can answers q's
 21:58:15 <vertis> but was for vagrant
 21:58:24 <vertis> -_-
 21:58:44 <vertis> samkottler: not sure how you had things working
 21:58:59 <jtimberman> brb
 21:59:08 <vertis> fullstack wouldn't install on the app server for me
 21:59:17 <vertis> and thats what it was set to use
 21:59:18 <jtimberman> ok
 21:59:20 <jtimberman> that's fine
 21:59:37 <vertis> samkottler: mostly have it working now
 21:59:41 <vertis> fingers crossed
 21:59:48 <jtimberman> Okay so, how can someara and I help here?
 21:59:50 <stevenhaddox> ssd7: I'm not sure how much I can give tonight before I'm completely drained, did you finish up with the vagrant stuff or is there more important things going on that we shouldn't worry about it right now?
 21:59:57 <stevenhaddox> ssd7: If you finished it, did it get merged back in?
 22:00:28 <samkottler> I got a gchat about an issue with solo. is that resolved?
 22:00:43 <vertis> jtimberman, someara: you've answered my questions for now
 22:00:48 <jtimberman> vertis: ok.
 22:00:51 <vertis> samkottler: it is now yeah
 22:01:00 <vertis> okay the app server is deployed
 22:01:01 <samkottler> what was the problem?
 22:01:19 <vertis> samkottler: quite a few things
 22:01:47 <vertis> samkottler: can document in an email
 22:02:06 <vertis> i'll just create a pull request with these changes now
 22:02:06 <samkottler> shk linux.com
 22:02:06 <jtimberman> vertis: can you document in a wiki page?
 22:02:15 <vertis> and then README
 22:02:20 <vertis> jtimberman: yes
 22:02:27 <jtimberman> sweet.
 22:02:34 <vertis> i'm all about the documentation of this process
 22:03:37 <samkottler> great thanks vertis
 22:04:30 <jtimberman> Cool. I've got this in a corner, mention me if you have any questions.
 22:04:41 <jtimberman> I'm going to work on more of the chef-server-refactor stuff.
 22:04:42 → GitHub145 joined ([email protected])
 22:04:42 <GitHub145> [rubygems-aws] vertis opened pull request #87: Getting things working from lb and app server (master...master)  http://git.io/5yjFJQ
 22:04:42 ← GitHub145 left ([email protected])
 22:06:08 <phlipper> vertis: merged
 22:06:10 → GitHub109 joined ([email protected])
 22:06:10 <GitHub109> [rubygems-aws] phlipper pushed 3 new commits to master: http://git.io/cYFc_A
 22:06:10 <GitHub109> rubygems-aws/master 6656f4b Luke Chadwick: Make cap rubygems.org chef:* work
 22:06:10 <GitHub109> rubygems-aws/master 0bc9253 Luke Chadwick: App server should install app role (for now)
 22:06:10 <GitHub109> rubygems-aws/master 906ac54 Phil Cohen: Merge pull request #87 from vertis/master...
 22:06:10 ← GitHub109 left ([email protected])
 22:07:26 <vertis> okay
 22:07:42 <vertis> lets just check it all still works
 22:09:24 <vertis> 1,259,533,358 downloads of 0 gems cut since July 2009
 22:09:36 <vertis> where does the 0 gems cut come from
 22:09:42 <qrush> Rubygem.count
 22:10:21 <qrush> https://github.com/rubygems/rubygems.org/blob/master/app/views/home/index.html.erb#L5
 22:10:26 → mr_ndrsn joined ([email protected])
 22:10:27 <qrush> https://github.com/rubygems/rubygems.org/blob/master/app/controllers/home_controller.rb#L3
 22:10:35 <qrush> https://github.com/rubygems/rubygems.org/blob/master/app/models/rubygem.rb#L49
 22:10:43 <vertis> thanks
 22:10:43 <qrush> ahem, it's with_versions.count.
 22:11:06 <vertis> so
 22:11:16 <vertis> that's coming out of postgres
 22:11:40 <qrush> yes
 22:11:41 <vertis> it shouldn't be zero surely?
 22:11:50 <qrush> No. :)
 22:11:59 <vertis> I thought we had a full dump in postgres
 22:12:29 <vertis> qrush, do you want to set 54.245.255.174 rubygems.org www.rubygems.org
 22:12:35 <vertis> in your hosts file and have a play
 22:12:57 <qrush> www.rubygems.org should redirect to rubygems.org
 22:13:13 <vertis> sure
 22:14:18 <qrush> yeah doesnt look like pg is working
 22:14:22 <qrush> or has an empty db
 22:14:23 phlipper → phlipper_
 22:14:28 <qrush> the downloads count is coming from redis
 22:14:34 <vertis> okay cool
 22:14:40 <qrush>  /stats looks sad
 22:14:42 <vertis> might be pointed at localhost
 22:14:47 <vertis> instead of where it should be
 22:18:18 <vertis> yep
 22:18:22 <vertis> localhost database
 22:18:31 <jtimberman> vertis: in database.yml?
 22:18:40 <vertis> chef recipes and/or node config needs to be added
 22:18:45 <vertis> jtimberman: yeah
 22:19:34 <vertis> tried restarting /etc/init.d/rubygems
 22:19:38 <vertis> it's not coming back
 22:19:44 <vertis> or it's taking it's sweet time
 22:20:23 <envygeeks> btw on Ubuntu you can do `restart rubygems`
 22:20:27 <jtimberman> that's this: https://github.com/rubygems/rubygems-aws/blob/master/chef/site-cookbooks/rubygems/recipes/rails.rb#L62
 22:20:50 <vertis> envygeeks: I do know that, old habits die hard
 22:21:01 <jtimberman> which we've done on the chef-server-refactor w/ search. for solo, i guess make the app role have an attribute and use that attribute in the recipe.
 22:21:14 <envygeeks> can't hate you for it vertis took me a long time to get over it too :P
 22:21:25 <vertis> envygeeks: also I mostly do centos
 22:21:33 <vertis> because that's what we use at work
 22:21:52 <vertis> jtimberman: I wish we weren't rushing off to do the server stuff before we had this in prod
 22:22:09 <vertis> we all agreed at the start that we would use chef-solo
 22:22:14 <teancom> Giving a talk on Why Being a Sysadmin is Awesome (No, Really!) tomorrow. (qrush - earmuffs. No spoilers for you!) Soliciting one-liners of why you guys think it's great. I'll go first - I love that the better I'm doing my job, the less I have to do (or at least, I then have time to do the *interesting* work). Developers who finish early just get their next project assigned. I will only accept answers for 5 minutes cuz I don't want to distract
 22:22:14 <teancom> from real work.
 22:22:53 <vertis> -_- OUT!
 22:22:56 <vertis> lol
 22:23:09 <envygeeks> I don't like being a sysadmin because I have to deal with idiots all day, idiots who think they can tell me how to run the systems they broke
 22:23:39 <jtimberman> I don't think this is going to lead to anything constructive.
 22:23:50 <teancom> :-/
 22:23:51 <mr_ndrsn> I love being able to tell the revs exactly what they did that broke the system...
 22:23:55 <jtimberman> vertis: I explained yesterday what someara and I were up to and why :).
 22:23:57 <mr_ndrsn> Err, debs
 22:24:05 <teancom> I actually do love being a sysad...
 22:24:09 <mr_ndrsn> Ducking autocorrect
 22:24:13 <vertis> jtimberman: sure
 22:24:36 <envygeeks> mr_ndrsn, sadly nobody ever asks me how :( they just demand I fix it and that it must have been my fault... then I get angry and mumble "it is because i let you have sudo >.>"
 22:24:48 → Mab879 joined ([email protected])
 22:25:40 <mr_ndrsn> Envygeeks: I never said they asked.  They get postmortems as a matter of course.
 22:27:13 ⇐ justincampbell quit ([email protected]) Remote host closed the connection
 22:28:14 <vertis> jtimberman: really some of this stuff should have been done to chef solo first though
 22:36:17 <vertis> okay
 22:36:27 <vertis> so after I changed the db config
 22:36:30 <vertis> (manually)
 22:36:43 <vertis> the db server isn't accepting connections
 22:37:00 <mr_ndrsn> Firewall?
 22:37:14 <vertis> maybe
 22:37:19 <vertis> or postgres settings
 22:37:40 <jtimberman> pg_hba.conf
 22:37:57 <vertis> :)
 22:37:59 <jtimberman> something like this: host rubygems_production rubygems 10.0.0.0/8 md5
 22:38:04 <vertis> jtimberman: excellent
 22:38:12 <mr_ndrsn> Is pg only listening on local host?
 22:38:18 <jtimberman> check that too :)
 22:38:42 <vertis> i would have thought we could get this into a role though
 22:39:47 <envygeeks> doesn't aws have sticky internal ip's now so that you don't have to open up like that in hba?
 22:39:57 <jtimberman> https://github.com/rubygems/rubygems-aws/blob/master/chef/roles/rubygems_db_master.rb#L18
 22:40:27 → kyd joined ([email protected])
 22:40:58 <mr_ndrsn> Man, I need to learn pg. 😳
 22:41:58 → ben_h joined ([email protected])
 22:42:20 <vertis> jtimberman: that doesn't seem to be being applied
 22:42:29 <vertis> i.e. it's not in the file
 22:42:33 <vertis> and that's the role
 22:42:42 <jtimberman> is the role correct on the system?
 22:42:45 <jtimberman> solo right?
 22:42:48 <vertis> maybe not
 22:44:25 <vertis> it is on the server
 22:44:39 <vertis> the chef run must be wrong somehow
 22:44:44 <vertis> chef-solo*
 22:45:21 <mr_ndrsn> This is ec2, not vagrant, right?
 22:45:49 <vertis> DEPLOY_USER=vertis DEPLOY_SSH_KEY=$HOME/.ssh/id_rsa cap rubygems.org chef:dbmaster
 22:45:51 <vertis> yeah
 22:45:53 <vertis> ec2
 22:46:00 <vertis> uploading
 22:46:36 → claco joined ([email protected])
 22:47:25 <vertis> * executing "sudo -p 'sudo password: ' /bin/bash -c 'cd /home/vertis/chef && /usr/bin/chef-solo -c solo.rb -j nodes/dbmaster.rubygems.org.json'"
 22:47:25 <vertis>     servers: ["ec2-54-245-133-190.us-west-2.compute.amazonaws.com"]
 22:47:27 <vertis>     [ec2-54-245-133-190.us-west-2.compute.amazonaws.com] executing command
 22:47:28 <vertis>  ** [out :: ec2-54-245-133-190.us-west-2.compute.amazonaws.com]
 22:47:30 <vertis>  ** [out :: ec2-54-245-133-190.us-west-2.compute.amazonaws.com] [2013-02-02T03:46:15+00:00] INFO: *** Chef 10.18.2 ***
 22:47:31 <vertis>  ** [out :: ec2-54-245-133-190.us-west-2.compute.amazonaws.com] [2013-02-02T03:46:16+00:00] INFO: Setting the run_list to ["role[rubygems_db_master]"] from JSON
 22:47:32 <vertis>  ** [out :: ec2-54-245-133-190.us-west-2.compute.amazonaws.com] [2013-02-02T03:46:16+00:00] INFO: Run List is [role[rubygems_db_master]]
 22:47:33 <vertis>  ** [out :: ec2-54-245-133-190.us-west-2.compute.amazonaws.com] [2013-02-02T03:46:16+00:00] INFO: Run List expands to [rubygems::users, apt, build-essential, xfs, curl, ntp, hostname, resolver, rsync, rubygems::system_ruby, xml, xslt, logrotate, logwatch, bash-completion, grc, screen, openssh, sudo, denyhosts, iptables, rubygems::ip_security, rubygems::iptables, htop, iftop, monit, newrelic-sysmond, munin, postfix, postfi
 22:47:37 <vertis> so it's definitely running the right thing
 22:48:50 <vertis> it's just not putting down that config
 22:50:27 <mr_ndrsn> Does the pg recipe take that json hash and write it to the config file?
 22:50:38 <vertis> don't know
 22:50:46 <vertis> given it's there
 22:50:53 <vertis> i would assume it's worked in the past
 22:50:57 <vertis> jtimberman: thoughts?
 22:51:30 <mr_ndrsn> I'm only asking because I don't know chef as well as I know puppet
 22:51:50 <jtimberman> vertis: looking
 22:52:16 <vertis> jtimberman: me too
 22:52:56 <vertis>  node[:postgresql][:pg_hba_defaults]
 22:53:04 <vertis> the hash is wrong
 22:53:18 <vertis> "pg_hba" currently
 22:53:25 <vertis> not pg_hba_defaults
 22:53:32 <mr_ndrsn> Woo!!!  I helped!
 22:53:52 <vertis> mr_ndrsn: :)
 22:54:37 <jtimberman> heh
 22:54:39 <jtimberman> :)
 22:56:27 <jtimberman> https://gist.github.com/869dcce2638061a41d07
 22:56:30 <jtimberman> should be good
 22:57:04 → GitHub166 joined ([email protected])
 22:57:04 <GitHub166> [rubygems-aws] anoldguy opened pull request #88: Fix json hash key for pg_hba (master...patch-2)  http://git.io/gmIYsQ
 22:57:04 ← GitHub166 left ([email protected])
 22:58:35 — ssd7 is back.
 22:58:53 <mr_ndrsn> Whoops 😳 the fix wasn't to rename the hash key to pg_hba_defaults?  I can close the PR
 22:59:04 <jtimberman> i mean, you can do that.
 22:59:08 <jtimberman> but the template will iterate over a hash
 22:59:17 <jtimberman> which is what i'm going to do to dynamically set this in the chef-server refactor :)
 22:59:30 <ssd7> Yo, let me know how I can help.
 23:00:34 <vertis> jtimberman: ?
 23:00:53 <vertis> the erg is looking for :pg_hba_defaults
 23:01:06 <vertis> erb*
 23:01:38 <mr_ndrsn> Btw, that github bot is sweet.
 23:02:10 <jtimberman> vertis: line 75 in the erb
 23:02:17 <qrush> sorry it wasn't set up properly before. been a crazy week
 23:02:19 <jtimberman> <% node["postgresql"]["pg_hba"].each do |hba| %>
 23:02:21 <jtimberman> :)
 23:02:48 <vertis> ah
 23:02:56 <vertis> yeah, wasn't reading that right
 23:03:00 <jtimberman> :)
 23:04:00 <ssd7> The pg_hba stuff should look like this
 23:04:00 <ssd7> https://github.com/stevendanna/rubygems-aws/blob/fixup-vagrant/chef/roles/vagrant.rb#L36-L42
 23:04:15 <ssd7> but with the correct database name, ip, etc
 23:04:54 <vertis> ssd7: thanks
 23:05:07 <vertis> it was more than vagrant borked though
 23:05:14 <ssd7> vertis: Yes, I know :)
 23:05:24 <vertis> i'm less than impressed at this point
 23:05:51 <jtimberman> with?
 23:06:12 <vertis> things not working at all
 23:06:33 <vertis> there was this whole it's working its working thing going on
 23:06:44 <vertis> and yet I can't take chef and run it
 23:06:46 <qrush> pateience man :)
 23:06:47 <vertis> and have it work
 23:07:01 <vertis> qrush: I am calm, I am calm
 23:07:07 <qrush> we have the time to do this right
 23:07:07 <vertis> sorry
 23:07:16 <vertis> qrush: yes I know
 23:07:32 <jtimberman> qrush: that's what i'm workin on :)
 23:08:01 <mr_ndrsn> How many pieces are converged correctly?
 23:08:08 <vertis> okay
 23:08:21 <vertis> qrush: the disabling message
 23:08:27 <vertis> is that in the db somewhere?
 23:08:34 → ReinH joined ([email protected])
 23:08:35 <ReinH> hai!
 23:08:40 <vertis> ReinH: hi
 23:08:45 <mr_ndrsn> Ohai
 23:08:49 <ReinH> great job everyone :)
 23:09:08 <qrush> vertis: disabling message?
 23:09:11 <qrush> sorry what?
 23:09:18 <vertis>  Pushing new gems is temporarily disabled due to security issues. Check the status site for more info. hide
 23:10:14 <qrush> thats the Announcements table
 23:10:18 <qrush> Announcement.first.destroy
 23:10:19 <vertis> okay
 23:10:24 <vertis> yeah that's fine
 23:10:30 <vertis> it's working now
 23:10:46 <vertis> I just need to go back and document everything I had to do
 23:11:11 <vertis> and create one pull request (maybe)
 23:11:20 <vertis> i think someone else may have
 23:11:31  → GitHub74 joined  ⇐ mr_ndrsn quit  ↔ envygeeks nipped out  
 23:14:50 <GitHub74> [rubygems-aws] vertis opened pull request #89: Put pg attributes in the correct format (master...master)  http://git.io/JNwi8Q
 23:14:50 ← GitHub74 left ([email protected])
 23:15:08 <vertis> anyone want to +1
 23:15:08 <evan> ok!
 23:15:14 <vertis> hey evan
 23:15:20 <vertis> welcome back
 23:15:34 <vertis> it's been quite a ride
 23:15:40 <ssd7> vertis: looks good
 23:15:45 <vertis> thanks
 23:15:49 <evan> oh? hows things going?
 23:16:15 <vertis> chef wasn't working on any of the boxes
 23:16:28 <vertis> I've fixed that
 23:16:32 <evan> interesting
 23:16:51 <vertis> last outstanding thing is the database.yml on the application server
 23:17:00 <vertis> which is challenging
 23:17:10 <evan> because of where to put it?
 23:17:16 <vertis> has password
 23:17:18 <vertis> yeah
 23:17:32 <ssd7> vertis: My suggestion would be drive it with a databag that we don't check into git
 23:17:41 <vertis> yeah will do
 23:17:48 <vertis> same as the production ssl certs
 23:17:50 <vertis> for now
 23:18:10 <ssd7> you can have a generic one for vagrant and then people who have the real details edit it for production
 23:18:20 <vertis> ssd7: yep
 23:18:28 <vertis> so long as it's documented it's fine
 23:18:35 <ssd7> Yup
 23:18:43 <vertis> evan do you want to start testing it?
 23:18:49 <evan> I do!
 23:18:59 <evan> we can start with the LBs
 23:19:02 <evan> what are their IPs?
 23:19:06 <vertis> 54.245.255.174 rubygems.org
 23:19:14 <vertis> there is only one LB currently
 23:19:18 <vertis> that I'm aware of
 23:19:23 <vertis> though the config is the same
 23:19:32 <vertis> so spinning up the second should be easy
 23:19:45 <evan> we'll be fine with 1 to start
 23:20:00 <vertis> 54.245.255.174 then
 23:20:18 <vertis> it's documented https://github.com/rubygems/rubygems-aws/wiki/Machines
 23:20:21 <evan> mmm
 23:20:28 <evan> I love a tidy "ps axw"
 23:20:45 <vertis> i take it the old one isn't?
 23:20:50 <evan> it's ok
 23:20:57 <evan> the LBs are pretty clean
 23:21:28 <vertis> the new app server is currently messy
 23:21:39 <vertis> has a not in use postgres
 23:21:48 <evan> eh?
 23:21:53 <vertis> cos it was there before the split happend
 23:21:58 <vertis> happened*
 23:22:16 <vertis> the machine hasn't been rebuilt from scratch
 23:22:24 <evan> gotcha
 23:22:26 <evan> thats ok
 23:22:33 <evan> hm, my password on the LB isn't working
 23:22:43 <vertis> hmmm
 23:22:53 <evan> for sudo
 23:23:03 <vertis> did you give that to me or sam?
 23:23:13 <evan> sam
 23:23:16 <vertis> do you have a copy of the hash you can give to me?
 23:23:24 <vertis> i'll run chef abain
 23:23:25 <vertis> 
 23:23:27 <qrush> hey looks like the aws setup has pg working
 23:23:29 <vertis> again*
 23:23:35 <vertis> qrush: yes
 23:23:38 <vertis> yes it does
 23:23:39 <evan> k
 23:23:39 <qrush> should i try to push a gem?
 23:23:45 <vertis> qrush: up to you
 23:23:52 <qrush> oh it's probably disabled :P
 23:24:00 <vertis> you can fix that though
 23:24:08 <vertis> your key should be on the box
 23:24:12 <vertis> app server box
 23:24:31 <evan> vertis: PM'd a hash
 23:24:35 <qrush> HTTP/1.1 500 Internal Server Error on curl -X POST /api/v1/gems
 23:24:36 <vertis> evan: cool
 23:24:50 <vertis> qrush: I don't pretend to know what that means
 23:24:51 <qrush> should be a 503
 23:24:54 <evan> qrush: i'm trying to look at the LB config
 23:24:54 <vertis> just cos disabled
 23:25:00 <evan> but I can't until I've got sudo
 23:25:03 <vertis> one moment
 23:25:54 <vertis> evan: it's just running now
 23:25:59 <evan> thanks
 23:26:16 <evan> so
 23:26:21 <evan> why does the LB have 2 nginx's running?
 23:26:30 <evan> one from /usr/sbin
 23:26:35 <evan> and one from /opt/nginx/sbin
 23:27:15 <evan> ok, sudo is working.
 23:27:16 <ssd7> evan: Because no one cleaned out the old one before deploying the new one
 23:27:28 <evan> which is the old one?
 23:27:29 <ssd7> evan: the one in opt is the custom built package with geoip
 23:27:49 <evan> does the old one just need the packages nuked?
 23:28:53 <ssd7> evan: I don't know what has happened on the server preciesly so I can't say exactly what needs to happen.  But I would kill the processes, remove the nginx packages and then let chef reinstall them.
 23:29:07 <evan> well, chef shouldn't be installing them
 23:29:10 <evan> I assume
 23:29:13 <evan> if we've got a custom nginx
 23:29:16 <ssd7> Chef installs the custom package
 23:29:58 <ssd7> that happens here: https://github.com/rubygems/rubygems-aws/blob/master/chef/site-cookbooks/rubygems/recipes/nginx_source.rb
 23:30:03 <vertis> i think we should destroy the servers and start from scratch
 23:30:06 <vertis> except db
 23:30:19 <evan> i'd prefer to not do that
 23:30:30 <vertis> evan why not?
 23:30:31 <evan> unless we've already done that once before
 23:30:36 <vertis> this should be a quick process
 23:30:38 <evan> but I don't want to waste an hour waiting for it to rebuild
 23:30:55 <vertis> we can spin up new ones before destroying old
 23:31:21 <vertis> I hear you
 23:31:25 <envygeeks> evan just snapshot the server at it's current state incase something goes wrong, and then you can feel safe about destroying it
 23:31:37 <vertis> just build first
 23:31:38 <evan> envygeeks: it's less that
 23:31:44 <vertis> destroy second
 23:31:50 <jtimberman> evan: in my testing for the chef-server-refactor branch, the app server takes the longest @ 5 minutes.
 23:31:52 <evan> and more that I want to get push access up in the next 30-60m
 23:32:07 <qrush> i'm seeing nothing in https://console.aws.amazon.com/ec2/v2/home?region=us-west-2
 23:32:11 <vertis> jtimberman: yeah
 23:32:23 <qrush> are these instances under a different AWS account than the S3 bucket?
 23:32:26 <vertis> qrush: I thought is was us-west-1
 23:32:27 <jtimberman> can take up to 7 minutes if shit is slow (like ubuntu mirrors etc)
 23:32:28 <evan> qrush: yeah
 23:32:30 <envygeeks> ah yeah I hear you on that evan, lack of push access is killing me because I want to push up new versions of my gems that are signed with my pub key
 23:32:33 <evan> qrush: they're not in your account
 23:32:38 <qrush> thank god :)
 23:32:39 <evan> the S3 bucket is your own account
 23:32:57 <evan> envygeeks: right, there is a lot of pent up demand.
 23:32:59 <qrush> i'd like to move that off soon - maybe lsegal can help us with that
 23:33:05 <evan> yeah
 23:33:10 <evan> we'll need help from amazon
 23:33:12 <evan> to move the S3 bucket
 23:33:23 <qrush> yay > 1 bus number!
 23:33:28 <vertis> lsegal == rubycentral
 23:33:39 <evan> vertis: ?
 23:33:42 <vertis> evan: how big is it in total?
 23:33:54 <envygeeks> I thought lsegal was AWS?
 23:33:59 <evan> 80G I think.
 23:34:04 <evan> envygeeks: he's at amazon now, yeah.
 23:34:16 <vertis> ah amazon
 23:34:21 <vertis> nm
 23:34:42 <vertis> that's not too big
 23:35:08 <vertis> why not adjust rubygems.org to push to two buckets
 23:35:15 <vertis> one on the new account
 23:35:18 <qrush> lets not worry about this right now
 23:35:23 <vertis> and then just slowly sync
 23:35:27 <vertis> then switch across
 23:35:44 ⇐ ben_h quit ([email protected]) Quit: ben_h
 23:36:04 <ssd7> So, fwiw,I am +1 on rebuiding the LB provided someone has the ssl certs.
 23:36:12 <evan> O_o
 23:36:15 <evan> I nuked the old nginx
 23:36:16 <vertis> i have the certs
 23:36:22 <evan> and the new one isn't listening on any ports...
 23:36:34 <envygeeks> did you have the certs resigned for the new server?
 23:36:40 <ssd7> evan: Try restarting it
 23:36:42 <vertis> the ones phlipper_ generated
 23:36:47 <envygeeks> oh okay
 23:36:50 <ssd7> evan: It probably couldn't bind the first time
 23:38:16 <evan> hm, /etc/init.d/nginx is busted.
 23:39:06 <ssd7> evan: That is generated by a template, mind posting it and I can probably tell why it wasn't generated correctly
 23:39:12 <ssd7> via a gist.
 23:39:15 <evan> fark.
 23:39:23 <evan> the new nginx is busted.
 23:39:25 <evan> there is no binary
 23:39:28 <envygeeks> :/
 23:39:32 <evan> there is no /opt/nginx/sbin directory at all.
 23:39:49 <ssd7> evan: Let's run chef on that machine and go from there
 23:39:56 <envygeeks> did somebody accidently build it with the --layout=debian without the --prefix=/opt/debian?
 23:40:10 <envygeeks> --prefix=/opt/nginx*
 23:40:21 <vertis> https://github.com/rubygems/rubygems-aws/wiki/Deploying-Prod-(WiP)
 23:40:22 <evan> how are we running chef on these machines? just `chef-client` manually?
 23:40:47 <vertis> evan: it's chef-solo
 23:41:06 <vertis> chef has been run recently
 23:41:12 <vertis> when I did evan's password
 23:41:15 <vertis> if it was right
 23:41:26 <vertis> then we wouldn't be having this conversation
 23:42:06 <evan> maybe my "apt-get remove nginx" screwed it up
 23:42:13 <evan> which I did to remove the old nginx
 23:42:15 <vertis> possibly
 23:42:16 <ssd7> evan yes that is my assumption
 23:42:19 <vertis> let me rerun
 23:42:24 <ssd7> vertis: thank you.
 23:42:32 <evan> oh actually
 23:42:34 <evan> it was me.
 23:42:39 <vertis> :D
 23:42:39 <evan> i'm a big dummy.
 23:42:44 <vertis> running now
 23:42:48 <evan> I didn't note the complete listing of "dpkg -S nginx"
 23:42:57 <evan> sorry guys!
 23:43:06 <envygeeks> did I see somebody type remove instead of autoremove --purge?
 23:43:11 <vertis> did you see my wiki page
 23:43:37 <vertis> i'm working through the others (app, db)
 23:43:38 <vertis> now
 23:43:56 <vertis> but, so you can all reproduce what I'm doing
 23:44:05 <vertis> that chef run is finished on the LB
 23:44:23 <evan> ok, there we go.
 23:44:33 <evan> thanks!
 23:46:02 <evan> vertis: what command are you running to execute chef-solo from your machine?
 23:46:15 <vertis> in the wiki page
 23:46:17 <ssd7> evan: https://github.com/rubygems/rubygems-aws/wiki/Deploying-Prod-%28WiP%29
 23:46:29 <vertis> which I'm still writing
 23:46:40 <qrush> i'm out for the night - just wanted to say thanks to everyone for kicking ass with this.
 23:46:46 <evan> hm, now nginx isn't listening on 443
 23:47:28 <vertis> qrush: talk to you tomorrow
 23:47:31 <evan> oh weird
 23:47:37 <evan> chef-solo started an nginx
 23:47:42 <evan> but not with the right config
 23:47:45 <evan> so a restart fixed it.
 23:47:56 <vertis> recipe must be wrong
 23:48:02 <vertis> i'll look into it
 23:48:08 <evan> thanks!
 23:49:38 <evan> how should I be changing the nginx config?
 23:49:39 <vertis> actually they all notify to reload/restart immediately
 23:49:43 <ssd7> vertis: Looks like potentially just flipping lines 28 and 29 of nginx_source will get you there, although the noticiations should have restarted it with the right config
 23:49:55 <vertis> ssd7: yeah
 23:50:16 <vertis> the config changes much later
 23:50:31 <vertis> i can't see how those two switching would make a difference
 23:50:42 <ssd7> vertis: Right, neither can I.
 23:50:53 <vertis> maybe reload doesn't work
 23:50:58 <ssd7> evan: via the cookbooks https://github.com/stevendanna/rubygems-aws/blob/fixup-vagrant/chef/site-cookbooks/rubygems/templates/default/nginx_balancer.conf.erb
 23:51:02 <ssd7> that is the load balancer
 23:51:08 <evan> k
 23:51:15 <ssd7> https://github.com/stevendanna/rubygems-aws/blob/fixup-vagrant/chef/site-cookbooks/rubygems/templates/default/nginx_application.conf.erb
 23:51:24 <ssd7> that is the nginx that runs in front of unicorn on the app server
 23:51:43 <ssd7> opps
 23:51:49 <ssd7> just realized I was linking to my account
 23:51:56 <ssd7> anyway, same path in the main repo should work :)
 23:51:57 <evan> I got it
 23:51:57 <evan> :)
 23:52:10 <evan> I need to add the lines for the bundler API
 23:52:14 <stevenhaddox> Sorry I couldn't help more tonight. Unfortunately this is all unfamiliar territory for me. I'll catch up on backchat tomorrow when I can and see if I can help in some way... Good luck everyone.
 23:52:23 <evan> stevenhaddox: thanks for helping!
 23:52:50 <stevenhaddox> evan: heh, really wish it was more. You guys are all amazeballs.
 23:53:28 <vertis> evan: is the geoip stuff where it should be?
 23:53:34 <evan> yep
 23:53:36 <evan> looks good.
 23:53:46 <vertis> *shrug*
 23:53:53 <vertis> there are reloads and restarts everywhere
 23:54:00 <vertis> in the balancer config
 23:54:06 <vertis> can't explain why it's not working
 23:54:30 <vertis> unless it can't find the service maybe?
 23:54:38 <vertis> notifies :reload, "service[nginx]", :immediately
 23:54:54 <vertis> do we know that it can find service[nginx]
 23:54:59 <ssd7> It would complain pretty loudly if that was the case.
 23:55:02 <vertis> even though it was installed from source
 23:55:04 <vertis> okay
 23:55:52 <vertis> i can't see it complaining in the logs
 23:55:52 <ssd7> So, if we stop nginx and rerun chef, does it happen or is this only on first install?
 23:56:22 <vertis> does it call out the notifies in the logs?
 23:56:27 <vertis> i thought that it did
 23:56:31 <vertis> and I can't see any
 23:56:49 <ssd7> if you don't see any it means that the notifications didn't trigger because the config didn't change
 23:57:17 <vertis> ah
 23:57:17 <ssd7> chef only sends the notify if it had to update the resource
 23:57:20 <vertis> that would do it
 23:57:36 <vertis> and because we had to reinstall the actual nginx but not the config
 23:57:41 <vertis> no restart/reload
 23:57:51 <vertis> evan: solved
 23:58:05 <evan> k
 23:58:14 <evan> can I go ahead and run chef?
 23:58:21 stevenhaddox → stevenhaddox|afk
 23:58:25 <vertis> do you have the certs?
 23:58:29 <evan> I do.
 23:58:38 <vertis> if you follow my instructions it should be fine
 23:58:42 <evan> k!
 23:58:47 <vertis> and if it's not then that's good too
 23:58:52 <evan> here
 23:58:54 <evan> we
 23:58:56 <evan> go!
 23:59:30 <evan> btw guys
 23:59:33 <evan> this setup is super rad.
 23:59:42 <vertis> evan: it's only the start
 23:59:51 <ssd7> vertis: So, are we aware of any problems with the LB atm?  Think we are good to continue with getting the app server sorted.
 23:59:53 <vertis> it gets better
 Saturday, February 2nd, 2013