gistfile1.txt

Current state:
  - README is the getting started point (it should be kept up to date)
  - App server and LB server both work in vagrant (thanks to opscode + others)
  - Orchestration support using capistrano has been added (instructions in readme)
  - Opscode staff will be back as soon as the east coasters start coming in
  - Getting EC2 provisioned and deployed is the current challenge
  - Issues, tasks & questions on the github issues page
  - Everything comes via a pull request
  - Evan has given permission for the DB to be broken off into it's own host
  - No rearchitecting till it reaches prod (we all want to)

gistfile2.txt

coderanger: and you might need to clear existing app code if you were experimenting with capistrano stuffs
[7:56pm] enviable left the chat room. (Quit: enviable)
[7:56pm] vertis: I wasn't
[7:57pm] vertis: in any case the capistrano stuff wasn't deployment
[7:57pm] vertis: just orchestration
[7:57pm] coderanger: Gotcha 
[7:57pm] vertis: it's confused a few people today
[7:57pm] timh_ left the chat room. (Read error: Connection reset by peer)
[7:58pm] kyd joined the chat room.
[7:58pm] coderanger: I get similar odd looks when I tell people I use Chef and Fabric together 
[7:58pm] vertis: easier at work, we have a custom toolkit
[7:58pm] vertis: so no weird looks
[7:59pm] gazoombo: hah, yeah
[7:59pm] gazoombo: we sometimes use capistrano to do both
[8:00pm] gazoombo: it actually started when we were still using mostly puppet and wanted to spin up puppet standalone instances
[8:00pm] coderanger: curl -H 'Host: rubygems.org' http://localhost:3000
[8:00pm] gazoombo: puppet didn't really have anything like knife at the time
[8:00pm] havenn joined the chat room.
[8:00pm] coderanger: Shows what looks like real content
[8:00pm] coderanger:
[8:01pm] vertis: coderanger: just waiting for chef
[8:02pm] coderanger: <h2>Server error.</h2>
[8:02pm] coderanger: derp
[8:02pm] coderanger: well its something at least
[8:02pm] • coderanger looks for error logs
[8:03pm] coderanger: PG::Error: ERROR:  relation "rubygems" does not exist
[8:03pm] vertis: hmmm
[8:03pm] vertis: rake db:migrate issue?
[8:03pm] coderanger: Yeah, I didn't activate migrations, derp
[8:03pm] coderanger: testing and pull req soon
[8:04pm] enviable joined the chat room.
[8:04pm] ssd7: gazoombo: Incase you want it for reference later. A rough cut of what I was thinking: https://github.com/rubygems/rubygems-aws/pull/41/files
[8:05pm] Luca joined the chat room.
[8:05pm] vertis: ssd7: is that working and good to go?
[8:05pm] Luca left the chat room.
[8:06pm] ryanf left the chat room. (Quit: leaving)
[8:06pm] ssd7: vertis: That works now, don't know if it makes what gazoombo is working on more difficult though.
[8:06pm] vertis: ssd7: I'll leave it alone then
[8:10pm] gazoombo: ssd7: interesting.
[8:10pm] gazoombo: I'm just tracking down some path issues I forgot to change
[8:11pm] gazoombo: I could rebase on that without too much work
[8:11pm] vertis: I manually ran the rake db:migrate
[8:11pm] vertis: it tries to redirect me to
[8:11pm] vertis: https://rubygems.phlippers.net/
[8:11pm] ssd7: vertis: Given the node data, that is likely expected?
[8:12pm] vertis: maybe
[8:12pm] ssd7:   // Hostname
[8:12pm] ssd7:   "set_fqdn": "rubygems.phlippers.net",
[8:12pm] coderanger: Yeah, it was trying to redirect me to rubygems.org originally
[8:12pm] coderanger: (when talking to it as localhost)
[8:14pm] gibsop1 joined the chat room.
[8:15pm] vertis: even after I changed that, it's still redirecting https:// and not the same port, I'm looking into it a bit more
[8:16pm] coderanger: vertis: https://github.com/rubygems/rubygems-aws/pull/42 has the migrate flag
[8:16pm] coderanger: and a minor dependency addition just to be formal
[8:16pm] vertis: merged
[8:17pm] coderanger: I'm not seeing redirect issues
[8:17pm] coderanger: running curl -H 'Host: rubygems.org' http://localhost:3000
[8:17pm] coderanger: <strong>Welcome to your community RubyGem host.</strong>
[8:18pm] vertis: you're running that on the box?
[8:18pm] coderanger: Yes
[8:18pm] vertis: i was working with port forwarded
[8:18pm] coderanger: Ahh
[8:18pm] vertis: and 80 -> 8888
[8:18pm] vertis: so nginx
[8:18pm] coderanger: And are you talking to unicorn or directly to nginx?
[8:18pm] coderanger: jinx 
[8:18pm] vertis: can change that
[8:19pm] coderanger: Maybe something funky in the nginx conf, will start reading that
[8:19pm] vertis: this box shouldn't have nginx anyway
[8:19pm] vertis: nginx goes on the load balancers
[8:21pm] coderanger: Yeah, I would expect different roles
[8:22pm] coderanger: Though this is where solo will start showing its rougher edges since we can't build up the node lists on the fly as much 
[8:23pm] vertis: i was trying to keep this as MVP as possible
[8:23pm] coderanger: +1
[8:23pm] coderanger: Do you think a single VM can handle your traffic for now or will you want to bring this up in a LB situation to start with?
[8:23pm] coderanger: or is that a version 2.0 feature? 
[8:24pm] coderanger: I don't know how your traffic compares to PyPI
[8:24pm] ssd7: Not an nginx config expert, but it looks like we redirect 80->443 which is where the hostname change probably came in too
[8:25pm] davidfstr joined the chat room.
[8:25pm] coderanger: Ahh yeah
[8:25pm] coderanger: Using $server_name
[8:25pm] coderanger: should be $host I think
[8:25pm] vertis: coderanger: I'm told LB
[8:25pm] vertis: before evan went to bed
[8:26pm] vertis: there has been a bunch of work done on it already
[8:26pm] coderanger: Roger roger, will start dividing things up then
[8:26pm] coderanger: Yeah, its close, just all currently glomed in a single "rubygems" role
[8:26pm] vertis: ssd7: yeah just setting up non nat connection in vagrant and seeing if it plays well
[8:26pm] vertis: I'm a little unsure how it serves the assets
[8:27pm] vertis: since unicorn doesn't
[8:27pm] vertis: re: nginx on the box
[8:27pm] vertis: coderanger:
[8:28pm] vertis: As in I do wonder if the nginx needs to stay on the box in addition the the LBs in front
[8:28pm] coderanger: In terms of keeping latency down, the "best" way is to check out the app code on the LBs and serving from there
[8:28pm] coderanger: But that can have nasty side effects
[8:28pm] coderanger: Also is there a reason to not just use Amazon's hosted ELB?
[8:28pm] vertis: coderanger: I just don't know
[8:28pm] vertis: coderanger: I asked that
[8:29pm] vertis: mostly not wanting architecture changes right now
[8:29pm] ssd7: First things first in terms of splitting this up, we need to teach nginx about upstreams that aren't 127.0.0.1.  Since we are using solo, we'll need a static list of what the upstreams wil be.
[8:29pm] coderanger: (good reasons include "it breaks", "lol lock in", etc)
[8:29pm] vertis: nginx it was how it worked
[8:29pm] vertis: so should stay how it works for now
[8:30pm] coderanger: Works for me 
[8:30pm] surfichris joined the chat room.
[8:31pm] vertis: let me just see if I can manage to look at the app through nginx
[8:31pm] rohrer joined the chat room.
[8:32pm] coderanger: The way it is building the upstream doesn't seem right
[8:34pm] aniero left the chat room. (Ping timeout: 245 seconds)
[8:35pm] aniero joined the chat room.
[8:36pm] gazoombo: bleh. I was staring at my mistake for 20 minutes
[8:36pm] fsvehla joined the chat room.
[8:37pm] gazoombo: it's late
[8:37pm] vertis: you should have handed over an hour ago
[8:37pm] vertis: no offense
[8:38pm] coderanger: vertis: Those fixes might help with nginx, going to reconfigure my vagrant so I can test it better
[8:38pm] vertis: coderanger: unicorn listens on just one port so yeah
[8:43pm] ssd7: coderanger: something like this perhaps: https://github.com/stevendanna/rubygems-aws/compare/nginx-fixup
[8:44pm] coderanger: ssd7: Yeah, but right now the balancer role isn't really doing anything
[8:44pm] coderanger: and I seem to have broken my vagrant by trying to turn on bridged networking
[8:45pm] vertis: coderanger: yeah mere too
[8:45pm] vertis: it boots
[8:45pm] vertis: ssh doesn't come up
[8:45pm] vertis: use app.vm.boot_mode = :gui to see problems
[8:45pm] vertis: i'm just destroying my node and recreating now
[8:46pm] coderanger: Yeah, trying a cold boot and then will try hostonly mode
[8:46pm] gazoombo: hostonly works
[8:46pm] gazoombo: I'm using that for the cap stuff
[8:46pm] vertis: it didn't for me
[8:46pm] vertis: but *shrug*
[8:46pm] coderanger: The future: not evenly distributed
[8:46pm] vertis: destroying and recreating has enabled bridged
[8:47pm] vertis: now I wait for 10 mins
[8:51pm] coderanger: vertis: This is why I have Penny Arcade Ep 3 open, keeping me marginally awake
[8:51pm] vertis: not you falling asleep as well
[8:52pm] vertis: don't you people have caffeine over there
[8:52pm] vertis: ;0
[8:52pm] vertis:
[8:52pm] vertis: I merged #43
[8:53pm] coderanger: It has been a lot of long nights lately, last night my girlfriend had to be rescued from SF at midnight (thats a 2.5 hour drive in all)
[8:53pm] vertis: ouch
[8:53pm] vertis: you can sign off if you want
[8:53pm] vertis: I will keep at it
[8:54pm] coderanger: Nah, my typo rate is within acceptable limits still 
[8:54pm] vertis: up to you
[8:54pm] vertis: but I have a rule about burning people out
[8:54pm] vertis: you're probably more valuable after 5-6 hours of sleep
[8:55pm] coderanger: heh, this is what I do in my spare time anyway, still picking up the pieces from the wiki.python.org hack
[8:55pm] vertis: fair enough
[8:55pm] gibsop1 left the chat room. (Quit: leaving)
[8:58pm] ssd7: So, were are we at the moment.  Is there anything that needs to happen on the application side? Or can we completely turn to the load balancer?
[8:58pm] vertis: mostly LB
[8:58pm] coderanger: I think vertis and I are trying to get into a position to test LBy things
[8:58pm] vertis: you could move straight onto that if you want
[8:59pm] vertis: it will need to be setup
[8:59pm] vertis: added to vagrant
[8:59pm] coderanger: curl -kvvv -H 'Host: rubygems.org' https://192.168.1.236
[8:59pm] coderanger: Is working
[8:59pm] vertis: nice
[9:00pm] coderanger: So yeah, ssd7 I think you had things going in the right direction
[9:02pm] ssd7: Yeah, I think the thing to do is get a Vagrant file that gets us an lb + app_server setup.  Then work on making sure all of these roles do the right thing.  Not know this app well, I can't say if more nginx changes are needed in addition to what I posted above.
[9:03pm] havenn left the chat room. (Remote host closed the connection)
[9:03pm] coderanger: ssd7: Yeah, the bit we need after that is to move most of the current rails_nginx to another recipe, probably balancer to match the role
[9:03pm] havenn joined the chat room.
[9:04pm] coderanger: and then rails_nginx should just install a simple vhost to serve static content and proxy to localhost:3000
[9:04pm] sam_ joined the chat room.
[9:04pm] vertis: gazoombo: awesome
[9:04pm] coderanger: I think thats the simplest architecture for the moment anyway
[9:05pm] coderanger: ssd7: You want to attack that or should I pull in your changes and go to town?
[9:05pm] Hypn joined the chat room.
[9:05pm] coderanger: You've probably been up a lot longer than I, would be happy to take over 
[9:05pm] vertis: gazoombo: that host only config is outside the app config
[9:06pm] gazoombo: ?
[9:06pm] vertis: that will break things if we add load balancers
[9:06pm] vertis: config.vm.network :hostonly, "33.0.3.3"
[9:06pm] gazoombo: oh, correct
[9:06pm] gazoombo: there was only one host at the time. not hard to add another
[9:07pm] gazoombo: you'll just need to update the stage in config/deploy/*.rb too
[9:08pm] havenn left the chat room. (Ping timeout: 255 seconds)
[9:08pm] davidfstr left the chat room.
[9:08pm] vertis: gazoombo: yeah I see what you mean
[9:08pm] gazoombo: do you want me to move that into the app block now?
[9:09pm] vertis: not if you have to head to bed, I can do it
[9:09pm] ssd7: coderanger: Want to pull my changes in and keep going, I was going to work on a Vagrant file will all the necessary bits for a few minutes.
[9:09pm] vertis: this doesn't support multiple servers well
[9:09pm] coderanger: Roger roger
[9:09pm] gazoombo:
[9:09pm] gazoombo: I should probably sleep a little bit before returning to work.
[9:11pm] vertis: gazoombo: good night
[9:12pm] gazoombo: FWIW, I think you could use capistrano roles to work with multiple servers in the same stage
[9:12pm] gazoombo: anyhow
[9:12pm] gazoombo: goodnight
[9:12pm] vertis: night
[9:13pm] vertis: coderanger, ssd7: I'm going to merge the cap stuff
[9:13pm] coderanger: Roger, does that affect anything in particular for testing?
[9:13pm] vertis: actually scratch that
[9:13pm] vertis: I'm going to fork his repo
[9:13pm] vertis: and fix some things first
[9:14pm] vertis: and make sure spinning up the balancer(s) is supported as well
[9:14pm] ssd7: vertis: OK, I'll hold on any more Vagrantfile changes for a bit then.  Don't really want to deal with merge conficts at this hour.
[9:15pm] vertis: what were you mixing up in the Vagrantfile anyway?
[9:16pm] ssd7: vertis: Getting a loadbalancer/app_server split config going
[9:16pm] vertis: right
[9:16pm] vertis: yeah
[9:16pm] vertis: give me 15 mins
[9:17pm] ssd7: np, I'm going to see how these roles are put together a bit more. Available to review or test any code coderanger has eventually
[9:22pm] sam_ left the chat room. (Ping timeout: 244 seconds)
[9:24pm] vertis: do the ec2 machines have chef on them already
[9:24pm] vertis: gazoombo has bootstrap stuff for chef
[9:24pm] sam_ joined the chat room.
[9:24pm] vertis: but I don't think we need it?
[9:24pm] vertis: maybe
[9:27pm] ssd7: vertis: Depends on the ami.  If you use the ones from canonical, they don't have Chef so we'll need a short bootstrap script
[9:28pm] vertis: which ones are the ones in the readme
[9:28pm] ssd7: checking
[9:28pm] kgrz left the chat room. (Ping timeout: 256 seconds)
[9:28pm] ssd7: spot check of two says they are from Canonical
[9:28pm] vertis: okay
[9:29pm] vertis: so the bootstrap stuff is good
[9:29pm] vertis: cool
[9:29pm] vertis: i'm just getting an error where its trying to remove some keys
[9:29pm] vertis: then I can merge
[9:30pm] vertis: something changed with sudo
[9:30pm] stevenhaddox joined the chat room.
[9:30pm] vertis: it's prompting for password from vagrant
[9:30pm] stevenhaddox: Ah-hah. I knew there had to be people still awake working on something.
[9:31pm] vertis: stevenhaddox: hello
[9:31pm] vertis: github.com/rubygems/rubygems-aws
[9:31pm] stevenhaddox: vertis: hello to you too.
[9:31pm] vertis: we're working through the last of the app server
[9:31pm] vertis: and then moving onto getting the Nginx Load balancers working
[9:32pm] stevenhaddox: sounds like you guys have moved fast
[9:32pm] vertis: ssd7 and code ranger are the only other two still standing
[9:32pm] vertis: but there has been a large contingent in here
[9:32pm] vertis: they got a long way before crashing
[9:33pm] stevenhaddox: Well, I'm not sure how much help I can be, but I'll do anything I can for the next few hours at least.
[9:34pm] coderanger: vertis: Just testing the new LB setup locally
[9:34pm] stevenhaddox: I'm a chef newb so unfortunately it looks like I may not be able to pitch in much on those issues...
[9:34pm] vertis: coderanger: sweet
[9:34pm] vertis: stevenhaddox: the readme is pretty good
[9:34pm] vertis: but it's just about to change
[9:35pm] vertis: so hold off for another 5-10 mins
[9:35pm] stevenhaddox: change is good
[9:35pm] stevenhaddox: no sweat there
[9:35pm] vertis: integrating capistrano for orchestration
[9:35pm] vertis: by gazoombo
[9:39pm] coderanger: Hmm, close, static content is working, but the rails app is stuck ina  direct loop
[9:40pm] ssd7: coderanger: s/direct/redirect/ ?
[9:41pm] coderanger: yeah, that 
[9:41pm] vertis: just creating the pull request
[9:41pm] vertis: ready momentarily
[9:41pm] dontbecold_ joined the chat room.
[9:42pm] sam_ left the chat room. (Quit: leaving)
[9:42pm] vertis: can someone check through it https://github.com/rubygems/rubygems-aws/pull/45
[9:42pm] ssd7: vertis: reading
[9:43pm] vertis: hmmm
[9:43pm] vertis: there is a commit I wasn't expecting in there
[9:43pm] vertis: it doesn't really matter
[9:43pm] vertis: its fairly well negated by the other changes
[9:43pm] enviable left the chat room. (Quit: enviable)
[9:46pm] ssd7: vertis: So, why don't we need to pass --path to librarian anymore?
[9:46pm] vertis: nope
[9:46pm] stevenhaddox: ssd7: it looks like there's a .librarian file that sets that config if I skimmed it right?
[9:47pm] vertis: it creates that dir the first time you run it with a path
[9:47pm] ssd7: ahh, there it is, missed that one.  Thanks stevenhaddox
[9:47pm] vertis: sort of like how bundle behaves putting down a .bundle directory
[9:48pm] stevenhaddox: ssd7: the very least I can do 
[9:48pm] ssd7: +1 here. Don't have a tone of cap experience, but it looks sane.
[9:48pm] vertis: it's a light use of cap
[9:48pm] vertis: just basically the ssh capabilities
[9:49pm] vertis: we'll need to refactor a little to add the balancer
[9:49pm] vertis: but it should be fine
[9:49pm] vertis: okay merging
[9:49pm] vertis: okay
[9:49pm] vertis: done
[9:49pm] vertis: pull
[9:50pm] vertis: you will need to destroy and recreate your vagrant machine to get the host only network
[9:50pm] vertis: (at least I did)
[9:50pm] stevenhaddox: It looked good to me from skimming it. (too late now either way)
[9:50pm] vertis: lol
[9:51pm] vertis: ssd7: you can do your changes now
[9:51pm] coderanger: http://i.coderanger.net/HoRfF.png
[9:52pm] coderanger:
[9:52pm] vertis: how did you get that
[9:52pm] vertis: my version has the disabled readonly thing on it
[9:52pm] coderanger: updated my hosts file
[9:52pm] coderanger: Also note the 0 downloads
[9:52pm] vertis:
[9:52pm] vertis: as did I
[9:53pm] coderanger: Okay, LB config pull req is up
[9:53pm] coderanger: https://github.com/rubygems/rubygems-aws/pull/46
[9:53pm] coderanger: ssd7: ^^
[9:53pm] vertis: actually never mind
[9:53pm] vertis: google chrome caching dns
[9:53pm] coderanger: Yeah, I use Fx for anything with a hosts override
[9:53pm] rohrer left the chat room. (Read error: Connection reset by peer)
[9:53pm] vertis: yeah mine is working too
[9:54pm] coderanger: So with that pull req the default vagrant VM just happens to have both rubygems and balancer roles applied, and is using ["127.0.0.1"] as the list of app servers
[9:54pm] coderanger: But you can split the roles out
[9:55pm] coderanger: just update the list in roles/balancer.rb
[9:55pm] vertis: i still have to work out how the static assets are handled
[9:55pm] vertis: unicorn doesn't serve them
[9:55pm] coderanger: Thats in there 
[9:55pm] vertis: oh
[9:55pm] coderanger: yep, nginx does
[9:56pm] ssd7: yeah +1, looks good
[9:56pm] vertis: yeah but if it's on a separate box how?
[9:56pm] coderanger: there are two trips through nginx
[9:56pm] coderanger: once on the LB, once on the app host
[9:56pm] vertis: cool
[9:56pm] coderanger: minor bump to latency on serving, but simple and easy to get up
[9:57pm] coderanger: should probably migrate to the asset_sync gem and serving from cloudfront or fastly later anyway 
[9:57pm] vertis: it needs to be rebased
[9:57pm] coderanger: Roger
[9:57pm] coderanger: Do you prefer rebase or merge?
[10:00pm] vertis: i rebase
[10:00pm] vertis: *shrug*
[10:01pm] coderanger: Done
[10:01pm] vertis: wow ajax
[10:01pm] • coderanger is usually an anti-rebase zealot 
[10:01pm] vertis: saw it go green in background
[10:01pm] • vertis likes rebasing
[10:01pm] vertis: clean history
[10:01pm] • stevenhaddox ducks
[10:01pm] jstr left the chat room. (Quit: sleep)
[10:02pm] vertis: merged
[10:02pm] vertis: right LB onto separate hosts in Vagrant
[10:03pm] vertis: then we try to move into getting it working in EC2
[10:03pm] coderanger: Okay, anything I can help with in particular?
[10:03pm] coderanger: Going to kill and rebuild my vagrant for the new world order
[10:03pm] vertis: yeah me to in a minute
[10:04pm] vertis: stevenhaddox: run through the read me and see if it works for you
[10:04pm] stevenhaddox: sure thing. Waiting on VirtualBox to install on my remote linux system I can access from this location.
[10:04pm] stevenhaddox: Once it's done I'll install vagrant and walk through it.
[10:06pm] ssd7: So, do we need to also work on seperating out the database now.  It is still living on the app server correct?
[10:06pm] vertis: ssd7: for now
[10:06pm] vertis: evan seems fine either way
[10:07pm] vertis: current prod has it on the app server
[10:07pm] coderanger: vertis: Bigger issue is the DB is hardcoded to look to localhost
[10:07pm] coderanger: so if there are more than one app servers, bad news bears
[10:07pm] ssd7: kk, I guess I figured that since we were load balancing, "Prod" would have >1 apps servers
[10:07pm] coderanger: Yeah
[10:08pm] coderanger: But can just be 1 for now?
[10:08pm] vertis: coderanger: current state is 1 app server though
[10:08pm] coderanger: Works for me 
[10:08pm] vertis: i'm just relaying
[10:08pm] vertis: i would have multiple app servers
[10:08pm] coderanger: (give or take that current hosting probably is much faster than EC2)
[10:08pm] vertis: behind a load balancer and multiple dbs
[10:08pm] vertis: '
[10:08pm] coderanger: but lets start at 1 for simplicity
[10:08pm] vertis: but that's just me
[10:09pm] coderanger: vertis: Baby steps 
[10:09pm] vertis: yeah
[10:09pm] • coderanger did this migration for PyPI 9 months ago
[10:09pm] coderanger: have to giggle at that
[10:10pm] coderanger: hmm, "cap bootstrap" wants a password for user vagrant
[10:10pm] vertis: skip that anyway
[10:11pm] vertis: image has chef on it already
[10:11pm] vertis: but vagrant password is vagrant
[10:11pm] vertis: it doesn't have passwordless sudo??
[10:11pm] melgray left the chat room. (Ping timeout: 240 seconds)
[10:11pm] stevenhaddox: meh... I use rbenv to avoid gemsets since Bundler has done so well lately. Bear with me as I go get the gemset plugin for rbenv so I can bundle 
[10:11pm] coderanger: This was for SSH I think
[10:11pm] vertis: stevenhaddox: what?
[10:12pm] coderanger: vertis: "cap chef" is running now
[10:12pm] coderanger: (without asking for a password, for whatever reason)
[10:12pm] vertis: stevenhaddox: i don't understand what your current problem is
[10:12pm] vertis: I use rbenv as well
[10:12pm] vertis: without gemsets
[10:12pm] stevenhaddox: hmmm I must be doing something wrong then
[10:13pm] vertis: 1.9.3?
[10:13pm] vertis: to be fair I haven't bundle installed since the last merge
[10:13pm] stevenhaddox: bundle install pukes w: $ rbenv: version `1.9.3@rubygems-infrastructure' is not installed
[10:13pm] coderanger: I have, worked fine
[10:14pm] coderanger: https://github.com/rubygems/rubygems-aws/blob/master/.ruby-version
[10:14pm] coderanger: stevenhaddox: ^^
[10:14pm] vertis: someone else was complaining about that
[10:14pm] stevenhaddox: I've got rbenv setup to respect local project versions... guessing @rubygems-infrastructure no matchy mine
[10:14pm] vertis: right
[10:14pm] coderanger: That file should have the gemset removed
[10:15pm] stevenhaddox: k
[10:15pm] vertis: rm that file
[10:15pm] stevenhaddox: I'll grab that if you guys want?
[10:15pm] vertis: yep
[10:15pm] stevenhaddox: we don't want to suggest 1.9.3?
[10:15pm] stevenhaddox: just yank the gemset or yank the whole file?
[10:15pm] coderanger: stevenhaddox: For the workstation setup it is pretty moot
[10:15pm] stevenhaddox: coderanger: good point
[10:15pm] coderanger: Just need vagrant and cap running
[10:15pm] stevenhaddox: rvm / rbenv can both handle that easily I'm sure
[10:15pm] stevenhaddox:
[10:16pm] vertis: has to be 1.9+ or will break anyway
[10:16pm] stevenhaddox: bundle is working now
[10:17pm] fsvehla left the chat room.
[10:17pm] coderanger: the overture to Phantom is extra epic at 3AM
[10:17pm] vertis: ?
[10:17pm] stevenhaddox: you guys are using Nginx, oui?
[10:18pm] coderanger: vertis: Showtunes ftw 
[10:18pm] coderanger: stevenhaddox: c
[10:18pm] rohrer joined the chat room.
[10:18pm] vertis: coderanger: linke
[10:18pm] vertis: link*
[10:18pm] vertis: stevenhaddox: yeah
[10:18pm] vertis: nginx in two layers
[10:18pm] stevenhaddox: noticed the passenger_apache2 cookbook. Is that needed? /me isn't familiar with what dependent cookbooks are needed for Nginx / vagrant
[10:18pm] vertis: LB + on app server
[10:18pm] coderanger: vertis: https://www.youtube.com/watch?v=NhttZz9lmpk
[10:18pm] coderanger: vertis: It is a classic in every sense of the word
[10:19pm] stevenhaddox: apache2 cookbook as well apparently installs
[10:19pm] stevenhaddox: just FYI
[10:19pm] vertis: was copied from from existing chef codebase
[10:19pm] vertis: feel free to clean that up and then test
[10:19pm] coderanger: stevenhaddox: application_ruby pulls it in
[10:19pm] coderanger: stevenhaddox: It isn't in use though
[10:19pm] vertis: ah nm
[10:19pm] coderanger: Just a dependency because application_ruby supports it as a deploy strategy
[10:20pm] stevenhaddox: k
[10:20pm] stevenhaddox: like I said, chef newb 
[10:21pm] stevenhaddox: is there a test suite per se with this or is the fact that vagrant runs our test suite?
[10:21pm] coderanger: stevenhaddox: Manual inspection so far
[10:21pm] ssd7: I'm using this https://github.com/stevendanna/rubygems-aws/compare/multi-node-vagrant seems to be working, configs look right, now verifying if the app is actually working.
[10:22pm] stevenhaddox: k. Made a pull request for the .ruby-version removal
[10:22pm] stevenhaddox: 61% downloaded on the vagrant box
[10:23pm] sampointer joined the chat room.
[10:23pm] vertis: merged
[10:23pm] vertis: coderanger: you're right phantom works well
[10:23pm] coderanger: vertis: Okay, so got a cap-powered VM up, but nginx is now dumping me to maintenance mode
[10:24pm] vertis: yours seems to be quicker than mine
[10:24pm] vertis: mine is still rebuilding again
[10:24pm] sampointer: Hey folks. I work for EA (here in a personal capacity). We have a large Chef code-base running many EC2 instances. I can't offer any practical help at the minute, but I'm happy to answer any questions I can. I'm in the UK.
[10:26pm] stevenhaddox: So as a newb to the README. Once I ran $ cap bootstrap; $ cap chef;  How do I verify it's running, etc? Do I need to figure that out from the Vagrantfile manually or do we want to mention where to look to save people poking around when getting started? I'm up to either (still waiting on the VM to boot so I haven't gotten this far yet)
[10:28pm] vertis: should be on 33.33.33.10 but put rubygems.org in your hosts file pointing at that IP
[10:28pm] vertis: sampointer: https://github.com/rubygems/rubygems-aws
[10:29pm] vertis: there is a readme file if you want to do some testinging
[10:29pm] vertis: and the github issues outline what has been worked on
[10:29pm] vertis: (i need to update it a bit)
[10:29pm] vertis: testing*
[10:30pm] coderanger: And now for some reason I'm not seeing the maint mode
[10:30pm] coderanger: Oh, I bet it was because unicorn was still booting
[10:30pm] coderanger: derp
[10:30pm] vertis: ssd7 waiting on you for the LB
[10:32pm] ssd7: coderanger: vertis: Want to give the above posted branch a look/spin.  It bring up two nodes, for some reason the app server isn't getting actually getting the app deployed though.
[10:33pm] vertis: sure
[10:34pm] coderanger: ssd7: What expanded cookbook list do you see on the app server?
[10:35pm] stevenhaddox: man this is taking forever to boot for me...
[10:35pm] vertis: if it sits around for too long it's not working
[10:36pm] vertis: i.e. vagrant up
[10:36pm] ssd7: coderanger: hrmm, they both seem to be getting the balancer role
[10:36pm] vertis: stevenhaddox: try adding the boot_mode = :gui or whatever to the Vagrantfile
[10:36pm] coderanger: ssd7: My cap knowledge is a bit low 
[10:37pm] vertis: yes that isn't how cap works
[10:37pm] vertis: hang on
[10:37pm] vertis: your config/deploy/vagrant.rb is wrong
[10:37pm] ssd7: oh bah
[10:37pm] ssd7: yeah
[10:37pm] stevenhaddox: I'm ssh'd in over a remote connection so :gui == bad for me I'm guessing
[10:37pm] vertis: lol probably
[10:38pm] stevenhaddox: I'll try and help with Capistrano stuff if I can get the VM to boot for me. I've done more than my fair share of it.
[10:39pm] vertis: just importing the lb box
[10:39pm] vertis: then cap vagrant-lb chef
[10:39pm] vertis: will see how it goes
[10:40pm] vertis: nope
[10:42pm] ssd7: yeah, ok, correctly deploying now.
[10:42pm] Holek joined the chat room.
[10:43pm] vertis: my balancer is running chef
[10:43pm] vertis: will let you know shortly
[10:44pm] vertis: ssd7, coderanger if the two of you need to sleep I can hold the fort
[10:44pm] vertis: you've put us in a good place
[10:44pm] coderanger: I'll stick around until 4, hopefully enough to get EC2 kicked off
[10:44pm] vertis: hi Holek
[10:45pm] vertis: sure
[10:45pm] coderanger: vertis: Another unanswered question right now is SSL key distribution
[10:45pm] coderanger: vertis: definitely not something to check in to git
[10:46pm] stevenhaddox: Failed to connect to VM via SSH. .. verify ... booted .. by looking at the VirtualBox GUI. #fml 
[10:47pm] richo joined the chat room.
[10:49pm] vertis: coderanger: yeah I don't know the answer to that
[10:50pm] vertis: they were discussing something around that before, and todo with gpg trust something something
[10:50pm] vertis: samkottler was leading
[10:50pm] vertis: he can pick that up when he comes back
[10:51pm] coderanger: Okay
[10:51pm] coderanger: I mean the simple option is just punt and copy them by hand onto the LB boxen
[10:51pm] vertis: *shrug*
[10:51pm] coderanger: For Python we store them in the Chef server in a data bag with locked down perms, if that helps 
[10:52pm] coderanger: (also where DB passwords and such live)
[10:52pm] vertis: that doesn't help at all and you know it 
[10:52pm] jensnorr_ left the chat room. (Quit: jensnorr_)
[10:52pm] coderanger:
[10:54pm] vertis: mind you this is MVP
[10:54pm] vertis: there are a couple of things that will make it worth switching to chef server afterwards
[10:55pm] ssd7: ok, these roles need a bit of cleanup. `cap vagrant-app chef; cap vagrant-lb chef` curently converging here.  Someone who will be living with this workflow on a more regular basis should probably eventually be the one to decide how the cap stuff is laid out
[10:55pm] vertis: yeah
[10:56pm] vertis: we need to switch it to roles not stages
[10:56pm] vertis: because the roles will run at the same time
[10:57pm] vertis: i'm refactoring
[10:59pm] ssd7: cool, yeah, I made some stages that happen to work, but roles look like the way to go.
[11:00pm] stevenhaddox: hmm... so vagrant vm appears to be up, but vagrant ssh pipes back: >> ssh_exchange_identification: Connection closed by remote host
[11:00pm] benchMark joined the chat room.
[11:02pm] dwradcliffe joined the chat room.
[11:04pm] ssd7: Alrighty, I'm tagging out in 27 minutes.  If we can verify the split setup, I think we'll be in a good place to moving onto EC2
[11:04pm] rohrer left the chat room. (Read error: Connection reset by peer)
[11:04pm] coderanger: Yeah, we can probably start booting EC2 machines at least
[11:08pm] vertis: sorry was focused
[11:09pm] ssd7: coderanger: have you seen this  ** [out :: 33.33.33.10] /etc/monit/conf.d/postgresql.monitrc:11: Error: cannot compute a checksum for file /usr/lib/postgresql/9.2/bin/postgres 'postgresql'
[11:10pm] vertis: it's all working for me
[11:10pm] vertis: i've made the changes to make it use roles
[11:10pm] vertis: just checking the logs on the LB
[11:12pm] kule joined the chat room.
[11:12pm] vertis: okay it's passing things through the lb
[11:14pm] ssd7: Cool, so.. "TO THE CLOUD!"?
[11:14pm] vertis: yeah just creating the pull request
[11:15pm] vertis: https://github.com/rubygems/rubygems-aws/pull/48
[11:15pm] vertis: check and +1 please
[11:15pm] ben_h joined the chat room.
[11:16pm] coderanger: Looks good
[11:16pm] vertis: merge?
[11:16pm] ssd7: So, I think with just that config the "app" server will still be running the fullstack role
[11:16pm] vertis: i saw that
[11:17pm] vertis: i didn't add that though
[11:17pm] coderanger: Ahh yeah, need to swap the run list in the old JSON
[11:17pm] vertis: shall I merge and then you can fix that?
[11:18pm] vertis: or do you want me to?
[11:18pm] vertis: nm I've done it
[11:18pm] ssd7:
[11:19pm] vertis: pull request updated
[11:20pm] ssd7: sorry, was looking at that rubygem role.  I think that either we need to add base to that role or add role['base'] to the run list as well
[11:20pm] mr_ndrsn joined the chat room.
[11:20pm] dwradcliffe: just  waking up - where do we need help?
[11:20pm] kek joined the chat room.
[11:21pm] coderanger: Yeah, and this means postgres server won't get installed
[11:21pm] coderanger:
[11:21pm] coderanger: Not sure whats going on with redis or memcache
[11:21pm] vertis: dwradcliffe: I can give you an update in a minute
[11:21pm] mr_ndrsn: Yes, I'm back too, for a bit.  Updating my repo and re-upping the vagrant box.
[11:21pm] coderanger: but for now recipe[postgresql::server] should be added to role[rubygems]
[11:22pm] coderanger: and role base
[11:22pm] vertis: I'll adjust the commit
[11:22pm] vertis: one moment
[11:22pm] ssd7: coderanger: vertis Looks like for sanity we might just want to make that rubygems role fullstack - balancer?
[11:22pm] vertis: yeah
[11:22pm] vertis: i'll create a new role
[11:22pm] vertis: almost_fullstack 
[11:24pm] coderanger: Just call it "spof" 
[11:24pm] vertis: pushing
[11:24pm] vertis: dwradcliffe: we almost have the LB and the rubygems app server running on vagrant
[11:25pm] vertis: using the capistrano stuff for orchestration
[11:25pm] mr_ndrsn: What's the current arch look like?  I'm still only seeing a single box in the vagrant config, but it sounds like you guys are moving to multi?
[11:25pm] vertis: just doing final tweaks now
[11:25pm] vertis: mr_ndrsn: it's in pull request
[11:25pm] vertis: that we're just about to merge
[11:25pm] vertis: check request again https://github.com/rubygems/rubygems-aws/pull/48
[11:25pm] mr_ndrsn: Got it.
[11:26pm] coderanger: vertis: Looks good to me
[11:26pm] ssd7: vertis: +1 Think I've found all I will find just by reading.
[11:26pm] vertis: okay merging
[11:26pm] vertis: done
[11:26pm] vertis: dwradcliffe: did that answer your question?
[11:27pm] dwradcliffe: yep
[11:29pm] vertis: so moving this into EC2 is the next priority
[11:29pm] kek_ joined the chat room.
[11:30pm] coderanger: So we need two boxes booted and some new JSON defs for prod (or closer to it) nodes?
[11:30pm] vertis: yep
[11:30pm] mr_ndrsn: Completely tangential, but the redis is localhost only, right?  If not, I was going to ask about how you guys are handling security on it, as I've never gotten it to work.  
[11:31pm] ssd7: mr_ndrsn: localhost only for now I believe, we should verify.
[11:31pm] vertis: not sure about bind ip
[11:31pm] coderanger: mr_ndrsn: I don't know if redis is being used currently, I'm not writing out a config for it
[11:31pm] coderanger: Unless the app just assumes it is on localhost
[11:32pm] kek left the chat room. (Ping timeout: 246 seconds)
[11:32pm] kek_ is now known as kek.
[11:32pm] ssd7: default in the cookbook is to bind to 127.0.0.1
[11:32pm] coderanger: ditto on memcache, which is also being installed on app servers
[11:32pm] dwradcliffe: looks like the app currently uses ENV['REDISTOGO_URL']
[11:33pm] vertis: lol
[11:33pm] kule left the chat room.
[11:33pm] coderanger: I've got to throw in the towel I think, time to pass out
[11:34pm] stevenhaddox: sweet!
[11:34pm] stevenhaddox: I got vagrant VM booting up now
[11:34pm] AK joined the chat room.
[11:34pm] stevenhaddox: vagrant ssh says I need to pass a specific VM name
[11:35pm] mr_ndrsn: Yeah, vagrant ssh app or vagrant ssh lb
[11:35pm] coderanger: vertis: Other opscode folks will probably be waking up shortly in UTC-5, they know how to find me if something in the application cookbook stuffs goes boom 
[11:35pm] vertis: i have a few hours more
[11:36pm] vertis: only 11:30pm here
[11:36pm] mr_ndrsn: Mine booted, but didn't run any new chef config, so I destroyed them, re-ran, and got the same.  Am I missing something, or are the vagrant boxes not tied into the new chef yet?
[11:36pm] vertis: i'm the connector
[11:36pm] vertis: keep things on track
[11:36pm] vertis: mr_ndrsn: readme again
[11:36pm] stevenhaddox: https://gist.github.com/1ca40f56c68d1515ec83 <-- vertis / coderanger neither of you are getting this correct? I've already fetched & pulled from master on upstream maybe 3 minutes ago
[11:37pm] mr_ndrsn: Sonofa!  *facepalm*
[11:37pm] vertis: now uses capistrano for the deploy
[11:37pm] benchMark left the chat room. (Quit: Computer has gone to sleep.)
[11:37pm] vertis: stevenhaddox: ah yes I removed that
[11:37pm] vertis: couldn't work out where it was used
[11:37pm] vertis: skip the cap bootstrap on vagrant
[11:37pm] vertis: it already has chef
[11:37pm] stevenhaddox: so what commands are we running? need me to update the README?
[11:37pm] ssd7: verified that the config has redis bidnign to local host.  Someone might want to netstat a running app server at some point and verify memcached, postegres, redis, etc
[11:38pm] stevenhaddox: that is pretty
[11:38pm] vertis: stevenhaddox: I had to remove that server_ip because we have two servers
[11:38pm] stevenhaddox: poop.... failed connection still
[11:38pm] vertis: and it was defined globally
[11:38pm] vertis: see if you can read the bootstrap and understand
[11:38pm] ssd7: vertis: I'm out now as well I think.  Will leave a note for east coase opscoders to potentially pop in the channel.
[11:38pm] vertis: and replace it in a better way maybe?
[11:39pm] vertis: ssd7: thanks
[11:39pm] vertis: coderanger: thanks
[11:39pm] vertis: coderanger, ssd7: have a good sleep
[11:40pm] ihoka joined the chat room.
[11:40pm] stevenhaddox: ssd7: Thanks for all you've done.
[11:42pm] ssd7: Happy to help  night.
[11:43pm] vertis: stevenhaddox: are you fixing that server_ip thing?
[11:44pm] stevenhaddox: vertis: I'm trying to locate it still.
[11:44pm] stevenhaddox: and make sure I understand it - if you're free feel free to jump on it
[11:45pm] stevenhaddox: I'm still trying to just keep up with what you and ssd7 have cranked out the last few hours
[11:46pm] mr_ndrsn: I looked at it, we might need to break the vagrant task out for the lb to it's own target?  There's no way to determine the role cap is running, so we can't determine it in-script.  Either set an ENV var and run it twice, or we hard-code the vagrant IPs?
[11:48pm] mr_ndrsn: Got to grab a shower and head off to $day_job.  I know I've not been much use, but I'll try to keep on IRC thru the day. 
[11:48pm] stevenhaddox: ah, I see what's going on there now. Unfortunately I still can't get my vagrant boxes to let me connect via ssh so I can't test deploy scripts.
[11:48pm] stevenhaddox: thought I had it
[11:49pm] vertis: yeah okay
[11:49pm] dwradcliffe: I think we can use $CAPISTRANO:HOST$
[11:49pm] vertis: dwradcliffe to the rescue
[11:50pm] dwradcliffe: still trying to dig though this, where is the IP used?
[11:50pm] mr_ndrsn: Can I get your twitter handles before I go?
[11:50pm] vertis: config/deploy.rb:    system("knife bootstrap -d chef-solo -x #{user} --sudo #{server_ip}")
[11:50pm] v2px joined the chat room.
[11:50pm] mr_ndrsn: deploy.rb line 12
[11:50pm] vertis: mr_ndrsn: vertis 
[11:51pm] dwradcliffe: mr_ndrson: dwradcliffe
[11:51pm] dwradcliffe: oops. fingers are still asleep
[11:51pm] teancom joined the chat room.
[11:52pm] dwradcliffe: vertis: that task isn't needed for vagrant, right?
[11:52pm] vertis: correct
[11:52pm] vertis: but we do need it for EC2
[11:53pm] vertis: and the same problem exists
[11:53pm] dwradcliffe: roger that
[11:53pm] bhenerey joined the chat room.
[11:54pm] mr_ndrsn left the chat room. (Quit: mr_ndrsn)
[11:55pm] jroes joined the chat room.
[11:56pm] vertis:     find_servers_for_task(current_task).each do |current_server|
[11:56pm] vertis:       p current_server
[11:56pm] vertis:     end
[11:56pm] teancom: Is there a log for this channel, so that I can catch up w/out repeating dumb questions?
[11:58pm] vertis: teancom: not that I'm aware of
[11:58pm] vertis: i can catch you up
[11:58pm] vertis: i have been here the whole time, minus 30mins
[11:59pm] vertis: dwradcliffe: I'm creating a pull request
[11:59pm] stevenhaddox: vertis: Perhaps make a gist of what you're about to say so we can repaste for others later?
[11:59pm] stevenhaddox: or copy & paste after wards 
[11:59pm] vertis: lol
[11:59pm] samkottler: morning
[11:59pm] vertis: morning sam
[12:00am] vertis: we've had productive evening/night
[12:00am] samkottler: vertis: do you have time to get a quick up on where things stand?
[12:00am] vertis: lb and app are now working in vagrant
[12:00am] vertis: yep
[12:00am] fromonesrc joined the chat room.
[12:00am] samkottler: vertis: excellent!
[12:00am] vertis: one sec
[12:00am] vertis: just want to get this pull request in
[12:00am] samkottler: kk
[12:00am] vertis: then google hangout?
[12:00am] stevenhaddox: really wish I had my laptop so I could actually use vagrant and be of use
[12:00am] teancom: vertis: In short, linux sysad with puppet/aws/vagrant/ruby experience, looking to help. Where to start?
[12:01am] vertis: stevenhaddox: switching to EC2 in a second
[12:01am] vertis: so
[12:01am] vertis: teancom: google hangout in a sec
[12:01am] stevenhaddox: lol, k
[12:01am] teancom: (yes, I realize you're using chef)
[12:01am] samkottler: vertis: I can't do a hangout right now - the internet is getting fixed at my house but wind took it out last night
[12:01am] vertis: samkottler: alright
[12:01am] vertis: just give me another few minutes
[12:01am] vertis: then brain dump
[12:02am] samkottler: vertis: just a 10 second "here's what's up" is fin
[12:02am] samkottler: fine**
[12:02am] vertis: okay
[12:02am] vertis: pull request merged
[12:03am] vertis: so, both the app server (and rubygems.org app) and the lb now work in vagrant
[12:03am] vertis: the read me has updated instructions
[12:03am] vertis: including the orchestration by capistrano
[12:04am] vertis: the database server still sits on the app server (as in current prod)
[12:05am] vertis: we're going to need to deal with ec2 machines, and also working out how to get sensitive stuff like the ssl certs in
[12:05am] vertis: which YOU know more about I think
[12:05am] vertis: teancom: does that make sense?
[12:06am] teancom: Yep
[12:06am] vertis: the current prod layout that we're replicating is 2 nginx LB servers
[12:06am] vertis: 1 app server, that has redis, memcache, db on it as well
[12:06am] vertis: i would love to change all that
[12:06am] samkottler: vertis: yep, that sounds good
[12:06am] vertis: but not right now
[12:06am] vertis: anything I've missed you need to know?
[12:07am] samkottler: vertis: the discrete services, like memcached and redis are easy to move
[12:07am] ben_h left the chat room. (Read error: Connection reset by peer)
[12:07am] vertis: database isn't too hard
[12:07am] teancom: Is nginx doing anything other than LB?
[12:07am] vertis: better if we have a postgresql expert to help with a master/slave or master/master setup
[12:08am] vertis: teancom: there is another nginx on the app server that sits in front of unicorn and serves static assets
[12:08am] vertis: less than happy about that
[12:08am] vertis: but it works for now
[12:08am] teancom: Was wondering if we could replace the two nginx LB servers with elastic LB, now that we're on AWS.
[12:08am] vertis: once this is in prod I'm going to start rearchitecting at a leisurely pace
[12:08am] vertis: teancom: I asked the same question
[12:09am] teancom: great minds 
[12:09am] vertis: answer from evan was, no rearchitecting
[12:09am] vertis: get it working
[12:09am] vertis: THEN we can play with improvements
[12:09am] vertis: which I agree with
[12:09am] teancom: k
[12:10am] dwradcliffe: so we need two nodes for ec2 stage now. RUBYGEMS_EC2_APP RUBYGEMS_EC2_LB?
[12:10am] vertis: yep
[12:10am] vertis: wait
[12:10am] vertis: 3
[12:10am] vertis: to match current prod
[12:10am] vertis: there are 2 LBs
[12:10am] snooc joined the chat room.
[12:10am] dwradcliffe: LB1, LB2?
[12:11am] vertis: something like that
[12:11am] samkottler: balancer01 and balancer02 and app01?
[12:11am] vertis: yep
[12:11am] samkottler: let's just get one balancer up for now
[12:11am] samkottler: are all the roles up to date?
[12:11am] vertis: yeah
[12:11am] vertis: balancer for balancer
[12:11am] dwradcliffe: I'm updating the cap roles and readme
[12:11am] samkottler: vertis: cool
[12:12am] vertis: almost_fullstack for everything else
[12:12am] samkottler: does anyone have thoughts on instance sizes?
[12:12am] vertis: large+
[12:12am] samkottler: we should be fine with a medium for the balancer, though
[12:12am] vertis: the amis are instance store I think
[12:12am] vertis: which may not be ideal
[12:12am] vertis: i.e. database
[12:13am] samkottler: vertis: we can use different ones
[12:13am] vertis: if using ebs we should turn on the ebs optimised flag
[12:13am] vertis: which creates a dedicated network to ebs
[12:13am] vertis: and potentially provisioned ups disk for the db
[12:13am] samkottler: yep, get that second NIC
[12:13am] vertis: iops*
[12:13am] samkottler: vertis: definitely need provisioned IOPS for the postgres data dir
[12:14am] havenn joined the chat room.
[12:14am] • samkottler provisions some machines
[12:14am] samkottler: has anyone worked on postgres yet?
[12:15am] vertis: nup
[12:15am] vertis: the rails app deployment runs db:migrate
[12:15am] vertis: so the app works
[12:16am] samkottler: cool
[12:17am] samkottler: m1.xlarge for the app sound reasonable?
[12:17am] samkottler: I have no idea about the memory or performance profile of the app so I'm basically just spitballing
[12:17am] vertis: really is a shame we don't have 2 app servers
[12:17am] vertis: no way of changing
[12:17am] vertis: after we pick
[12:18am] teancom: That's app + memcache + redis + postgres (at least for now)?
[12:18am] vertis: yeah
[12:18am] samkottler: yeah the reason I'm worried about putting the db server on an app machines is because then the app tier isn't stateless
[12:18am] teancom: I think we're going to need  a bigger boat...
[12:18am] vertis: samkottler: don't tell me
[12:18am] vertis: tell evan
[12:18am] havenn left the chat room. (Ping timeout: 248 seconds)
[12:18am] samkottler: teancom: m2.989xl? 
[12:18am] samkottler: vertis: gonna have to wait a few hours
[12:19am] vertis: we got permission from evan to move the database off
[12:19am] vertis: which might be better
[12:19am] vertis: one of the exceptions to rearchitecting
[12:19am] vertis: sorry should have mentioned that earlier
[12:20am] teancom: If we could even pull pg from it, then yeah, an m1.xlarge should suffice.
[12:20am] samkottler: once evan is up we can talk about putting redis on that db server, too
[12:20am] teancom: (says the guy that has no idea of the performance characteristics of the app, or the average traffic, or ...)
[12:20am] dwradcliffe: so 3 nodes for now: app, db, lb
[12:20am] dwradcliffe: adding 2nd lb later
[12:21am] vertis: we should just make it, 2 app servers / lb behind ELB + 2 db servers with redis / memcache
[12:21am] vertis: or something
[12:21am] samkottler: okay so 1) m1.xl ebs optimized app machine
[12:21am] samkottler: what about the size for the DB?
[12:21am] samkottler: anyone able to comment on the reliability of different AZ's in us-west-2? or is that region not as horrid as us-east-1?
[12:22am] vertis: samkottler:  ReinH was going to get us a dump when he woke up etc
[12:22am] samkottler: vertis: of the database?
[12:22am] vertis: mostly because I knew we wouldn't be up to that until later
[12:22am] vertis: samkottler: yeah
[12:23am] samkottler: vertis: okay gotcha
[12:23am] teancom: Is the pg server directly in the path for any incoming read requests (not writes)? Anyone have an architecture diagram or even flow chart for the site?
[12:23am] vertis: teancom: you're imagining one exists outside of qrush/evan etcs head
[12:24am] vertis: teancom: you'll have to start asking those questions when the pair of them wake up
[12:24am] teancom: Dare to hope, prepare to be disappointed. I have the t-shirt 
[12:24am] samkottler: app server coming up now, starting on lb
[12:24am] vertis: they've been out for quite a while
[12:24am] vertis: should be soon now
[12:24am] teancom: @qrush is up
[12:24am] qrush: teancom: "directly in the path?"
[12:24am] teancom: Does a read-request to the site result in a query on the pg database?
[12:25am] qrush: of course
[12:25am] qrush: almost all of them
[12:25am] samkottler: this is rails 
[12:25am] vertis: lol
[12:25am] vertis: qrush: we're progressing nicely
[12:25am] qrush: we dont have any caching in place really and we have never needed it
[12:25am] teancom: Sorry - at my day job we have pg and riak for a service, where pg isn't touched by end-users. All read requests go to riak. Was wondering if there was anything similar here.
[12:26am] teancom: with the redis/memcache combination
[12:26am] samkottler: dwradcliffe, qrush, vertis: how big do you want the DB server to be?
[12:26am] samkottler: why isn't this instance coming up...
[12:26am] qrush: teancom: yep, pg is everything here. maybe you should get the site set up locally first https://github.com/rubygems/rubygems.org/wiki/Development-Setup
[12:26am] benchMark joined the chat room.
[12:27am] vertis: qrush: our vagrant setup does that
[12:27am] qrush: vertis: thanks for your work. TBH i have a rough idea of how things are setup
[12:27am] teancom: Almost done with that. Stupid TW cable took forever to get the vagrant instance downloaded
[12:27am] qrush: samkottler: i can't speak to sizes of instances - tahts all up to evan
[12:27am] vertis: qrush: might be more questions for evan...yeah
[12:29am] benchMark: Above, re: redis. With password + tight security groups, it should be relatively safe.
[12:29am] samkottler: I'm just gonna make it up and go with a m1.large for the balancer
[12:29am] vertis: qrush: i can give you bit of an update if you want
[12:29am] vertis: google hangout or otherwise
[12:30am] qrush: vertis: let's wait for evan - i'm really not an ops guy and he & rubycentral hold those keys
[12:30am] vertis: qrush: okay
[12:30am] samkottler: qrush: so should I hold off on provisioning these boxes?
[12:31am] benchMark: IMO, you should go ahead. It's going to cost a few bucks, tops. If Evan doesn't like the approach when he wakes up you can turn them off.
[12:31am] vertis: samkottler: lets get them working from a dev perspective and worry about sizing later
[12:31am] samkottler: +1
[12:31am] samkottler: bootstrap now...
[12:31am] vertis: there is still work to be done
[12:31am] vertis: these ec2 boxes may not be the prod ones
[12:32am] samkottler: can anyone take some time to update the readme? it only works for provisioning app boxes right now
[12:32am] dwradcliffe: vertis: sent a PR with cap config
[12:33am] dwradcliffe: samkottler: PR#51
[12:33am] vertis: samkottler: in ec2
[12:33am] qrush: benchMark: samkottler agreed if you can afford it 
[12:33am] qrush: I can provide a db dump
[12:33am] samkottler: qrush: $5 - I got ya covered 
[12:33am] qrush: will that help at this point?
[12:33am] qrush: (sans users table)
[12:33am] samkottler: qrush: it will pretty soon
[12:34am] vertis: teancom: can you add anything you think we might need to consider to the github issues
[12:34am] vertis: DB sizing etc
[12:34am] vertis: i need to mark a few issues as done
[12:34am] teancom: sure, I'll take a look
[12:35am] samkottler: <3 librarian-chef
[12:35am] vertis: don't we all
[12:36am] vertis: but by far my favourite thing in the last 24hours was mitchellh stopping by to help me setup vagrant
[12:36am] snooc left the chat room. (Quit: Colloquy for iPhone - http://colloquy.mobi)
[12:36am] maledale joined the chat room.
[12:37am] teancom: Ah, hrrm. Github has stopped responding for me. Anyone else?
[12:37am] vertis: working for me
[12:37am] samkottler: teancom: don't say that - you'll make benchMark and z nervous 
[12:37am] benchMark: teancom: Not seeing any problems.
[12:37am] vertis: dwradcliffe: should I merge that now?
[12:37am] vertis: 51?
[12:38am] dwradcliffe: yeah
[12:38am] teancom: I guess it's time to kick safari in the head.
[12:38am] dwradcliffe: good enuf to start with 1 of each role
[12:38am] dkannan joined the chat room.
[12:38am] vertis: merged
[12:39am] vertis: regarding github issues, same goes for everyone else
[12:39am] vertis: if you're waiting for something go check out the issues
[12:39am] vertis: and add anything you think might have been forgotten to  get this into prod
[12:39am] vertis: or tasks that need to be done
[12:40am] kek left the chat room. (Remote host closed the connection)
[12:40am] kek joined the chat room.
[12:41am] teancom_ joined the chat room.
[12:42am] teancom_: It was just TWC's DNS servers dropping offline. "yay"
[12:43am] samkottler: anyone ever seen this? https://gist.github.com/194f45c28550ad6d2ae7
[12:43am] samkottler: all the directories are correct
[12:43am] caspyin joined the chat room.
[12:44am] teancom left the chat room. (Ping timeout: 264 seconds)
[12:44am] vertis: teancom_: TWC?
[12:45am] benchMark: Time Warner Cable.
[12:45am] vertis: ah
[12:45am] vertis: of course
[12:45am] vertis: TWC means something different to me
[12:45am] vertis: Thoughtworks China
[12:46am] maledale_ joined the chat room.
[12:46am] vertis: samkottler: i haven't heard anyone report that
[12:46am] dwradcliffe: samkottler: can you use the cap task?
[12:46am] vertis: dwradcliffe: didn't even twig
[12:47am] vertis: it must be getting late
[12:49am] maledale left the chat room. (Ping timeout: 264 seconds)
[12:49am] maledale_ is now known as maledale.
[12:49am] samkottler:   * 2013-01-31 08:48:42 executing `ec2'
[12:49am] samkottler: /Users/samkottler/.rvm/gems/ruby-1.9.3-p362/gems/capistrano-2.14.1/lib/capistrano/server_definition.rb:16:in `initialize': undefined method `match' for nil:NilClass (NoMethodError)
[12:49am] samkottler: computers suck
[12:50am] dwradcliffe: samkottler: I saw that before, check your ENV vars
[12:50am] dwradcliffe: probably needs all 3
[12:50am] vertis: I see the problem here, you're one of those rvm users
[12:50am] samkottler: ╰─ echo $RUBYGEMS_EC2_APP                                                                                1 ↵
[12:50am] samkottler: ec2-50-112-220-235.us-west-2.compute.amazonaws.com
[12:50am] • vertis looks down his nose at samkottler
[12:50am] samkottler: dwradcliffe: really? okay...
[12:50am] samkottler: there isn't a DB machine right now
[12:50am] • samkottler opens emacs
[12:50am] dwradcliffe: I'll work on making them optional
[12:51am] vertis: should be easy
[12:51am] vertis: just ip an if ENV after each one
[12:51am] vertis: put
[12:51am] dwradcliffe: yep
[12:51am] blasterpal joined the chat room.
[12:51am] Holek left the chat room. (Ping timeout: 246 seconds)
[12:52am] vertis: come on evan
[12:52am] samkottler: still didn't work
[12:52am] vertis: or ReinH
[12:52am] vertis: let me check
[12:52am] samkottler: they are both west coasters, it'll probably be another hour or two
[12:53am] danmcclain joined the chat room.
[12:53am] vertis: if anyone is interested in getting involved I'm happy to spend some time bringing you up to speed
[12:54am] shapeshed joined the chat room.
[12:54am] snooc joined the chat room.
[12:54am] teancom_: Just added issue #52 (and found out github doesn't like emoji in its issue comments - no pile of poo for you!). AFK for a while, commuting.
[12:54am] samkottler: this is a knife solo bug
[12:54am] samkottler: goodie!
[12:55am] vertis: dwradcliffe: fine. dry up my code
[12:56am] vertis: dwradcliffe: I've merged that
[12:56am] dwradcliffe: 2 copies was fine, 3…not so much 
[12:56am] vertis: yeah
[12:57am] dwradcliffe: samkottler: servers should be optional now
[12:57am] fidothe joined the chat room.
[12:58am] benchMark: teancom_: That's incorrect!
[12:58am] vertis: samkottler: are you using the api to provision or console?
[12:58am] benchMark: We use emoji in issue comments all the time. 
[12:58am] samkottler: vertis: all set now
[12:59am] dwradcliffe: teancom_ benchMark we do too!
[12:59am] vertis: api though?
[01:00am] vertis: samkottler: someone could just work on a fog script or something to create their own EC2 machines for testing
[01:00am] vertis: and hook that part in before capistrano
[01:01am] samkottler: that'd be sweet - that app machine and LB are provisioning now
[01:01am] vertis: anyone work with fog?
[01:02am] vertis: stevenhaddox: do you use fog?
[01:03am] samkottler: this is an interesting race condition
[01:04am] bradland joined the chat room.
[01:04am] vertis: http://twitpic.com/bzqiko
[01:05am] stevenhaddox: vertis: no, I've always wanted to, but devops has always been too low on my client's priorities to get them to invest my time in
[01:05am] vertis: stevenhaddox: perfect, now's your chance
[01:05am] vertis:
[01:05am] stevenhaddox:
[01:05am] samkottler: the app machine is up
[01:05am] stevenhaddox: let me catch up
[01:06am] dwradcliffe: vertis: nice!
[01:06am] vertis: dwradcliffe: that was a few hours ago now
[01:06am] dwradcliffe: vertis: ha, ok
[01:06am] vertis: dwradcliffe: just letting the current shift know what the competition looks like
[01:07am] stevenhaddox: vertis: I have to step afk a few, but I'll look into fog when I get back if nobody else is on it.
[01:07am] vertis: cool
[01:07am] qrush: vertis: rubygems.org uses fog
[01:07am] qrush: only s3 stuff
[01:07am] vertis: qrush: for s3?
[01:07am] vertis: yeah I use fog all the time at work
[01:07am] vertis: but I've been at this for 14hours
[01:07am] qrush: geemus is a rubygems.org contributor - do we need him for something ? 
[01:08am] qrush: yeah dude, take a break 
[01:08am] samkottler: root@rubygems:/home/samkottler# lsof -i :3000
[01:08am] samkottler: COMMAND    PID  USER  FD  TYPE DEVICE SIZE/OFF NODE NAME
[01:08am] samkottler: ruby1.9.1 14409 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14486 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14525 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14650 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] kgrz joined the chat room.
[01:08am] samkottler: ruby1.9.1 14680 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14714 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] vertis: qrush: can't till I hand off to evan or someone
[01:08am] samkottler: ruby1.9.1 14728 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14735 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: ruby1.9.1 14744 deploy    9u  IPv4  64818      0t0  TCP *:3000 (LISTEN)
[01:08am] samkottler: win!
[01:09am] qrush: vertis: braindump into a wiki page on the rubygems-aws repo - no one is saying you have to push yourself
[01:09am] vertis: qrush: the suggestion around fog was that we could have a script to provision some AWS instances so that people looking to help could test in EC2 now
[01:09am] vertis: qrush: I will
[01:10am] shapeshed left the chat room. (Quit: leaving)
[01:10am] teancom_ left the chat room. (Remote host closed the connection)
[01:10am] dwradcliffe: I can try to work on a fog script. looks like it's essentially just one line
[01:11am] vertis: dwradcliffe: it's not really that hard
[01:11am] vertis: just beyond what I can muster
[01:15am] vertis: dwradcliffe: can you merge pull requests?
[01:16am] dwradcliffe: sure. I don't have access tho
[01:16am] vertis: dwradcliffe: was a question of access
[01:16am] vertis:
[01:16am] dwradcliffe: so no 
[01:16am] vertis: might poke rush and see who he wants to add in the mean time
[01:17am] vertis: dammit autocorrect
[01:17am] vertis: qrush*
[01:18am] qrush: Added dwradcliffe
[01:18am] qrush: and teancom too, not sure where he went
[01:18am] vertis: qrush: commuting
[01:18am] qrush: Ah 
[01:19am] dwradcliffe: qrush: thx
[01:19am] vertis: okay
[01:19am] vertis: Current state:
[01:19am] vertis:   - README is the getting started point (it should be kept up to date)
[01:19am] vertis:   - App server and LB server both work in vagrant (thanks to opscode + others)
[01:19am] vertis:   - Orchestration support using capistrano has been added (instructions in readme)
[01:19am] vertis:   - Opscode staff will be back as soon as the east coasters start coming in
[01:19am] vertis:   - Getting EC2 provisioned and deployed is the current challenge
[01:19am] vertis:   - Issues, tasks & questions on the github issues page
[01:19am] vertis:   - Everything comes via a pull request
[01:19am] vertis:   - Evan has given permission for the DB to be broken off into it's own host
[01:19am] vertis:   - No rearchitecting till it reaches prod (we all want to)
[01:19am] Hypn left the chat room. (Ping timeout: 248 seconds)
[01:20am] samkottler: vertis: that is a great snapshot of where we stand
[01:20am] samkottler: vertis: get some rest!
[01:20am] samkottler: vertis++++++++++++++
[01:20am] vertis: talk to you all in a few hours
[01:20am] dwradcliffe: vertis: thanks for your work!
[01:20am] samkottler: vertis: get as much rest as you need 
[01:21am] You are now known as vertis_sleeping.
[01:22am] qrush: vertis_sleeping: thanks man
[01:22am] snooc left the chat room. (Quit: snooc)
[01:22am] vertis_sleeping: I'm just going to dump my chat logs in a gist before I goCurrent state:
  - README is the getting started point (it should be kept up to date)
  - App server and LB server both work in vagrant (thanks to opscode + others)
  - Orchestration support using capistrano has been added (instructions in readme)
  - Opscode staff will be back as soon as the east coasters start coming in
  - Getting EC2 provisioned and deployed is the current challenge
  - Issues, tasks & questions on the github issues page
  - Everything comes via a pull request
  - Evan has given permission for the DB to be broken off into it's own host
  - No rearchitecting till it reaches prod (we all want to)