terrbear · October 19, 2011 03:04
diff --git a/snippet.txt b/snippet.txt
 In case y'all wanted to play with it - I'm working on getting a shitload of data into the index to see how aggressively we need to look into this. We should also compare numbers y'all are seeing vs the ones I am, since I'm running everything locally and don't have the same VM I/O performance issues that you might be running into (which might be a big deal if we host a lot of instances on VMs)

 https://github.com/bhb/rack-perftools_profiler

 I think the VM I/O performance is worth evaluating a little more (probably not before the dupe logic stuff or getting a proper CI environment or anything, but by end of 2011). Here are some (admittedly, older) posts about VM I/O:

 http://www.codinghorror.com/blog/2006/10/the-single-most-important-virtual-machine-performance-tip.html
 http://www.codinghorror.com/blog/2005/02/virtual-pc-2004-tips.html
 http://www.hanselman.com/blog/CommentView.aspx?guid=097ce75a-838a-4511-a858-d6de8e8e78a9
 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008885

 The only reason this stuck out in my head is that PeopleAdmin (a company we shared space with and that has dealt with some really-damn-big performance / scalability issues in the last few years w/ Rails) saw something similar. Earlier this year, when AWS crashed, all of PeopleAdmin's new offerings were down. Most colleges you can name off the top of your head use PA for their HR stuff (they have something insane like 80% market penetration). 

 After they crashed, because they were down for 36 hours (think of all the HR stuff that a college does… that's a long time), they decided to leave Amazon for Rackspace. They have some privacy concerns, and because RS doesn't offer VMs that aren't exposed externally, they settled with a hybrid solution, where their database and search servers were put onto physical dedicated machines, while their app servers were kept on VMs.

 I was talking to the head of development there and he said it was shocking. They saw a 50% performance increase just by changing setups. 

 I don't think this necessarily means we should try to figure out some hybrid solution or whatever, but there might be some smarter strategies we can employ on VMs to help reduce disk I/O dependence (MyISAM, more memory, fewer synchronous writes), or maybe it is worth evaluating the performance 


 that I was doing some searches locally and I think they must have been significantly faster than what Suresh was seeing, because of the dataset I loaded and ran against with almost no performance impact (8500 vulnerabilities were searched for 'cross' and 2500 came back in 200ms). 




 th
	In case y'all wanted to play with it - I'm working on getting a shitload of data into the index to see how aggressively we need to look into this. We should also compare numbers y'all are seeing vs the ones I am, since I'm running everything locally and don't have the same VM I/O performance issues that you might be running into (which might be a big deal if we host a lot of instances on VMs)

	https://github.com/bhb/rack-perftools_profiler

	I think the VM I/O performance is worth evaluating a little more (probably not before the dupe logic stuff or getting a proper CI environment or anything, but by end of 2011). Here are some (admittedly, older) posts about VM I/O:

	http://www.codinghorror.com/blog/2006/10/the-single-most-important-virtual-machine-performance-tip.html
	http://www.codinghorror.com/blog/2005/02/virtual-pc-2004-tips.html
	http://www.hanselman.com/blog/CommentView.aspx?guid=097ce75a-838a-4511-a858-d6de8e8e78a9
	http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008885

	The only reason this stuck out in my head is that PeopleAdmin (a company we shared space with and that has dealt with some really-damn-big performance / scalability issues in the last few years w/ Rails) saw something similar. Earlier this year, when AWS crashed, all of PeopleAdmin's new offerings were down. Most colleges you can name off the top of your head use PA for their HR stuff (they have something insane like 80% market penetration).

	After they crashed, because they were down for 36 hours (think of all the HR stuff that a college does… that's a long time), they decided to leave Amazon for Rackspace. They have some privacy concerns, and because RS doesn't offer VMs that aren't exposed externally, they settled with a hybrid solution, where their database and search servers were put onto physical dedicated machines, while their app servers were kept on VMs.

	I was talking to the head of development there and he said it was shocking. They saw a 50% performance increase just by changing setups.

	I don't think this necessarily means we should try to figure out some hybrid solution or whatever, but there might be some smarter strategies we can employ on VMs to help reduce disk I/O dependence (MyISAM, more memory, fewer synchronous writes), or maybe it is worth evaluating the performance


	that I was doing some searches locally and I think they must have been significantly faster than what Suresh was seeing, because of the dataset I loaded and ran against with almost no performance impact (8500 vulnerabilities were searched for 'cross' and 2500 came back in 200ms).




	th