20131003 - 76 running chef-client processes exhausting system memory

chef woes

ISSUE
===========
76 running chef-client processes exhausting system memory, causing other apps to crash (in this case zabbix), chef-client is set to run once an hour, currently showing 76 running processes.
 
[root@dc2mgmtsavpd01 ~]# ps -fe | grep chef-client | wc -l
76
 
Also, when I have failed chef-client runs like this I always find this in the ps; a chef-client worker process that is old (hung?), and a yum-dump.py that always needs to be killed before a chef-client run will be successful.
 
# ps -fe|grep chef
root      7082 13613 23 09:12 ?        00:00:00 chef-client worker: ppid=13613;start=09:12:03;
root      7221  7082 26 09:12 ?        00:00:00 /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides
 
After killing the 76 processes, and a kill -9 on the chef-client work and yum-dump.py, a chef-client run will be successful, as will others that are run automatically after that - it's just at some point down the line one will get hung up and cause this cascade of processes.
 
Additionally, I'm currently running 6 nodes, and the 4 that this always happens to are the web servers (which have 15+ recipes/packages assigned to them), the 2 db servers (which only have 2 recipes assigned) never have this issue, which makes me think something is happening with yum to gum things up.



SPECS
===========
# chef-client -v
Chef: 11.6.0
 
# cat /etc/issue.net
CentOS release 6.4 (Final)
Kernel \r on an \m
 
Vmware instance with one core and 3G RAM
 

 
LOGS
===========
[...]
[2013-10-03T09:06:37-05:00] INFO: Forking chef instance to converge...
[2013-10-03T09:06:37-05:00] INFO: *** Chef 11.6.0 ***
[2013-10-03T09:06:37-05:00] WARN: unable to detect ipaddress
[2013-10-03T09:06:37-05:00] WARN: unable to detect macaddress
[2013-10-03T09:06:37-05:00] WARN: unable to detect ip6address
[2013-10-03T09:06:37-05:00] INFO: Run List is [recipe[base], recipe[pd-web], recipe[diptables]]
[2013-10-03T09:06:37-05:00] INFO: Run List expands to [base, pd-web, diptables]
[2013-10-03T09:06:37-05:00] INFO: Starting Chef Run for dc2mgmtsavpd01.mgmt.sdirect
[2013-10-03T09:06:37-05:00] INFO: Running start handlers
[2013-10-03T09:06:37-05:00] INFO: Start handlers complete.
[2013-10-03T09:06:38-05:00] INFO: Loading cookbooks [apache2, apt, aws, base, build-essential, chef-client, chef-varnish, chef_handler, couchbase, cron, database, diptables, dmg, drush, firewall, git, line, mysql, nfs, ntp, openssl, php, postfix, postgresql, rsyslog, runit, savviscom-pd-web, splunk_handler, timezone-ii, ufw, windows, xfs, xml, yum, zabbix]
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for template[/etc/php.ini] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/php.ini]: /var/chef/cache/cookbooks/pd-web/recipes/default.rb:4:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for cookbook_file[/etc/httpd/conf/httpd.conf] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous cookbook_file[/etc/httpd/conf/httpd.conf]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:55:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  cookbook_file[/etc/httpd/conf/httpd.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:221:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for service[apache2] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:228:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for service[moxi-server] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:56:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:113:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for service[varnishlog] from prior resource (CHEF-3694)
[2013-10-03T09:06:39-05:00] WARN: Previous service[varnishlog]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:81:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Current  service[varnishlog]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:120:in `from_file'
[2013-10-03T09:06:39-05:00] WARN: Don't know how to set up automatic iptables on your distribution, sorry. Please submit a bug ticket at https://github.com/wk8/cookbook-iptables/issues
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.profile] action create (base::default line 1)
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.bash_profile] action create (base::default line 8)
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.curlrc] action create (base::default line 15)
[2013-10-03T09:06:39-05:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
[2013-10-03T09:06:39-05:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)
[2013-10-03T09:06:39-05:00] INFO: Processing package[tzdata] action install (timezone-ii::default line 16)
[2013-10-03T09:06:39-05:00] INFO: Running queued delayed notifications before re-raising exception
[2013-10-03T09:06:39-05:00] ERROR: Running exception handlers
[2013-10-03T09:06:39-05:00] ERROR: Exception handlers complete
[2013-10-03T09:06:39-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-10-03T09:06:39-05:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2013-10-03T09:06:39-05:00] ERROR: Sleeping for 3600 seconds before trying again
 
 
 
STACKTRACE
===========
# cat /var/chef/cache/chef-stacktrace.out
Generated at 2013-10-03 09:06:39 -0500
Errno::ENOMEM: package[tzdata] (timezone-ii::default line 16) had an error: Errno::ENOMEM: Cannot allocate memory - fork(2)
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork_subprocess'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in `run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in `shell_out'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in `refresh'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in `load_current_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'