chef-client keeps randomly failing. I login to the node, stop chef-client, run it manually, and it works. 1) Why is it timing out on NTP, 4 times in a row? 2) Why doesn't it fix itself on the next run? It seems to fail, and then never recover.

gistfile1.txt

[...full client.log output...]
[2013-11-19T09:31:05-06:00] INFO: Forking chef instance to converge...
[2013-11-19T09:31:05-06:00] INFO: *** Chef 11.6.0 ***
[2013-11-19T09:31:06-06:00] INFO: Run List is [role[base], role[web], recipe[logging], recipe[app-website]]
[2013-11-19T09:31:06-06:00] INFO: Run List expands to [base, web, logging, app-website]
[2013-11-19T09:31:06-06:00] INFO: Starting Chef Run for website03.corpycorp.com
[2013-11-19T09:31:06-06:00] INFO: Running start handlers
[2013-11-19T09:31:06-06:00] INFO: Start handlers complete.
[2013-11-19T09:31:06-06:00] INFO: Loading cookbooks [apache2, app-website, apt, aws, base, build-essential, chef-client, chef-varnish, chef_handler, couchbase, cron, database, dmg, drush, firewall, git, line, logging, logrotate, mysql, nfs, ntp, openssl, php, postgresql, runit, splunk_cookbook, sudo, ufw, users, web, windows, xfs, xml, yum, zabbix]
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for service[apache2] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:210:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for service[splunk] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous service[splunk]: /var/chef/cache/cookbooks/splunk_cookbook/recipes/forwarder.rb:73:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  service[splunk]: /var/chef/cache/cookbooks/splunk_cookbook/recipes/forwarder.rb:160:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for template[/opt/splunkforwarder/etc/system/local/outputs.conf] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous template[/opt/splunkforwarder/etc/system/local/outputs.conf]: /var/chef/cache/cookbooks/splunk_cookbook/recipes/forwarder.rb:124:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  template[/opt/splunkforwarder/etc/system/local/outputs.conf]: /var/chef/cache/cookbooks/logging/recipes/default.rb:51:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for file[/etc/httpd/conf.d/ssl.conf] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:84:in `block in from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  file[/etc/httpd/conf.d/ssl.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:28:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for template[/etc/httpd/ports.conf] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:185:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  template[/etc/httpd/ports.conf]: /var/chef/cache/cookbooks/apache2/recipes/mod_ssl.rb:34:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:60:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/app-savviscom/recipes/default.rb:133:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for service[moxi-server] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for package[postfix] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous package[postfix]: /var/chef/cache/cookbooks/base/recipes/default.rb:82:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  package[postfix]: /var/chef/cache/cookbooks/app-website/recipes/default.rb:182:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for package[rsyslog] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous package[rsyslog]: /var/chef/cache/cookbooks/base/recipes/default.rb:90:in `block in from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  package[rsyslog]: /var/chef/cache/cookbooks/app-website/recipes/default.rb:185:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Cloning resource attributes for template[/etc/php.ini] from prior resource (CHEF-3694)
[2013-11-19T09:31:07-06:00] WARN: Previous template[/etc/php.ini]: /var/chef/cache/cookbooks/app-website/recipes/default.rb:96:in `from_file'
[2013-11-19T09:31:07-06:00] WARN: Current  template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from_file'
[2013-11-19T09:31:07-06:00] INFO: Processing template[/root/.profile] action create (base::default line 14)
[2013-11-19T09:31:07-06:00] INFO: Processing template[/root/.curlrc] action create (base::default line 20)
[2013-11-19T09:31:07-06:00] INFO: Processing template[/root/.bash_profile] action create (base::default line 26)
[2013-11-19T09:31:07-06:00] INFO: Processing service[iptables] action disable (base::default line 32)
[2013-11-19T09:31:07-06:00] INFO: Processing service[iptables] action stop (base::default line 32)
[2013-11-19T09:31:07-06:00] INFO: Processing template[/etc/profile.d/proxy.sh] action create (base::default line 35)
[2013-11-19T09:31:07-06:00] INFO: Processing cookbook_file[/etc/selinux/config] action create (base::default line 41)
[2013-11-19T09:31:07-06:00] INFO: Processing cookbook_file[/etc/yum/pluginconf.d/fastestmirror.conf] action create (base::default line 57)
[2013-11-19T09:31:07-06:00] INFO: Processing cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6] action create (base::default line 63)
[2013-11-19T09:31:07-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)
[2013-11-19T09:31:07-06:00] INFO: Processing yum_repository[epel] action add (yum::epel line 27)
[2013-11-19T09:31:07-06:00] INFO: Processing package[ntp] action install (base::default line 77)
[2013-11-19T09:46:18-06:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py exceeded timeout 900
[2013-11-19T09:46:18-06:00] INFO: Retrying execution of package[ntp], 3 attempt(s) left
[2013-11-19T09:46:44-06:00] INFO: Retrying execution of package[ntp], 2 attempt(s) left
[2013-11-19T09:47:09-06:00] INFO: Retrying execution of package[ntp], 1 attempt(s) left
[2013-11-19T09:47:34-06:00] INFO: Retrying execution of package[ntp], 0 attempt(s) left
[2013-11-19T09:47:59-06:00] INFO: Running queued delayed notifications before re-raising exception
[2013-11-19T09:47:59-06:00] ERROR: Running exception handlers
[2013-11-19T09:47:59-06:00] ERROR: Exception handlers complete
[2013-11-19T09:47:59-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-19T09:47:59-06:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2013-11-19T09:47:59-06:00] ERROR: Sleeping for 1800 seconds before trying again



[...stacktrace.out...]
Generated at 2013-11-19 10:38:02 -0600
Mixlib::ShellOut::ShellCommandFailed: package[ntp] (base::default line 77) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '200'
---- Begin output of /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides ----
STDOUT: [option installonlypkgs] kernel kernel-bigmem installonlypkg(kernel-module) installonlypkg(vm) kernel-enterprise kernel-smp kernel-debug kernel-unsupported kernel-source kernel-devel kernel-PAE kernel-PAE-debug
STDERR: yum-dump Locking Error! Couldn't obtain an exclusive yum lock in 10 seconds. Giving up.
---- End output of /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides ----
Ran /usr/bin/python /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py --options --installed-provides returned 200
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:251:in `invalid!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:237:in `error!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:36:in `shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in `refresh'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in `load_current_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in `do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:23:in `load'
/usr/bin/chef-client:23:in `<main>'