Created
July 20, 2016 16:52
-
-
Save yebyen/802a2122d6a02134b0001b8723afa1aa to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:14:44UTC deis-database[1]: data=data) | |
| 2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:14:44UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:14:44UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:14:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:14:44UTC deis-database[1]: raise ex | |
| 2016-07-20T15:14:44UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:14:44UTC deis-database[1]: | |
| 2016-07-20T15:14:44UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:14:44UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:14:44.891200-00 pid=3939 | |
| 2016-07-20T15:15:47UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:15:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:15:47UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:15:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:15:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:15:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:15:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:15:47UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:15:47UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:15:47UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:15:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:15:47UTC deis-database[1]: raise ex | |
| 2016-07-20T15:15:47UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:15:47UTC deis-database[1]: <Greenlet at 0x7f17de5c7af0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:15:47UTC deis-database[1]: | |
| 2016-07-20T15:15:48UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:15:48UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:15:48UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:15:48UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:15:48UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:15:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:15:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:15:48UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:15:48UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:15:48UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:15:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:15:48UTC deis-database[1]: raise ex | |
| 2016-07-20T15:15:48UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:15:48UTC deis-database[1]: <Greenlet at 0x7f17de35f190: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:15:48UTC deis-database[1]: | |
| 2016-07-20T15:15:55UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:15:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:15:55UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:15:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:15:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:15:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:15:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:15:55UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:15:55UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:15:55UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:15:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:15:55UTC deis-database[1]: raise ex | |
| 2016-07-20T15:15:55UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:15:55UTC deis-database[1]: <Greenlet at 0x7f17de35f050: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:15:55UTC deis-database[1]: | |
| 2016-07-20T15:16:04UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:16:04UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:16:04UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:16:04UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:16:04UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:16:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:16:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:16:04UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:16:04UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:16:04UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:16:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:16:04UTC deis-database[1]: raise ex | |
| 2016-07-20T15:16:04UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:16:04UTC deis-database[1]: <Greenlet at 0x7f17de35f0f0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:16:04UTC deis-database[1]: | |
| 2016-07-20T15:16:15UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:16:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:16:15UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:16:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:16:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:16:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:16:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:16:15UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:16:15UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:16:15UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:16:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:16:15UTC deis-database[1]: raise ex | |
| 2016-07-20T15:16:15UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:16:15UTC deis-database[1]: <Greenlet at 0x7f17de5c7eb0: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:16:15UTC deis-database[1]: | |
| 2016-07-20T15:16:16UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:16:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:16:16UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:16:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:16:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:16:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:16:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:16:16UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:16:16UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:16:16UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:16:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:16:16UTC deis-database[1]: raise ex | |
| 2016-07-20T15:16:16UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:16:16UTC deis-database[1]: <Greenlet at 0x7f17de5c7f50: <wal_e.worker.upload.WalUploader object at 0x7f17de331450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:16:16UTC deis-database[1]: | |
| 2016-07-20T15:16:17UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:16:17UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:16:17UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:16:17UTC deis-database[1]: group.join() | |
| 2016-07-20T15:16:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:16:17UTC deis-database[1]: raise val | |
| 2016-07-20T15:16:17UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:16:17UTC deis-database[1]: | |
| 2016-07-20T15:16:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:17.431441-00 pid=668 | |
| 2016-07-20T15:16:17UTC deis-database[1]: 2016-07-20 15:16:17 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:16:17UTC deis-database[1]: 2016-07-20 15:16:17 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:16:18UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:16:18UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:16:18UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:16:18UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:16:18UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:16:18UTC deis-database[1]: data=data) | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:16:18UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:16:18UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:16:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:16:18UTC deis-database[1]: raise ex | |
| 2016-07-20T15:16:18UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:16:18UTC deis-database[1]: | |
| 2016-07-20T15:16:18UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.421876-00 pid=3939 | |
| 2016-07-20T15:16:18UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:16:18UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.795435-00 pid=833 | |
| 2016-07-20T15:16:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.820306-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:16:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:16:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:18.966591-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.108685-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.247890-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.387682-00 pid=833 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.528131-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.673231-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:16:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:16:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:16:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:16:19.816419-00 pid=833 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:17:14UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:17:14UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:17:14UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:17:14UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:17:14UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:17:14UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:17:14UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:17:14UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:17:14UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:17:14UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:17:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:17:14UTC deis-database[1]: raise ex | |
| 2016-07-20T15:17:14UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:17:14UTC deis-database[1]: <Greenlet at 0x7fab9395d190: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:17:14UTC deis-database[1]: | |
| 2016-07-20T15:17:18UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:17:18UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:17:18UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:17:18UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:17:18UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:17:18UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:17:18UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:17:18UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:17:18UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:17:18UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:17:18UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:17:18UTC deis-database[1]: raise ex | |
| 2016-07-20T15:17:18UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:17:18UTC deis-database[1]: <Greenlet at 0x7fab93bc5af0: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:17:18UTC deis-database[1]: | |
| 2016-07-20T15:17:21UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:17:21UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:17:21UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:17:21UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:17:21UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:17:21UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:17:21UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:17:21UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:17:21UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:17:21UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:17:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:17:21UTC deis-database[1]: raise ex | |
| 2016-07-20T15:17:21UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:17:21UTC deis-database[1]: <Greenlet at 0x7fab9395d050: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:17:21UTC deis-database[1]: | |
| 2016-07-20T15:17:34UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:17:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:17:34UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:17:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:17:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:17:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:17:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:17:34UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:17:34UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:17:34UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:17:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:17:34UTC deis-database[1]: raise ex | |
| 2016-07-20T15:17:34UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:17:34UTC deis-database[1]: <Greenlet at 0x7fab9395d0f0: <wal_e.worker.upload.WalUploader object at 0x7fab9392f450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:17:34UTC deis-database[1]: | |
| 2016-07-20T15:17:44UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:17:44UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:17:44UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:17:44UTC deis-database[1]: group.join() | |
| 2016-07-20T15:17:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:17:44UTC deis-database[1]: raise val | |
| 2016-07-20T15:17:44UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:17:44UTC deis-database[1]: | |
| 2016-07-20T15:17:44UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:44.968579-00 pid=833 | |
| 2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:17:44UTC deis-database[1]: 2016-07-20 15:17:44 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.316185-00 pid=956 | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.341471-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.490105-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.628498-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.769306-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:17:45UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:45UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:17:45UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:45.914565-00 pid=956 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.072386-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.209305-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:17:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:17:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:17:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:17:46.360380-00 pid=956 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:18:35UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:18:35UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:18:35UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:18:35UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:18:35UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:18:35UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:18:35UTC deis-database[1]: data=data) | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:18:35UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:18:35UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:18:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:18:35UTC deis-database[1]: raise ex | |
| 2016-07-20T15:18:35UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:18:35UTC deis-database[1]: | |
| 2016-07-20T15:18:35UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:18:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:18:35.336780-00 pid=3939 | |
| 2016-07-20T15:18:58UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:18:58UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:18:58UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:18:58UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:18:58UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:18:58UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:18:58UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:18:58UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:18:58UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:18:58UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:18:58UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:18:58UTC deis-database[1]: raise ex | |
| 2016-07-20T15:18:58UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:18:58UTC deis-database[1]: <Greenlet at 0x7fce507c2af0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:18:58UTC deis-database[1]: | |
| 2016-07-20T15:19:03UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:19:03UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:19:03UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:19:03UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:19:03UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:19:03UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:19:03UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:19:03UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:19:03UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:19:03UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:19:03UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:19:03UTC deis-database[1]: raise ex | |
| 2016-07-20T15:19:03UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:03UTC deis-database[1]: <Greenlet at 0x7fce5055a0f0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:19:03UTC deis-database[1]: | |
| 2016-07-20T15:19:05UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:19:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:19:05UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:19:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:19:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:19:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:19:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:19:05UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:19:05UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:19:05UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:19:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:19:05UTC deis-database[1]: raise ex | |
| 2016-07-20T15:19:05UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:05UTC deis-database[1]: <Greenlet at 0x7fce5055a190: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:19:05UTC deis-database[1]: | |
| 2016-07-20T15:19:17UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:19:17UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:19:17UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:19:17UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:19:17UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:19:17UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:19:17UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:19:17UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:19:17UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:19:17UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:19:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:19:17UTC deis-database[1]: raise ex | |
| 2016-07-20T15:19:17UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:17UTC deis-database[1]: <Greenlet at 0x7fce5055a230: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:19:17UTC deis-database[1]: | |
| 2016-07-20T15:19:25UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:19:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:19:25UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:19:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:19:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:19:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:19:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:19:25UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:19:25UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:19:25UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:19:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:19:25UTC deis-database[1]: raise ex | |
| 2016-07-20T15:19:25UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:25UTC deis-database[1]: <Greenlet at 0x7fce507c2eb0: <wal_e.worker.upload.WalUploader object at 0x7fce5052c450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:19:25UTC deis-database[1]: | |
| 2016-07-20T15:19:28UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:19:28UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:19:28UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:19:28UTC deis-database[1]: group.join() | |
| 2016-07-20T15:19:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:19:28UTC deis-database[1]: raise val | |
| 2016-07-20T15:19:28UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:28UTC deis-database[1]: | |
| 2016-07-20T15:19:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:28.122127-00 pid=956 | |
| 2016-07-20T15:19:28UTC deis-database[1]: 2016-07-20 15:19:28 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:19:28UTC deis-database[1]: 2016-07-20 15:19:28 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:19:29UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:19:29UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.495208-00 pid=1103 | |
| 2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.525129-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.676790-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.825022-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:19:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:19:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:29.967671-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.121673-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.266397-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.413886-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:19:30UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:19:30UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:19:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:30.565544-00 pid=1103 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:19:55UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:19:55UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:19:55UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:19:55UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:19:55UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:19:55UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:19:55UTC deis-database[1]: data=data) | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:19:55UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:19:55UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:19:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:19:55UTC deis-database[1]: raise ex | |
| 2016-07-20T15:19:55UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:19:55UTC deis-database[1]: | |
| 2016-07-20T15:19:55UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:19:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:19:55.498308-00 pid=3939 | |
| 2016-07-20T15:20:20UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:20:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:20:20UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:20:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:20:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:20:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:20:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:20:20UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:20:20UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:20:20UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:20:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:20:20UTC deis-database[1]: raise ex | |
| 2016-07-20T15:20:20UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:20:20UTC deis-database[1]: <Greenlet at 0x7fafc4f8b0f0: <wal_e.worker.upload.WalUploader object at 0x7fafc4f5d450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:20:20UTC deis-database[1]: | |
| 2016-07-20T15:20:40UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:20:40UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:20:40UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:20:40UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:20:40UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:20:40UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:20:40UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:20:40UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:20:40UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:20:40UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:20:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:20:40UTC deis-database[1]: raise ex | |
| 2016-07-20T15:20:40UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:20:40UTC deis-database[1]: <Greenlet at 0x7fafc51f3eb0: <wal_e.worker.upload.WalUploader object at 0x7fafc4f5d450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:20:40UTC deis-database[1]: | |
| 2016-07-20T15:20:50UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:20:50UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:20:50UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:20:50UTC deis-database[1]: group.join() | |
| 2016-07-20T15:20:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:20:50UTC deis-database[1]: raise val | |
| 2016-07-20T15:20:50UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:20:50UTC deis-database[1]: | |
| 2016-07-20T15:20:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:50.311307-00 pid=1103 | |
| 2016-07-20T15:20:50UTC deis-database[1]: 2016-07-20 15:20:50 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:20:50UTC deis-database[1]: 2016-07-20 15:20:50 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:20:51UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:20:51UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.676403-00 pid=1216 | |
| 2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.703263-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.840990-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:20:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:20:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:51.978360-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.123078-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.264729-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.404008-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.545761-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:20:52UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:20:52UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:20:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:20:52.697470-00 pid=1216 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:21:27UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:21:27UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:21:27UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:21:27UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:21:27UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:21:27UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:21:27UTC deis-database[1]: data=data) | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:21:27UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:21:27UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:21:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:21:27UTC deis-database[1]: raise ex | |
| 2016-07-20T15:21:27UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:21:27UTC deis-database[1]: | |
| 2016-07-20T15:21:27UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:21:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:21:27.734061-00 pid=3939 | |
| 2016-07-20T15:22:19UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:19UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:19UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:19UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:19UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:19UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:19UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:19UTC deis-database[1]: <Greenlet at 0x7ff645e26af0: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:19UTC deis-database[1]: | |
| 2016-07-20T15:22:27UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:27UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:27UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:27UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:27UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:27UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:27UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:27UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:27UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:27UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:27UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:27UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:27UTC deis-database[1]: <Greenlet at 0x7ff645bbe050: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:27UTC deis-database[1]: | |
| 2016-07-20T15:22:35UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:35UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:35UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:35UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:35UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:35UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:35UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:35UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:35UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:35UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:35UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:35UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:35UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:35UTC deis-database[1]: <Greenlet at 0x7ff645e26eb0: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:35UTC deis-database[1]: | |
| 2016-07-20T15:22:37UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:37UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:37UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:37UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:37UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:37UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:37UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:37UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:37UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:37UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:37UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:37UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:37UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:37UTC deis-database[1]: <Greenlet at 0x7ff645e26b90: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:37UTC deis-database[1]: | |
| 2016-07-20T15:22:39UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:39UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:39UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:39UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:39UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:39UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:39UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:39UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:39UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:39UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:39UTC deis-database[1]: <Greenlet at 0x7ff645bbe190: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:39UTC deis-database[1]: | |
| 2016-07-20T15:22:43UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:22:43UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:22:43UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:22:43UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:22:43UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:22:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:22:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:22:43UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:43UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:43UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:43UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:43UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:43UTC deis-database[1]: <Greenlet at 0x7ff645e26f50: <wal_e.worker.upload.WalUploader object at 0x7ff645b90450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:22:43UTC deis-database[1]: | |
| 2016-07-20T15:22:49UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:22:49UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:22:49UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:22:49UTC deis-database[1]: group.join() | |
| 2016-07-20T15:22:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:22:49UTC deis-database[1]: raise val | |
| 2016-07-20T15:22:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:49UTC deis-database[1]: | |
| 2016-07-20T15:22:49UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:49.648320-00 pid=1216 | |
| 2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:22:49UTC deis-database[1]: 2016-07-20 15:22:49 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.008261-00 pid=1381 | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.032432-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.175039-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.318255-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.462605-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.604756-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.747163-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:22:50UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:50UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:22:50UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:50.885062-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:22:51UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:22:51UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:22:51UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:51.026287-00 pid=1381 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:22:52UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:22:52UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:22:52UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:22:52UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:22:52UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:22:52UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:22:52UTC deis-database[1]: data=data) | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:22:52UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:22:52UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:22:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:22:52UTC deis-database[1]: raise ex | |
| 2016-07-20T15:22:52UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:22:52UTC deis-database[1]: | |
| 2016-07-20T15:22:52UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:22:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:22:52.777145-00 pid=3939 | |
| 2016-07-20T15:24:10UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:24:10UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:24:10UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:24:10UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:24:10UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:24:10UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:24:10UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:24:10UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:10UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:10UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:10UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:10UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:10UTC deis-database[1]: <Greenlet at 0x7fb32d00eeb0: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:24:10UTC deis-database[1]: | |
| 2016-07-20T15:24:11UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:24:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:24:11UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:24:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:24:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:24:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:24:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:24:11UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:11UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:11UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:11UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:11UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:11UTC deis-database[1]: <Greenlet at 0x7fb32d00ef50: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:24:11UTC deis-database[1]: | |
| 2016-07-20T15:24:15UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:24:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:24:15UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:24:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:24:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:24:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:24:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:24:15UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:15UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:15UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:15UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:15UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:15UTC deis-database[1]: <Greenlet at 0x7fb32cda6050: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:24:15UTC deis-database[1]: | |
| 2016-07-20T15:24:20UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:24:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:24:20UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:24:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:24:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:24:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:24:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:24:20UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:20UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:20UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:20UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:20UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:20UTC deis-database[1]: <Greenlet at 0x7fb32cda60f0: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:24:20UTC deis-database[1]: | |
| 2016-07-20T15:24:21UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:24:21UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:24:21UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:24:21UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:24:21UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:24:21UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:24:21UTC deis-database[1]: data=data) | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:21UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:21UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:21UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:21UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:21UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:21UTC deis-database[1]: | |
| 2016-07-20T15:24:21UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:24:21UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:21.850205-00 pid=3939 | |
| 2016-07-20T15:24:22UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:24:22UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:24:22UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:24:22UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:24:22UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:24:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:24:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:24:22UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:24:22UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:24:22UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:24:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:24:22UTC deis-database[1]: raise ex | |
| 2016-07-20T15:24:22UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:22UTC deis-database[1]: <Greenlet at 0x7fb32d00eb90: <wal_e.worker.upload.WalUploader object at 0x7fb32cd78450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:24:22UTC deis-database[1]: | |
| 2016-07-20T15:24:40UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:24:40UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:24:40UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:24:40UTC deis-database[1]: group.join() | |
| 2016-07-20T15:24:40UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:24:40UTC deis-database[1]: raise val | |
| 2016-07-20T15:24:40UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:24:40UTC deis-database[1]: | |
| 2016-07-20T15:24:40UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:40.140473-00 pid=1381 | |
| 2016-07-20T15:24:40UTC deis-database[1]: 2016-07-20 15:24:40 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:24:40UTC deis-database[1]: 2016-07-20 15:24:40 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:24:41UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:24:41UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.495699-00 pid=1534 | |
| 2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.522590-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.670634-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.814237-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:24:41UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:41UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:24:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:41.966028-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.132804-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.284598-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.435181-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:24:42UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:24:42UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:24:42UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:24:42.582480-00 pid=1534 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:25:54UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:25:54UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:25:54UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:25:54UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:25:54UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:25:54UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:25:54UTC deis-database[1]: data=data) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:25:54UTC deis-database[1]: raise ex | |
| 2016-07-20T15:25:54UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:25:54UTC deis-database[1]: | |
| 2016-07-20T15:25:54UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:25:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:25:54.229196-00 pid=3939 | |
| 2016-07-20T15:25:54UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:25:54UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:25:54UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:25:54UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:25:54UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:25:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:25:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:25:54UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:25:54UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:25:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:25:54UTC deis-database[1]: raise ex | |
| 2016-07-20T15:25:54UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:25:54UTC deis-database[1]: <Greenlet at 0x7fbf6d816190: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:25:54UTC deis-database[1]: | |
| 2016-07-20T15:26:00UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:26:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:26:00UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:26:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:26:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:26:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:26:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:26:00UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:26:00UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:26:00UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:26:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:26:00UTC deis-database[1]: raise ex | |
| 2016-07-20T15:26:00UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:26:00UTC deis-database[1]: <Greenlet at 0x7fbf6da7eb90: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:26:00UTC deis-database[1]: | |
| 2016-07-20T15:26:15UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:26:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:26:15UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:26:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:26:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:26:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:26:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:26:15UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:26:15UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:26:15UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:26:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:26:15UTC deis-database[1]: raise ex | |
| 2016-07-20T15:26:15UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:26:15UTC deis-database[1]: <Greenlet at 0x7fbf6da7eeb0: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:26:15UTC deis-database[1]: | |
| 2016-07-20T15:26:16UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:26:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:26:16UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:26:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:26:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:26:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:26:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:26:16UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:26:16UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:26:16UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:26:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:26:16UTC deis-database[1]: raise ex | |
| 2016-07-20T15:26:16UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:26:16UTC deis-database[1]: <Greenlet at 0x7fbf6d816230: <wal_e.worker.upload.WalUploader object at 0x7fbf6d7e8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:26:16UTC deis-database[1]: | |
| 2016-07-20T15:26:24UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:26:24UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:26:24UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:26:24UTC deis-database[1]: group.join() | |
| 2016-07-20T15:26:24UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:26:24UTC deis-database[1]: raise val | |
| 2016-07-20T15:26:24UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:26:24UTC deis-database[1]: | |
| 2016-07-20T15:26:24UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:24.464774-00 pid=1534 | |
| 2016-07-20T15:26:24UTC deis-database[1]: 2016-07-20 15:26:24 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:26:24UTC deis-database[1]: 2016-07-20 15:26:24 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:26:25UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:26:25UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.820185-00 pid=1675 | |
| 2016-07-20T15:26:25UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:25UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.848009-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:26:25UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:25UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:26:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:25.992467-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.136684-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.285525-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.437192-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.586899-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.727482-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:26:26UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:26:26UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:26:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:26:26.869152-00 pid=1675 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:27:41UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:27:41UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:27:41UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:27:41UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:27:41UTC deis-database[1]: raise ex | |
| 2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:27:41UTC deis-database[1]: <Greenlet at 0x7ff25647af50: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:27:41UTC deis-database[1]: | |
| 2016-07-20T15:27:41UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:27:41UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:27:41UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:27:41UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:27:41UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:27:41UTC deis-database[1]: data=data) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:27:41UTC deis-database[1]: raise ex | |
| 2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:27:41UTC deis-database[1]: | |
| 2016-07-20T15:27:41UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:27:41UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:27:41.630147-00 pid=3939 | |
| 2016-07-20T15:27:41UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:27:41UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:27:41UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:27:41UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:27:41UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:27:41UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:27:41UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:27:41UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:27:41UTC deis-database[1]: raise ex | |
| 2016-07-20T15:27:41UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:27:41UTC deis-database[1]: <Greenlet at 0x7ff25647aaf0: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:27:41UTC deis-database[1]: | |
| 2016-07-20T15:27:53UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:27:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:27:53UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:27:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:27:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:27:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:27:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:27:53UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:27:53UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:27:53UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:27:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:27:53UTC deis-database[1]: raise ex | |
| 2016-07-20T15:27:53UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:27:53UTC deis-database[1]: <Greenlet at 0x7ff256212230: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:27:53UTC deis-database[1]: | |
| 2016-07-20T15:28:00UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:28:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:28:00UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:28:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:28:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:28:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:28:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:28:00UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:28:00UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:28:00UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:28:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:28:00UTC deis-database[1]: raise ex | |
| 2016-07-20T15:28:00UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:28:00UTC deis-database[1]: <Greenlet at 0x7ff25647ab90: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:28:00UTC deis-database[1]: | |
| 2016-07-20T15:28:09UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:28:09UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:28:09UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:28:09UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:28:09UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:28:09UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:28:09UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:28:09UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:28:09UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:28:09UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:28:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:28:09UTC deis-database[1]: raise ex | |
| 2016-07-20T15:28:09UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:28:09UTC deis-database[1]: <Greenlet at 0x7ff256212050: <wal_e.worker.upload.WalUploader object at 0x7ff2561e4450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:28:09UTC deis-database[1]: | |
| 2016-07-20T15:28:11UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:28:11UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:28:11UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:28:11UTC deis-database[1]: group.join() | |
| 2016-07-20T15:28:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:28:11UTC deis-database[1]: raise val | |
| 2016-07-20T15:28:11UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:28:11UTC deis-database[1]: | |
| 2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.569622-00 pid=1675 | |
| 2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:28:11UTC deis-database[1]: 2016-07-20 15:28:11 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:28:11UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:28:11UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.927515-00 pid=1815 | |
| 2016-07-20T15:28:11UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:11UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:28:11UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:11.952909-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.092486-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.234062-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.369795-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.505606-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.643469-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.786116-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:28:12UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:28:12UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:28:12UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:28:12.930001-00 pid=1815 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:29:01UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:29:01UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:29:01UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:29:01UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:29:01UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:29:01UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:29:01UTC deis-database[1]: data=data) | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:29:01UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:29:01UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:29:01UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:29:01UTC deis-database[1]: raise ex | |
| 2016-07-20T15:29:01UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:01UTC deis-database[1]: | |
| 2016-07-20T15:29:01UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:29:01UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:01.411161-00 pid=3939 | |
| 2016-07-20T15:29:02UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:29:02UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:29:02UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:29:02UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:29:02UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:29:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:29:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:29:02UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:29:02UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:29:02UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:29:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:29:02UTC deis-database[1]: raise ex | |
| 2016-07-20T15:29:02UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:02UTC deis-database[1]: <Greenlet at 0x7fdafdfedf50: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:29:02UTC deis-database[1]: | |
| 2016-07-20T15:29:05UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:29:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:29:05UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:29:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:29:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:29:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:29:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:29:05UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:29:05UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:29:05UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:29:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:29:05UTC deis-database[1]: raise ex | |
| 2016-07-20T15:29:05UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:05UTC deis-database[1]: <Greenlet at 0x7fdafdd85050: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:29:05UTC deis-database[1]: | |
| 2016-07-20T15:29:15UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:29:15UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:29:15UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:29:15UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:29:15UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:29:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:29:15UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:29:15UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:29:15UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:29:15UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:29:15UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:29:15UTC deis-database[1]: raise ex | |
| 2016-07-20T15:29:15UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:15UTC deis-database[1]: <Greenlet at 0x7fdafdfedaf0: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:29:15UTC deis-database[1]: | |
| 2016-07-20T15:29:30UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:29:30UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:29:30UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:29:30UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:29:30UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:29:30UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:29:30UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:29:30UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:29:30UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:29:30UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:29:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:29:30UTC deis-database[1]: raise ex | |
| 2016-07-20T15:29:30UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:30UTC deis-database[1]: <Greenlet at 0x7fdafdd85190: <wal_e.worker.upload.WalUploader object at 0x7fdafdd57450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:29:30UTC deis-database[1]: | |
| 2016-07-20T15:29:32UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:29:32UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:29:32UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:29:32UTC deis-database[1]: group.join() | |
| 2016-07-20T15:29:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:29:32UTC deis-database[1]: raise val | |
| 2016-07-20T15:29:32UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:29:32UTC deis-database[1]: | |
| 2016-07-20T15:29:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:32.926825-00 pid=1815 | |
| 2016-07-20T15:29:32UTC deis-database[1]: 2016-07-20 15:29:32 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:29:32UTC deis-database[1]: 2016-07-20 15:29:32 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.291263-00 pid=1943 | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.315443-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.465485-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.600021-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.748890-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:29:34UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:34UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:29:34UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:34.889596-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.024928-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.164896-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:29:35UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:29:35UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:29:35UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:29:35.307267-00 pid=1943 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:30:26UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:30:26UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:30:26UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:30:26UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:30:26UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:30:26UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:30:26UTC deis-database[1]: data=data) | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:30:26UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:30:26UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:30:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:30:26UTC deis-database[1]: raise ex | |
| 2016-07-20T15:30:26UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:30:26UTC deis-database[1]: | |
| 2016-07-20T15:30:26UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:30:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:30:26.970691-00 pid=3939 | |
| 2016-07-20T15:30:36UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:30:36UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:30:36UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:30:36UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:30:36UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:30:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:30:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:30:36UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:30:36UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:30:36UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:30:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:30:36UTC deis-database[1]: raise ex | |
| 2016-07-20T15:30:36UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:30:36UTC deis-database[1]: <Greenlet at 0x7f415ad70b90: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:30:36UTC deis-database[1]: | |
| 2016-07-20T15:30:44UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:30:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:30:44UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:30:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:30:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:30:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:30:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:30:44UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:30:44UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:30:44UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:30:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:30:44UTC deis-database[1]: raise ex | |
| 2016-07-20T15:30:44UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:30:44UTC deis-database[1]: <Greenlet at 0x7f415ab08050: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:30:44UTC deis-database[1]: | |
| 2016-07-20T15:30:45UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:30:45UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:30:45UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:30:45UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:30:45UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:30:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:30:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:30:45UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:30:45UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:30:45UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:30:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:30:45UTC deis-database[1]: raise ex | |
| 2016-07-20T15:30:45UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:30:45UTC deis-database[1]: <Greenlet at 0x7f415ad70f50: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:30:45UTC deis-database[1]: | |
| 2016-07-20T15:30:48UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:30:48UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:30:48UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:30:48UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:30:48UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:30:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:30:48UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:30:48UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:30:48UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:30:48UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:30:48UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:30:48UTC deis-database[1]: raise ex | |
| 2016-07-20T15:30:48UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:30:48UTC deis-database[1]: <Greenlet at 0x7f415ab08230: <wal_e.worker.upload.WalUploader object at 0x7f415aada450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:30:48UTC deis-database[1]: | |
| 2016-07-20T15:31:06UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:31:06UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:31:06UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:31:06UTC deis-database[1]: group.join() | |
| 2016-07-20T15:31:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:31:06UTC deis-database[1]: raise val | |
| 2016-07-20T15:31:06UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:31:06UTC deis-database[1]: | |
| 2016-07-20T15:31:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:06.550339-00 pid=1943 | |
| 2016-07-20T15:31:06UTC deis-database[1]: 2016-07-20 15:31:06 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:31:06UTC deis-database[1]: 2016-07-20 15:31:06 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:31:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:31:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:31:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:07.914574-00 pid=2071 | |
| 2016-07-20T15:31:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:31:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:07.940828-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.095672-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.245805-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.385529-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.527124-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.668800-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.812164-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:31:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:31:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:31:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:08.960555-00 pid=2071 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:31:56UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:31:56UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:31:56UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:31:56UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:31:56UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:31:56UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:31:56UTC deis-database[1]: data=data) | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:31:56UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:31:56UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:31:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:31:56UTC deis-database[1]: raise ex | |
| 2016-07-20T15:31:56UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:31:56UTC deis-database[1]: | |
| 2016-07-20T15:31:56UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:31:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:31:56.321967-00 pid=3939 | |
| 2016-07-20T15:32:16UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:16UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:16UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:16UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:16UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:16UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:16UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:16UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:16UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:16UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:16UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:16UTC deis-database[1]: <Greenlet at 0x7f4654d27050: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:16UTC deis-database[1]: | |
| 2016-07-20T15:32:19UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:19UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:19UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:19UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:19UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:19UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:19UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:19UTC deis-database[1]: <Greenlet at 0x7f4654d27190: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:19UTC deis-database[1]: | |
| 2016-07-20T15:32:25UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:25UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:25UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:25UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:25UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:25UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:25UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:25UTC deis-database[1]: <Greenlet at 0x7f4654d27230: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:25UTC deis-database[1]: | |
| 2016-07-20T15:32:28UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:28UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:28UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:28UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:28UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:28UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:28UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:28UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:28UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:28UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:28UTC deis-database[1]: <Greenlet at 0x7f4654f8feb0: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:28UTC deis-database[1]: | |
| 2016-07-20T15:32:29UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:29UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:29UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:29UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:29UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:29UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:29UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:29UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:29UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:29UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:29UTC deis-database[1]: <Greenlet at 0x7f4654f8faf0: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:29UTC deis-database[1]: | |
| 2016-07-20T15:32:39UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:32:39UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:32:39UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:32:39UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:32:39UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:32:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:32:39UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:32:39UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:32:39UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:32:39UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:32:39UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:32:39UTC deis-database[1]: raise ex | |
| 2016-07-20T15:32:39UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:39UTC deis-database[1]: <Greenlet at 0x7f4654f8ff50: <wal_e.worker.upload.WalUploader object at 0x7f4654cf9450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:32:39UTC deis-database[1]: | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:32:46UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:32:46UTC deis-database[1]: group.join() | |
| 2016-07-20T15:32:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:32:46UTC deis-database[1]: raise val | |
| 2016-07-20T15:32:46UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:32:46UTC deis-database[1]: | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.105032-00 pid=2071 | |
| 2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:32:46UTC deis-database[1]: 2016-07-20 15:32:46 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.455597-00 pid=2211 | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.480692-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.622949-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.760061-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:32:46UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:46UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:32:46UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:46.904812-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.042464-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.182300-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.317402-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:32:47UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:32:47UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:32:47UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:32:47.454389-00 pid=2211 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:33:56UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:33:56UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:33:56UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:33:56UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:33:56UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:33:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:33:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:33:56UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:33:56UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:33:56UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:33:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:33:56UTC deis-database[1]: raise ex | |
| 2016-07-20T15:33:56UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:33:56UTC deis-database[1]: <Greenlet at 0x7f63e95ae190: <wal_e.worker.upload.WalUploader object at 0x7f63e9580450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:33:56UTC deis-database[1]: | |
| 2016-07-20T15:33:57UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:33:57UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:33:57UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:33:57UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:33:57UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:33:57UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:33:57UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:33:57UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:33:57UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:33:57UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:33:57UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:33:57UTC deis-database[1]: raise ex | |
| 2016-07-20T15:33:57UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:33:57UTC deis-database[1]: <Greenlet at 0x7f63e9816f50: <wal_e.worker.upload.WalUploader object at 0x7f63e9580450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:33:57UTC deis-database[1]: | |
| 2016-07-20T15:33:59UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:33:59UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:33:59UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:33:59UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:33:59UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:33:59UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:33:59UTC deis-database[1]: data=data) | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:33:59UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:33:59UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:33:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:33:59UTC deis-database[1]: raise ex | |
| 2016-07-20T15:33:59UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:33:59UTC deis-database[1]: | |
| 2016-07-20T15:33:59UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:33:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:33:59.245414-00 pid=3939 | |
| 2016-07-20T15:34:26UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:34:26UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:34:26UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:34:26UTC deis-database[1]: group.join() | |
| 2016-07-20T15:34:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:34:26UTC deis-database[1]: raise val | |
| 2016-07-20T15:34:26UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:34:26UTC deis-database[1]: | |
| 2016-07-20T15:34:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:26.636581-00 pid=2211 | |
| 2016-07-20T15:34:26UTC deis-database[1]: 2016-07-20 15:34:26 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:34:26UTC deis-database[1]: 2016-07-20 15:34:26 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:34:27UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:34:27UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:34:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:27.985965-00 pid=2352 | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.011209-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.157331-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.300840-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.443553-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.586209-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.722892-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:34:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:34:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:28.862425-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:34:29UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:34:29UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:34:29UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:34:29.013908-00 pid=2352 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:35:00UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:35:00UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:35:00UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:35:00UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:35:00UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:35:00UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:35:00UTC deis-database[1]: data=data) | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:00UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:00UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:00UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:00UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:00UTC deis-database[1]: | |
| 2016-07-20T15:35:00UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:35:00UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:00.068286-00 pid=3939 | |
| 2016-07-20T15:35:23UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:23UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:23UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:23UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:23UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:23UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:23UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:23UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:23UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:23UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:23UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:23UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:23UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:23UTC deis-database[1]: <Greenlet at 0x7fb203dfdb90: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:23UTC deis-database[1]: | |
| 2016-07-20T15:35:36UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:36UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:36UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:36UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:36UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:36UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:36UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:36UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:36UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:36UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:36UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:36UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:36UTC deis-database[1]: <Greenlet at 0x7fb203b95190: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:36UTC deis-database[1]: | |
| 2016-07-20T15:35:38UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:38UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:38UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:38UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:38UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:38UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:38UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:38UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:38UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:38UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:38UTC deis-database[1]: <Greenlet at 0x7fb203dfdaf0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:38UTC deis-database[1]: | |
| 2016-07-20T15:35:44UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:44UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:44UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:44UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:44UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:44UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:44UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:44UTC deis-database[1]: <Greenlet at 0x7fb203b95230: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:44UTC deis-database[1]: | |
| 2016-07-20T15:35:49UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:49UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:49UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:49UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:49UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:49UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:49UTC deis-database[1]: <Greenlet at 0x7fb203b950f0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:49UTC deis-database[1]: | |
| 2016-07-20T15:35:52UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:35:52UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:35:52UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:35:52UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:35:52UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:35:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:35:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:35:52UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:35:52UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:35:52UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:35:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:35:52UTC deis-database[1]: raise ex | |
| 2016-07-20T15:35:52UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:52UTC deis-database[1]: <Greenlet at 0x7fb203dfdeb0: <wal_e.worker.upload.WalUploader object at 0x7fb203b67450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:35:52UTC deis-database[1]: | |
| 2016-07-20T15:35:53UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:35:53UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:35:53UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:35:53UTC deis-database[1]: group.join() | |
| 2016-07-20T15:35:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:35:53UTC deis-database[1]: raise val | |
| 2016-07-20T15:35:53UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:35:53UTC deis-database[1]: | |
| 2016-07-20T15:35:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:53.620273-00 pid=2352 | |
| 2016-07-20T15:35:53UTC deis-database[1]: 2016-07-20 15:35:53 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:35:53UTC deis-database[1]: 2016-07-20 15:35:53 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:35:54UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:35:54UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:35:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:54.964755-00 pid=2468 | |
| 2016-07-20T15:35:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:35:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:54.992378-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.140222-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.284697-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.451196-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.598838-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.743298-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:35:55UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:55UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:35:55UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:55.885363-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:35:56UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:35:56UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:35:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:35:56.026037-00 pid=2468 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:36:13UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:36:13UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:36:13UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:36:13UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:36:13UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:36:13UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:36:13UTC deis-database[1]: data=data) | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:36:13UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:36:13UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:36:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:36:13UTC deis-database[1]: raise ex | |
| 2016-07-20T15:36:13UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:36:13UTC deis-database[1]: | |
| 2016-07-20T15:36:13UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:36:13UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:36:13.500027-00 pid=3939 | |
| 2016-07-20T15:36:50UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:36:50UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:36:50UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:36:50UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:36:50UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:36:50UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:36:50UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:36:50UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:36:50UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:36:50UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:36:50UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:36:50UTC deis-database[1]: raise ex | |
| 2016-07-20T15:36:50UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:36:50UTC deis-database[1]: <Greenlet at 0x7f587d0bf0f0: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:36:50UTC deis-database[1]: | |
| 2016-07-20T15:36:55UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:36:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:36:55UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:36:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:36:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:36:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:36:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:36:55UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:36:55UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:36:55UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:36:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:36:55UTC deis-database[1]: raise ex | |
| 2016-07-20T15:36:55UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:36:55UTC deis-database[1]: <Greenlet at 0x7f587d327eb0: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:36:55UTC deis-database[1]: | |
| 2016-07-20T15:37:04UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:37:04UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:37:04UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:37:04UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:37:04UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:37:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:37:04UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:37:04UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:37:04UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:37:04UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:37:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:37:04UTC deis-database[1]: raise ex | |
| 2016-07-20T15:37:04UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:37:04UTC deis-database[1]: <Greenlet at 0x7f587d0bf190: <wal_e.worker.upload.WalUploader object at 0x7f587d091450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:37:04UTC deis-database[1]: | |
| 2016-07-20T15:37:20UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:37:20UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:37:20UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:37:20UTC deis-database[1]: group.join() | |
| 2016-07-20T15:37:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:37:20UTC deis-database[1]: raise val | |
| 2016-07-20T15:37:20UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:37:20UTC deis-database[1]: | |
| 2016-07-20T15:37:20UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:20.613984-00 pid=2468 | |
| 2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:37:20UTC deis-database[1]: 2016-07-20 15:37:20 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.173720-00 pid=2608 | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.198996-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.341395-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.492550-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.636081-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.781720-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:37:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:37:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:38.922592-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:37:39UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:39UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:37:39UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:39.099734-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:37:39UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:37:39UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:37:39UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:37:39.243137-00 pid=2608 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:38:05UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:38:05UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:38:05UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:38:05UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:38:05UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:38:05UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:38:05UTC deis-database[1]: data=data) | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:38:05UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:38:05UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:38:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:38:05UTC deis-database[1]: raise ex | |
| 2016-07-20T15:38:05UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:38:05UTC deis-database[1]: | |
| 2016-07-20T15:38:05UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:38:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:38:05.631248-00 pid=3939 | |
| 2016-07-20T15:38:44UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:38:44UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:38:44UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:38:44UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:38:44UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:38:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:38:44UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:38:44UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:38:44UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:38:44UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:38:44UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:38:44UTC deis-database[1]: raise ex | |
| 2016-07-20T15:38:44UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:38:44UTC deis-database[1]: <Greenlet at 0x7f855c806230: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:38:44UTC deis-database[1]: | |
| 2016-07-20T15:38:53UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:38:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:38:53UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:38:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:38:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:38:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:38:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:38:53UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:38:53UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:38:53UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:38:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:38:53UTC deis-database[1]: raise ex | |
| 2016-07-20T15:38:53UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:38:53UTC deis-database[1]: <Greenlet at 0x7f855c806190: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:38:53UTC deis-database[1]: | |
| 2016-07-20T15:39:02UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:39:02UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:39:02UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:39:02UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:39:02UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:39:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:39:02UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:39:02UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:39:02UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:39:02UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:39:02UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:39:02UTC deis-database[1]: raise ex | |
| 2016-07-20T15:39:02UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:39:02UTC deis-database[1]: <Greenlet at 0x7f855ca6eaf0: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:39:02UTC deis-database[1]: | |
| 2016-07-20T15:39:11UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:39:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:39:11UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:39:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:39:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:39:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:39:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:39:11UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:39:11UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:39:11UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:39:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:39:11UTC deis-database[1]: raise ex | |
| 2016-07-20T15:39:11UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:39:11UTC deis-database[1]: <Greenlet at 0x7f855ca6eeb0: <wal_e.worker.upload.WalUploader object at 0x7f855c7d8450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:39:11UTC deis-database[1]: | |
| 2016-07-20T15:39:14UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:39:14UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:39:14UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:39:14UTC deis-database[1]: group.join() | |
| 2016-07-20T15:39:14UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:39:14UTC deis-database[1]: raise val | |
| 2016-07-20T15:39:14UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:39:14UTC deis-database[1]: | |
| 2016-07-20T15:39:14UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:14.834065-00 pid=2608 | |
| 2016-07-20T15:39:14UTC deis-database[1]: 2016-07-20 15:39:14 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:39:14UTC deis-database[1]: 2016-07-20 15:39:14 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.174055-00 pid=2748 | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.201868-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.347398-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.490148-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.640133-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.785058-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:39:16UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:16UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:39:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:16.921151-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:39:17UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:17UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:39:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:17.070456-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:39:17UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:39:17UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:39:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:39:17.208249-00 pid=2748 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:40:10UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:40:10UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:40:10UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:40:10UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:40:10UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:40:10UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:40:10UTC deis-database[1]: data=data) | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:10UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:10UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:10UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:10UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:10UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:10UTC deis-database[1]: | |
| 2016-07-20T15:40:10UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:40:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:10.985192-00 pid=3939 | |
| 2016-07-20T15:40:26UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:40:26UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:40:26UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:40:26UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:40:26UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:40:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:40:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:40:26UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:26UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:26UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:26UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:26UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:26UTC deis-database[1]: <Greenlet at 0x7fc45483f0f0: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:40:26UTC deis-database[1]: | |
| 2016-07-20T15:40:28UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:40:28UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:40:28UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:40:28UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:40:28UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:40:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:40:28UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:40:28UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:28UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:28UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:28UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:28UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:28UTC deis-database[1]: <Greenlet at 0x7fc45483f190: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:40:28UTC deis-database[1]: | |
| 2016-07-20T15:40:32UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:40:32UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:40:32UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:40:32UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:40:32UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:40:32UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:40:32UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:40:32UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:32UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:32UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:32UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:32UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:32UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:32UTC deis-database[1]: <Greenlet at 0x7fc45483f230: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:40:32UTC deis-database[1]: | |
| 2016-07-20T15:40:34UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:40:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:40:34UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:40:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:40:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:40:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:40:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:40:34UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:34UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:34UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:34UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:34UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:34UTC deis-database[1]: <Greenlet at 0x7fc45483f050: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:40:34UTC deis-database[1]: | |
| 2016-07-20T15:40:45UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:40:45UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:40:45UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:40:45UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:40:45UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:40:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:40:45UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:40:45UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:40:45UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:40:45UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:40:45UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:40:45UTC deis-database[1]: raise ex | |
| 2016-07-20T15:40:45UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:45UTC deis-database[1]: <Greenlet at 0x7fc454aa7f50: <wal_e.worker.upload.WalUploader object at 0x7fc454811450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:40:45UTC deis-database[1]: | |
| 2016-07-20T15:40:56UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:40:56UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:40:56UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:40:56UTC deis-database[1]: group.join() | |
| 2016-07-20T15:40:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:40:56UTC deis-database[1]: raise val | |
| 2016-07-20T15:40:56UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:40:56UTC deis-database[1]: | |
| 2016-07-20T15:40:56UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:56.734755-00 pid=2748 | |
| 2016-07-20T15:40:56UTC deis-database[1]: 2016-07-20 15:40:56 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:40:56UTC deis-database[1]: 2016-07-20 15:40:56 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.074102-00 pid=2888 | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.099153-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.238059-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.373391-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.511076-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.653949-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.788161-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:40:58UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:58UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:40:58UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:58.927898-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:40:59UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:40:59UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:40:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:40:59.067591-00 pid=2888 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:41:47UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:41:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:41:47UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:41:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:41:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:41:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:41:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:41:47UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:41:47UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:41:47UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:41:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:41:47UTC deis-database[1]: raise ex | |
| 2016-07-20T15:41:47UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:41:47UTC deis-database[1]: <Greenlet at 0x7f7ed76bab90: <wal_e.worker.upload.WalUploader object at 0x7f7ed7424450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:41:47UTC deis-database[1]: | |
| 2016-07-20T15:42:05UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:42:05UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:42:05UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:42:05UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:42:05UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:42:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:42:05UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:42:05UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:42:05UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:42:05UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:42:05UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:42:05UTC deis-database[1]: raise ex | |
| 2016-07-20T15:42:05UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:42:05UTC deis-database[1]: <Greenlet at 0x7f7ed76baeb0: <wal_e.worker.upload.WalUploader object at 0x7f7ed7424450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:42:05UTC deis-database[1]: | |
| 2016-07-20T15:42:17UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:42:17UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:42:17UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:42:17UTC deis-database[1]: group.join() | |
| 2016-07-20T15:42:17UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:42:17UTC deis-database[1]: raise val | |
| 2016-07-20T15:42:17UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:42:17UTC deis-database[1]: | |
| 2016-07-20T15:42:17UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:17.378498-00 pid=2888 | |
| 2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:42:17UTC deis-database[1]: 2016-07-20 15:42:17 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:42:25UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:42:25UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:42:25UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:42:25UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:42:25UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:42:25UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:42:25UTC deis-database[1]: data=data) | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:42:25UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:42:25UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:42:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:42:25UTC deis-database[1]: raise ex | |
| 2016-07-20T15:42:25UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:42:25UTC deis-database[1]: | |
| 2016-07-20T15:42:25UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:42:25UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:25.035232-00 pid=3939 | |
| 2016-07-20T15:42:37UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:42:37UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.616912-00 pid=3031 | |
| 2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.643983-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.788249-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:42:37UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:37UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:42:37UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:37.924596-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.076934-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.217048-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.353703-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.503778-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:42:38UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:42:38UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:42:38UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:42:38.646837-00 pid=3031 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:43:49UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:43:49UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:43:49UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:43:49UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:43:49UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:43:49UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:43:49UTC deis-database[1]: data=data) | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:43:49UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:43:49UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:43:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:43:49UTC deis-database[1]: raise ex | |
| 2016-07-20T15:43:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:43:49UTC deis-database[1]: | |
| 2016-07-20T15:43:49UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:43:49UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:43:49.953199-00 pid=3939 | |
| 2016-07-20T15:44:00UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:00UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:00UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:00UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:00UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:00UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:00UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:00UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:00UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:00UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:00UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:00UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:00UTC deis-database[1]: <Greenlet at 0x7fb46d3ef0f0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:00UTC deis-database[1]: | |
| 2016-07-20T15:44:06UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:06UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:06UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:06UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:06UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:06UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:06UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:06UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:06UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:06UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:06UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:06UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:06UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:06UTC deis-database[1]: <Greenlet at 0x7fb46d657af0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:06UTC deis-database[1]: | |
| 2016-07-20T15:44:11UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:11UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:11UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:11UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:11UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:11UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:11UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:11UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:11UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:11UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:11UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:11UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:11UTC deis-database[1]: <Greenlet at 0x7fb46d3ef050: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:11UTC deis-database[1]: | |
| 2016-07-20T15:44:19UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:19UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:19UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:19UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:19UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:19UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:19UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:19UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:19UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:19UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:19UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:19UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:19UTC deis-database[1]: <Greenlet at 0x7fb46d657eb0: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:19UTC deis-database[1]: | |
| 2016-07-20T15:44:26UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:26UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:26UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:26UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:26UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:26UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:26UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:26UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:26UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:26UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:26UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:26UTC deis-database[1]: <Greenlet at 0x7fb46d3ef230: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:26UTC deis-database[1]: | |
| 2016-07-20T15:44:29UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:44:29UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:44:29UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:44:29UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:44:29UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:44:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:44:29UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:44:29UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:29UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:29UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:29UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:29UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:29UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:29UTC deis-database[1]: <Greenlet at 0x7fb46d3ef190: <wal_e.worker.upload.WalUploader object at 0x7fb46d3c1450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:44:29UTC deis-database[1]: | |
| 2016-07-20T15:44:30UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:44:30UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:44:30UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:44:30UTC deis-database[1]: group.join() | |
| 2016-07-20T15:44:30UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:44:30UTC deis-database[1]: raise val | |
| 2016-07-20T15:44:30UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:30UTC deis-database[1]: | |
| 2016-07-20T15:44:30UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:30.496921-00 pid=3031 | |
| 2016-07-20T15:44:30UTC deis-database[1]: 2016-07-20 15:44:30 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:44:30UTC deis-database[1]: 2016-07-20 15:44:30 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:44:31UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:44:31UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:44:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:31.846497-00 pid=3185 | |
| 2016-07-20T15:44:31UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:31UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:44:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:31.872108-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.017260-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.170971-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.307232-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.442922-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.580884-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.724810-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:44:32UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:44:32UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:44:32UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:32.863568-00 pid=3185 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:44:43UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:44:43UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:44:43UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:44:43UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:44:43UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:44:43UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:44:43UTC deis-database[1]: data=data) | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:44:43UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:44:43UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:44:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:44:43UTC deis-database[1]: raise ex | |
| 2016-07-20T15:44:43UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:44:43UTC deis-database[1]: | |
| 2016-07-20T15:44:43UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:44:43UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:44:43.478962-00 pid=3939 | |
| 2016-07-20T15:45:46UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:45:46UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:45:46UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:45:46UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:45:46UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:45:46UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:45:46UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:45:46UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:45:46UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:45:46UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:45:46UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:45:46UTC deis-database[1]: raise ex | |
| 2016-07-20T15:45:46UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:45:46UTC deis-database[1]: <Greenlet at 0x7ff578292230: <wal_e.worker.upload.WalUploader object at 0x7ff578264450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:45:46UTC deis-database[1]: | |
| 2016-07-20T15:45:52UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:45:52UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:45:52UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:45:52UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:45:52UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:45:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:45:52UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:45:52UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:45:52UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:45:52UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:45:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:45:52UTC deis-database[1]: raise ex | |
| 2016-07-20T15:45:52UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:45:52UTC deis-database[1]: <Greenlet at 0x7ff5784faeb0: <wal_e.worker.upload.WalUploader object at 0x7ff578264450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:45:52UTC deis-database[1]: | |
| 2016-07-20T15:46:09UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:46:09UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:46:09UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:46:09UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:46:09UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:46:09UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:46:09UTC deis-database[1]: data=data) | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:46:09UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:46:09UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:46:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:46:09UTC deis-database[1]: raise ex | |
| 2016-07-20T15:46:09UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:46:09UTC deis-database[1]: | |
| 2016-07-20T15:46:09UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:46:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:09.137421-00 pid=3939 | |
| 2016-07-20T15:46:16UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:46:16UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:46:16UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:46:16UTC deis-database[1]: group.join() | |
| 2016-07-20T15:46:16UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:46:16UTC deis-database[1]: raise val | |
| 2016-07-20T15:46:16UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:46:16UTC deis-database[1]: | |
| 2016-07-20T15:46:16UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:16.763932-00 pid=3185 | |
| 2016-07-20T15:46:16UTC deis-database[1]: 2016-07-20 15:46:16 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:46:16UTC deis-database[1]: 2016-07-20 15:46:16 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.131062-00 pid=3336 | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.157612-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.298275-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.440470-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.605423-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.748424-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:46:18UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:18UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:46:18UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:18.895359-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:46:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:46:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:19.072569-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:46:19UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:46:19UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:46:19UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:46:19.215672-00 pid=3336 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:47:28UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:47:28UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:47:28UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:47:28UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:47:28UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:47:28UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:47:28UTC deis-database[1]: data=data) | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:28UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:28UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:28UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:28UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:28UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:28UTC deis-database[1]: | |
| 2016-07-20T15:47:28UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:47:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:47:28.152232-00 pid=3939 | |
| 2016-07-20T15:47:34UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:34UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:34UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:34UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:34UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:34UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:34UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:34UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:34UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:34UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:34UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:34UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:34UTC deis-database[1]: <Greenlet at 0x7fa6a44bf0f0: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:34UTC deis-database[1]: | |
| 2016-07-20T15:47:47UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:47UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:47UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:47UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:47UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:47UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:47UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:47UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:47UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:47UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:47UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:47UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:47UTC deis-database[1]: <Greenlet at 0x7fa6a4727b90: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:47UTC deis-database[1]: | |
| 2016-07-20T15:47:49UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:49UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:49UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:49UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:49UTC deis-database[1]: <Greenlet at 0x7fa6a44bf190: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:49UTC deis-database[1]: | |
| 2016-07-20T15:47:49UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:49UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:49UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:49UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:49UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:49UTC deis-database[1]: <Greenlet at 0x7fa6a4727af0: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:49UTC deis-database[1]: | |
| 2016-07-20T15:47:53UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:53UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:53UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:53UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:53UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:53UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:53UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:53UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:53UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:53UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:53UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:53UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:53UTC deis-database[1]: <Greenlet at 0x7fa6a44bf050: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:53UTC deis-database[1]: | |
| 2016-07-20T15:47:55UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:47:55UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:47:55UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:47:55UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:47:55UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:47:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:47:55UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:47:55UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:47:55UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:47:55UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:47:55UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:47:55UTC deis-database[1]: raise ex | |
| 2016-07-20T15:47:55UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:47:55UTC deis-database[1]: <Greenlet at 0x7fa6a44bf230: <wal_e.worker.upload.WalUploader object at 0x7fa6a4491450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:47:55UTC deis-database[1]: | |
| 2016-07-20T15:48:04UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:48:04UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:48:04UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:48:04UTC deis-database[1]: group.join() | |
| 2016-07-20T15:48:04UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:48:04UTC deis-database[1]: raise val | |
| 2016-07-20T15:48:04UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:48:04UTC deis-database[1]: | |
| 2016-07-20T15:48:04UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:04.872569-00 pid=3336 | |
| 2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:48:04UTC deis-database[1]: 2016-07-20 15:48:04 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.225375-00 pid=3477 | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.250716-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.393853-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.542910-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.687694-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.823241-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:48:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:05UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:48:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:05.960127-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:48:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:06UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:48:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:06.099615-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:48:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:48:06UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:48:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:06.249689-00 pid=3477 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:48:31UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:48:31UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:48:31UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:48:31UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:48:31UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:48:31UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:48:31UTC deis-database[1]: data=data) | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:48:31UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:48:31UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:48:31UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:48:31UTC deis-database[1]: raise ex | |
| 2016-07-20T15:48:31UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:48:31UTC deis-database[1]: | |
| 2016-07-20T15:48:31UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:48:31UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:48:31.333767-00 pid=3939 | |
| 2016-07-20T15:49:22UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:49:22UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:49:22UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:49:22UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:49:22UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:49:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:49:22UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:49:22UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:49:22UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:49:22UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:49:22UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:49:22UTC deis-database[1]: raise ex | |
| 2016-07-20T15:49:22UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:49:22UTC deis-database[1]: <Greenlet at 0x7f5a8b709230: <wal_e.worker.upload.WalUploader object at 0x7f5a8b6db450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:49:22UTC deis-database[1]: | |
| 2016-07-20T15:49:51UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:49:51UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:49:51UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:49:51UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:49:51UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:49:51UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:49:51UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:49:51UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:49:51UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:49:51UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:49:51UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:49:51UTC deis-database[1]: raise ex | |
| 2016-07-20T15:49:51UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:49:51UTC deis-database[1]: <Greenlet at 0x7f5a8b971b90: <wal_e.worker.upload.WalUploader object at 0x7f5a8b6db450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:49:51UTC deis-database[1]: | |
| 2016-07-20T15:49:52UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:49:52UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:49:52UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:49:52UTC deis-database[1]: group.join() | |
| 2016-07-20T15:49:52UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:49:52UTC deis-database[1]: raise val | |
| 2016-07-20T15:49:52UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:49:52UTC deis-database[1]: | |
| 2016-07-20T15:49:52UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:52.259243-00 pid=3477 | |
| 2016-07-20T15:49:52UTC deis-database[1]: 2016-07-20 15:49:52 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:49:52UTC deis-database[1]: 2016-07-20 15:49:52 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:49:53UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:49:53UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.614013-00 pid=3630 | |
| 2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.639713-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.781682-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:49:53UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:53UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:49:53UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:53.927603-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.069398-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.222308-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.369621-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.519578-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:49:54UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:49:54UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:49:54UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:49:54.671798-00 pid=3630 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:50:27UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:50:27UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:50:27UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:50:27UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:50:27UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:50:27UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:50:27UTC deis-database[1]: data=data) | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:50:27UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:50:27UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:50:27UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:50:27UTC deis-database[1]: raise ex | |
| 2016-07-20T15:50:27UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:50:27UTC deis-database[1]: | |
| 2016-07-20T15:50:27UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:50:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:50:27.925245-00 pid=3939 | |
| 2016-07-20T15:50:56UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:50:56UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:50:56UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:50:56UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:50:56UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:50:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:50:56UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:50:56UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:50:56UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:50:56UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:50:56UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:50:56UTC deis-database[1]: raise ex | |
| 2016-07-20T15:50:56UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:50:56UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebaf0: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:50:56UTC deis-database[1]: | |
| 2016-07-20T15:51:13UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:51:13UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:51:13UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:51:13UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:51:13UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:51:13UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:51:13UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:51:13UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:51:13UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:51:13UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:51:13UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:51:13UTC deis-database[1]: raise ex | |
| 2016-07-20T15:51:13UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:51:13UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebeb0: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:51:13UTC deis-database[1]: | |
| 2016-07-20T15:51:20UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:51:20UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:51:20UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:51:20UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:51:20UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:51:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:51:20UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:51:20UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:51:20UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:51:20UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:51:20UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:51:20UTC deis-database[1]: raise ex | |
| 2016-07-20T15:51:20UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:51:20UTC deis-database[1]: <Greenlet at 0x7f9bfb483050: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:51:20UTC deis-database[1]: | |
| 2016-07-20T15:51:25UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:51:25UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:51:25UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:51:25UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:51:25UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:51:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:51:25UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:51:25UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:51:25UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:51:25UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:51:25UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:51:25UTC deis-database[1]: raise ex | |
| 2016-07-20T15:51:25UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:51:25UTC deis-database[1]: <Greenlet at 0x7f9bfb6ebf50: <wal_e.worker.upload.WalUploader object at 0x7f9bfb455450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:51:25UTC deis-database[1]: | |
| 2016-07-20T15:51:26UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:51:26UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:51:26UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:51:26UTC deis-database[1]: group.join() | |
| 2016-07-20T15:51:26UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:51:26UTC deis-database[1]: raise val | |
| 2016-07-20T15:51:26UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:51:26UTC deis-database[1]: | |
| 2016-07-20T15:51:26UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:26.385369-00 pid=3630 | |
| 2016-07-20T15:51:26UTC deis-database[1]: 2016-07-20 15:51:26 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:51:26UTC deis-database[1]: 2016-07-20 15:51:26 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:51:27UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:51:27UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.770945-00 pid=3761 | |
| 2016-07-20T15:51:27UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:27UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.805291-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:51:27UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:27UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:51:27UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:27.947909-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.085938-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.222462-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.360340-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.506321-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.649026-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:51:28UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:51:28UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:51:28UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:28.795619-00 pid=3761 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T15:51:59UTC deis-database[1]: wal_e.retries WARNING MSG: retrying after encountering exception | |
| 2016-07-20T15:51:59UTC deis-database[1]: DETAIL: Exception information dump: | |
| 2016-07-20T15:51:59UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/retries.py", line 62, in shim | |
| 2016-07-20T15:51:59UTC deis-database[1]: return f(*args, **kwargs) | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/s3/s3_deleter.py", line 27, in _delete_batch | |
| 2016-07-20T15:51:59UTC deis-database[1]: bucket.delete_keys([key.name for key in page]) | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 727, in delete_keys | |
| 2016-07-20T15:51:59UTC deis-database[1]: while delete_keys2(headers): | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 715, in delete_keys2 | |
| 2016-07-20T15:51:59UTC deis-database[1]: data=data) | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:51:59UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:51:59UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:51:59UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:51:59UTC deis-database[1]: raise ex | |
| 2016-07-20T15:51:59UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:51:59UTC deis-database[1]: | |
| 2016-07-20T15:51:59UTC deis-database[1]: HINT: A better error message should be written to handle this exception. Please report this output and, if possible, the situation under which it arises. | |
| 2016-07-20T15:51:59UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:51:59.294902-00 pid=3939 | |
| 2016-07-20T15:52:38UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:52:38UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:52:38UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:52:38UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:52:38UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:52:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:52:38UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:52:38UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:52:38UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:52:38UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:52:38UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:52:38UTC deis-database[1]: raise ex | |
| 2016-07-20T15:52:38UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:52:38UTC deis-database[1]: <Greenlet at 0x7f76f9cd4050: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:52:38UTC deis-database[1]: | |
| 2016-07-20T15:52:42UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:52:42UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:52:42UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:52:42UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:52:42UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:52:42UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:52:42UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:52:42UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:52:42UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:52:42UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:52:42UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:52:42UTC deis-database[1]: raise ex | |
| 2016-07-20T15:52:42UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:52:42UTC deis-database[1]: <Greenlet at 0x7f76f9f3cf50: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:52:42UTC deis-database[1]: | |
| 2016-07-20T15:52:43UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:52:43UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:52:43UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:52:43UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:52:43UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:52:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:52:43UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:52:43UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:52:43UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:52:43UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:52:43UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:52:43UTC deis-database[1]: raise ex | |
| 2016-07-20T15:52:43UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:52:43UTC deis-database[1]: <Greenlet at 0x7f76f9f3cb90: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:52:43UTC deis-database[1]: | |
| 2016-07-20T15:52:49UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:52:49UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:52:49UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:52:49UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:52:49UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:52:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:52:49UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:52:49UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:52:49UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:52:49UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:52:49UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:52:49UTC deis-database[1]: raise ex | |
| 2016-07-20T15:52:49UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:52:49UTC deis-database[1]: <Greenlet at 0x7f76f9cd40f0: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:52:49UTC deis-database[1]: | |
| 2016-07-20T15:52:54UTC deis-database[1]: Traceback (most recent call last): | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run | |
| 2016-07-20T15:52:54UTC deis-database[1]: result = self._run(*self.args, **self.kwargs) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/upload.py", line 50, in __call__ | |
| 2016-07-20T15:52:54UTC deis-database[1]: self.gpg_key_id) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/worker_util.py", line 40, in do_lzop_put | |
| 2016-07-20T15:52:54UTC deis-database[1]: k = blobstore.uri_put_file(creds, url, tf) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/blobstore/s3/s3_util.py", line 55, in uri_put_file | |
| 2016-07-20T15:52:54UTC deis-database[1]: k.set_contents_from_file(fp, encrypt_key=True) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file | |
| 2016-07-20T15:52:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file | |
| 2016-07-20T15:52:54UTC deis-database[1]: chunked_transfer=chunked_transfer, size=size) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal | |
| 2016-07-20T15:52:54UTC deis-database[1]: query_args=query_args | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request | |
| 2016-07-20T15:52:54UTC deis-database[1]: retry_handler=retry_handler | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request | |
| 2016-07-20T15:52:54UTC deis-database[1]: retry_handler=retry_handler) | |
| 2016-07-20T15:52:54UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe | |
| 2016-07-20T15:52:54UTC deis-database[1]: raise ex | |
| 2016-07-20T15:52:54UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:52:54UTC deis-database[1]: <Greenlet at 0x7f76f9cd4190: <wal_e.worker.upload.WalUploader object at 0x7f76f9ca6450>(<wal_e.worker.pg.wal_transfer.WalSegment object at)> failed with timeout | |
| 2016-07-20T15:52:54UTC deis-database[1]: | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.main CRITICAL MSG: An unprocessed exception has avoided all error handling | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Traceback (most recent call last): | |
| 2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/cmd.py", line 586, in main | |
| 2016-07-20T15:53:09UTC deis-database[1]: concurrency=args.pool_size) | |
| 2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/operator/backup.py", line 285, in wal_archive | |
| 2016-07-20T15:53:09UTC deis-database[1]: group.join() | |
| 2016-07-20T15:53:09UTC deis-database[1]: File "/usr/lib/python2.7/site-packages/wal_e/worker/pg/wal_transfer.py", line 144, in join | |
| 2016-07-20T15:53:09UTC deis-database[1]: raise val | |
| 2016-07-20T15:53:09UTC deis-database[1]: timeout: timed out | |
| 2016-07-20T15:53:09UTC deis-database[1]: | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.000469-00 pid=3761 | |
| 2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC LOG: archive command failed with exit code 2 | |
| 2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC DETAIL: The failed archive command was: envdir /etc/wal-e.d/env wal-e wal-push pg_xlog/000000010000000200000034 | |
| 2016-07-20T15:53:09UTC deis-database[1]: 2016-07-20 15:53:09 UTC WARNING: archiving transaction log file "000000010000000200000034" failed too many times, will try again later | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.357210-00 pid=3922 | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000034" to "s3://db_wal/wal_005/000000010000000200000034.lzo". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.382696-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000034.lzo prefix= seg=000000010000000200000034 state=begin | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000035" to "s3://db_wal/wal_005/000000010000000200000035.lzo". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.529969-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000035.lzo prefix= seg=000000010000000200000035 state=begin | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000037" to "s3://db_wal/wal_005/000000010000000200000037.lzo". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.699376-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000037.lzo prefix= seg=000000010000000200000037 state=begin | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000038" to "s3://db_wal/wal_005/000000010000000200000038.lzo". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.851905-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000038.lzo prefix= seg=000000010000000200000038 state=begin | |
| 2016-07-20T15:53:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000200000039" to "s3://db_wal/wal_005/000000010000000200000039.lzo". | |
| 2016-07-20T15:53:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:09.997291-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/000000010000000200000039.lzo prefix= seg=000000010000000200000039 state=begin | |
| 2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003A" to "s3://db_wal/wal_005/00000001000000020000003A.lzo". | |
| 2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.142817-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003A.lzo prefix= seg=00000001000000020000003A state=begin | |
| 2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003B" to "s3://db_wal/wal_005/00000001000000020000003B.lzo". | |
| 2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.287335-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003B.lzo prefix= seg=00000001000000020000003B state=begin | |
| 2016-07-20T15:53:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T15:53:10UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000020000003C" to "s3://db_wal/wal_005/00000001000000020000003C.lzo". | |
| 2016-07-20T15:53:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T15:53:10.428345-00 pid=3922 action=push-wal key=s3://db_wal/wal_005/00000001000000020000003C.lzo prefix= seg=00000001000000020000003C state=begin | |
| 2016-07-20T16:05:44UTC deis-database[1]: database: waiting for confd to write initial templates... | |
| 2016-07-20T16:05:54UTC deis-database[1]: The files belonging to this database system will be owned by user "postgres". | |
| 2016-07-20T16:05:54UTC deis-database[1]: This user must also own the server process. | |
| 2016-07-20T16:05:54UTC deis-database[1]: | |
| 2016-07-20T16:05:54UTC deis-database[1]: The database cluster will be initialized with locales | |
| 2016-07-20T16:05:54UTC deis-database[1]: COLLATE: C | |
| 2016-07-20T16:05:54UTC deis-database[1]: CTYPE: C.UTF-8 | |
| 2016-07-20T16:05:54UTC deis-database[1]: MESSAGES: C | |
| 2016-07-20T16:05:54UTC deis-database[1]: MONETARY: C | |
| 2016-07-20T16:05:54UTC deis-database[1]: NUMERIC: C | |
| 2016-07-20T16:05:54UTC deis-database[1]: TIME: C | |
| 2016-07-20T16:05:54UTC deis-database[1]: The default database encoding has accordingly been set to "UTF8". | |
| 2016-07-20T16:05:54UTC deis-database[1]: The default text search configuration will be set to "english". | |
| 2016-07-20T16:05:54UTC deis-database[1]: | |
| 2016-07-20T16:05:54UTC deis-database[1]: Data page checksums are disabled. | |
| 2016-07-20T16:05:54UTC deis-database[1]: | |
| 2016-07-20T16:05:54UTC deis-database[1]: creating directory /var/lib/postgresql/9.3/main ... ok | |
| 2016-07-20T16:05:54UTC deis-database[1]: creating subdirectories ... ok | |
| 2016-07-20T16:05:54UTC deis-database[1]: selecting default max_connections ... 100 | |
| 2016-07-20T16:05:54UTC deis-database[1]: selecting default shared_buffers ... 128MB | |
| 2016-07-20T16:05:54UTC deis-database[1]: creating configuration files ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating template1 database in /var/lib/postgresql/9.3/main/base/1 ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: initializing pg_authid ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: initializing dependencies ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating system views ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: loading system objects' descriptions ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: sh: locale: not found | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating collations ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: No usable system locales were found. | |
| 2016-07-20T16:05:55UTC deis-database[1]: Use the option "--debug" to see details. | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating conversions ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating dictionaries ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: setting privileges on built-in objects ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: creating information schema ... ok | |
| 2016-07-20T16:05:55UTC deis-database[1]: loading PL/pgSQL server-side language ... ok | |
| 2016-07-20T16:05:56UTC deis-database[1]: vacuuming database template1 ... ok | |
| 2016-07-20T16:05:56UTC deis-database[1]: copying template1 to template0 ... ok | |
| 2016-07-20T16:05:56UTC deis-database[1]: copying template1 to postgres ... ok | |
| 2016-07-20T16:06:00UTC deis-database[1]: | |
| 2016-07-20T16:06:00UTC deis-database[1]: WARNING: enabling "trust" authentication for local connections | |
| 2016-07-20T16:06:00UTC deis-database[1]: You can change this by editing pg_hba.conf or using the option -A, or | |
| 2016-07-20T16:06:00UTC deis-database[1]: --auth-local and --auth-host, the next time you run initdb. | |
| 2016-07-20T16:06:00UTC deis-database[1]: syncing data to disk ... ok | |
| 2016-07-20T16:06:00UTC deis-database[1]: | |
| 2016-07-20T16:06:00UTC deis-database[1]: Success. | |
| 2016-07-20T16:06:00UTC deis-database[1]: | |
| 2016-07-20T16:06:01UTC deis-database[1]: database: expecting initialization id: e45a7062-84a0-4dd5-978b-f471e0d88138 | |
| 2016-07-20T16:06:01UTC deis-database[1]: database: no existing database found or it is outdated. | |
| 2016-07-20T16:06:01UTC deis-database[1]: database: no backups found. Initializing a new database... | |
| 2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: could not open usermap file "/etc/postgresql/main/pg_ident.conf": No such file or directory | |
| 2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: database system was shut down at 2016-07-20 16:05:56 UTC | |
| 2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: MultiXact member wraparound protections are now enabled | |
| 2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: database system is ready to accept connections | |
| 2016-07-20T16:06:02UTC deis-database[1]: 2016-07-20 16:06:02 UTC LOG: autovacuum launcher started | |
| 2016-07-20T16:06:02UTC deis-database[1]: ALTER ROLE | |
| 2016-07-20T16:06:02UTC deis-database[1]: CREATE ROLE | |
| 2016-07-20T16:06:02UTC deis-database[1]: ALTER ROLE | |
| 2016-07-20T16:06:03UTC deis-database[1]: CREATE DATABASE | |
| 2016-07-20T16:06:03UTC deis-database[1]: database: performing an initial backup... | |
| 2016-07-20T16:06:03UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:06:03UTC deis-database[1]: DETAIL: The subcommand is "backup-push". | |
| 2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.624440-00 pid=167 | |
| 2016-07-20T16:06:03UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:06:03UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.973826-00 pid=177 | |
| 2016-07-20T16:06:03UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:06:03UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000001" to "s3://db_wal/wal_005/000000010000000000000001.lzo". | |
| 2016-07-20T16:06:03UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:03.997350-00 pid=177 action=push-wal key=s3://db_wal/wal_005/000000010000000000000001.lzo prefix= seg=000000010000000000000001 state=begin | |
| 2016-07-20T16:06:04UTC deis-database[1]: wal_e.operator.backup INFO MSG: start upload postgres version metadata | |
| 2016-07-20T16:06:04UTC deis-database[1]: DETAIL: Uploading to s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/extended_version.txt. | |
| 2016-07-20T16:06:04UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:04.412497-00 pid=167 | |
| 2016-07-20T16:06:05UTC deis-database[1]: wal_e.operator.backup INFO MSG: postgres version metadata upload complete | |
| 2016-07-20T16:06:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:05.364806-00 pid=167 | |
| 2016-07-20T16:06:05UTC deis-database[1]: wal_e.worker.upload INFO MSG: beginning volume compression | |
| 2016-07-20T16:06:05UTC deis-database[1]: DETAIL: Building volume 0. | |
| 2016-07-20T16:06:05UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:05.407180-00 pid=167 | |
| 2016-07-20T16:06:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin uploading a base backup volume | |
| 2016-07-20T16:06:06UTC deis-database[1]: DETAIL: Uploading to "s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo". | |
| 2016-07-20T16:06:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:06.174663-00 pid=167 | |
| 2016-07-20T16:06:06UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:06:06UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000001.lzo" complete at 1437.06KiB/s. | |
| 2016-07-20T16:06:06UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:06.416166-00 pid=177 action=push-wal key=s3://db_wal/wal_005/000000010000000000000001.lzo prefix= rate=1437.06 seg=000000010000000000000001 state=complete | |
| 2016-07-20T16:06:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: finish uploading a base backup volume | |
| 2016-07-20T16:06:07UTC deis-database[1]: DETAIL: Uploading to "s3://db_wal/basebackups_005/base_000000010000000000000002_00000040/tar_partitions/part_00000000.tar.lzo" complete at 3564.64KiB/s. | |
| 2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.370915-00 pid=167 | |
| 2016-07-20T16:06:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:06:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.843843-00 pid=191 | |
| 2016-07-20T16:06:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:06:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000002" to "s3://db_wal/wal_005/000000010000000000000002.lzo". | |
| 2016-07-20T16:06:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:07.866923-00 pid=191 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.lzo prefix= seg=000000010000000000000002 state=begin | |
| 2016-07-20T16:06:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:06:08UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000002.lzo" complete at 439.525KiB/s. | |
| 2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.354917-00 pid=191 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.lzo prefix= rate=439.525 seg=000000010000000000000002 state=complete | |
| 2016-07-20T16:06:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:06:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.664902-00 pid=198 | |
| 2016-07-20T16:06:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:06:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000002.00000028.backup" to "s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo". | |
| 2016-07-20T16:06:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:08.687202-00 pid=198 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo prefix= seg=000000010000000000000002.00000028.backup state=begin | |
| 2016-07-20T16:06:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:06:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo" complete at 00KiB/s. | |
| 2016-07-20T16:06:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:06:09.105333-00 pid=198 action=push-wal key=s3://db_wal/wal_005/000000010000000000000002.00000028.backup.lzo prefix= rate=00 seg=000000010000000000000002.00000028.backup state=complete | |
| 2016-07-20T16:06:09UTC deis-database[1]: NOTICE: pg_stop_backup complete, all required WAL segments have been archived | |
| 2016-07-20T16:06:09UTC deis-database[1]: database: postgres is running... | |
| 2016-07-20T16:07:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:07:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:07:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:08.936854-00 pid=284 | |
| 2016-07-20T16:07:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:07:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000003" to "s3://db_wal/wal_005/000000010000000000000003.lzo". | |
| 2016-07-20T16:07:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:08.957889-00 pid=284 action=push-wal key=s3://db_wal/wal_005/000000010000000000000003.lzo prefix= seg=000000010000000000000003 state=begin | |
| 2016-07-20T16:07:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:07:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000003.lzo" complete at 492.032KiB/s. | |
| 2016-07-20T16:07:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:07:09.657617-00 pid=284 action=push-wal key=s3://db_wal/wal_005/000000010000000000000003.lzo prefix= rate=492.032 seg=000000010000000000000003 state=complete | |
| 2016-07-20T16:08:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:08:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:08:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:08.217418-00 pid=369 | |
| 2016-07-20T16:08:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:08:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000004" to "s3://db_wal/wal_005/000000010000000000000004.lzo". | |
| 2016-07-20T16:08:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:08.275400-00 pid=369 action=push-wal key=s3://db_wal/wal_005/000000010000000000000004.lzo prefix= seg=000000010000000000000004 state=begin | |
| 2016-07-20T16:08:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:08:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000004.lzo" complete at 478.695KiB/s. | |
| 2016-07-20T16:08:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:08:09.799474-00 pid=369 action=push-wal key=s3://db_wal/wal_005/000000010000000000000004.lzo prefix= rate=478.695 seg=000000010000000000000004 state=complete | |
| 2016-07-20T16:09:07UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:09:07UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:09:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:07.625836-00 pid=441 | |
| 2016-07-20T16:09:07UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:09:07UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000005" to "s3://db_wal/wal_005/000000010000000000000005.lzo". | |
| 2016-07-20T16:09:07UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:07.647171-00 pid=441 action=push-wal key=s3://db_wal/wal_005/000000010000000000000005.lzo prefix= seg=000000010000000000000005 state=begin | |
| 2016-07-20T16:09:10UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:09:10UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000005.lzo" complete at 40.4188KiB/s. | |
| 2016-07-20T16:09:10UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:09:10.259547-00 pid=441 action=push-wal key=s3://db_wal/wal_005/000000010000000000000005.lzo prefix= rate=40.4188 seg=000000010000000000000005 state=complete | |
| 2016-07-20T16:14:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:14:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:14:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:08.500846-00 pid=807 | |
| 2016-07-20T16:14:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:14:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000006" to "s3://db_wal/wal_005/000000010000000000000006.lzo". | |
| 2016-07-20T16:14:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:08.590810-00 pid=807 action=push-wal key=s3://db_wal/wal_005/000000010000000000000006.lzo prefix= seg=000000010000000000000006 state=begin | |
| 2016-07-20T16:14:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:14:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000006.lzo" complete at 9505.92KiB/s. | |
| 2016-07-20T16:14:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:14:09.768045-00 pid=807 action=push-wal key=s3://db_wal/wal_005/000000010000000000000006.lzo prefix= rate=9505.92 seg=000000010000000000000006 state=complete | |
| 2016-07-20T16:18:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:18:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:18:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:08.304874-00 pid=1111 | |
| 2016-07-20T16:18:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:18:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000007" to "s3://db_wal/wal_005/000000010000000000000007.lzo". | |
| 2016-07-20T16:18:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:08.335939-00 pid=1111 action=push-wal key=s3://db_wal/wal_005/000000010000000000000007.lzo prefix= seg=000000010000000000000007 state=begin | |
| 2016-07-20T16:18:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:18:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000007.lzo" complete at 1149.33KiB/s. | |
| 2016-07-20T16:18:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:18:09.227802-00 pid=1111 action=push-wal key=s3://db_wal/wal_005/000000010000000000000007.lzo prefix= rate=1149.33 seg=000000010000000000000007 state=complete | |
| 2016-07-20T16:23:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:23:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:23:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:08.483431-00 pid=1489 | |
| 2016-07-20T16:23:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:23:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000008" to "s3://db_wal/wal_005/000000010000000000000008.lzo". | |
| 2016-07-20T16:23:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:08.514281-00 pid=1489 action=push-wal key=s3://db_wal/wal_005/000000010000000000000008.lzo prefix= seg=000000010000000000000008 state=begin | |
| 2016-07-20T16:23:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:23:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000008.lzo" complete at 523.868KiB/s. | |
| 2016-07-20T16:23:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:23:09.107577-00 pid=1489 action=push-wal key=s3://db_wal/wal_005/000000010000000000000008.lzo prefix= rate=523.868 seg=000000010000000000000008 state=complete | |
| 2016-07-20T16:28:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:28:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:28:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:08.506632-00 pid=1855 | |
| 2016-07-20T16:28:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:28:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/000000010000000000000009" to "s3://db_wal/wal_005/000000010000000000000009.lzo". | |
| 2016-07-20T16:28:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:08.529235-00 pid=1855 action=push-wal key=s3://db_wal/wal_005/000000010000000000000009.lzo prefix= seg=000000010000000000000009 state=begin | |
| 2016-07-20T16:28:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:28:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/000000010000000000000009.lzo" complete at 2422.68KiB/s. | |
| 2016-07-20T16:28:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:28:09.389993-00 pid=1855 action=push-wal key=s3://db_wal/wal_005/000000010000000000000009.lzo prefix= rate=2422.68 seg=000000010000000000000009 state=complete | |
| 2016-07-20T16:33:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:33:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:33:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:08.666435-00 pid=2233 | |
| 2016-07-20T16:33:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:33:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000A" to "s3://db_wal/wal_005/00000001000000000000000A.lzo". | |
| 2016-07-20T16:33:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:08.688055-00 pid=2233 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000A.lzo prefix= seg=00000001000000000000000A state=begin | |
| 2016-07-20T16:33:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:33:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000A.lzo" complete at 598.405KiB/s. | |
| 2016-07-20T16:33:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:33:09.202731-00 pid=2233 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000A.lzo prefix= rate=598.405 seg=00000001000000000000000A state=complete | |
| 2016-07-20T16:38:08UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:38:08UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:38:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:08.878056-00 pid=2598 | |
| 2016-07-20T16:38:08UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:38:08UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000B" to "s3://db_wal/wal_005/00000001000000000000000B.lzo". | |
| 2016-07-20T16:38:08UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:08.914100-00 pid=2598 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000B.lzo prefix= seg=00000001000000000000000B state=begin | |
| 2016-07-20T16:38:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:38:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000B.lzo" complete at 6696.11KiB/s. | |
| 2016-07-20T16:38:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:38:09.900464-00 pid=2598 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000B.lzo prefix= rate=6696.11 seg=00000001000000000000000B state=complete | |
| 2016-07-20T16:43:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:43:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.005446-00 pid=2976 | |
| 2016-07-20T16:43:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:43:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000C" to "s3://db_wal/wal_005/00000001000000000000000C.lzo". | |
| 2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.033349-00 pid=2976 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000C.lzo prefix= seg=00000001000000000000000C state=begin | |
| 2016-07-20T16:43:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:43:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000C.lzo" complete at 3187.16KiB/s. | |
| 2016-07-20T16:43:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:43:09.491392-00 pid=2976 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000C.lzo prefix= rate=3187.16 seg=00000001000000000000000C state=complete | |
| 2016-07-20T16:48:09UTC deis-database[1]: wal_e.main INFO MSG: starting WAL-E | |
| 2016-07-20T16:48:09UTC deis-database[1]: DETAIL: The subcommand is "wal-push". | |
| 2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.132989-00 pid=3354 | |
| 2016-07-20T16:48:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: begin archiving a file | |
| 2016-07-20T16:48:09UTC deis-database[1]: DETAIL: Uploading "pg_xlog/00000001000000000000000D" to "s3://db_wal/wal_005/00000001000000000000000D.lzo". | |
| 2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.160614-00 pid=3354 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000D.lzo prefix= seg=00000001000000000000000D state=begin | |
| 2016-07-20T16:48:09UTC deis-database[1]: wal_e.worker.upload INFO MSG: completed archiving to a file | |
| 2016-07-20T16:48:09UTC deis-database[1]: DETAIL: Archiving to "s3://db_wal/wal_005/00000001000000000000000D.lzo" complete at 1523.32KiB/s. | |
| 2016-07-20T16:48:09UTC deis-database[1]: STRUCTURED: time=2016-07-20T16:48:09.465534-00 pid=3354 action=push-wal key=s3://db_wal/wal_005/00000001000000000000000D.lzo prefix= rate=1523.32 seg=00000001000000000000000D state=complete |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment