logs-dead-replica.log

$ docker logs peter

ttl: &ttl 30
loop_wait: &loop_wait 10
scope: &scope bI508YaO
restapi:
  listen: 127.0.0.1:8008
  connect_address: 127.0.0.1:8008
etcd:
  scope: *scope
  ttl: *ttl
  host: 172.17.0.11:4001
postgresql:
  name: postgresql_172_17_0_12 ## Replication slots do not allow dots in their name
  scope: *scope
  listen: 0.0.0.0:5432
  connect_address: 172.17.0.12:5432
  data_dir: data/postgresql0
  maximum_lag_on_failover: 1048576 # 1 megabyte in bytes
  pg_hba:
  - host all all 0.0.0.0/0 md5
  - hostssl all all 0.0.0.0/0 md5
  - host replication replicator 172.17.0.12/16    md5
  replication:
    username: replicator
    password: rep-pass
    network:  127.0.0.1/32
  superuser:
    password: zalando
  restore: patroni/scripts/restore.py
  admin:
    username: admin
    password: admin
  parameters:
    archive_mode: "on"
    wal_level: hot_standby
    archive_command: mkdir -p ../wal_archive && cp %p ../wal_archive/%f
    max_wal_senders: 20
    listen_addresses: 0.0.0.0
    wal_keep_segments: 8
    archive_timeout: 1800s
    max_replication_slots: 20
    hot_standby: "on"
2015-10-20 13:24:28,202 INFO: Starting new HTTP connection (1): 172.17.0.11
2015-10-20 13:24:28,936 INFO: waiting for leader to bootstrap
2015-10-20 13:24:38,938 INFO: trying to bootstrap from leader
NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
waiting for server to start....LOG:  database system was interrupted; last known up at 2015-10-20 13:24:40 UTC
FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.LOG:  entering standby mode
LOG:  started streaming WAL from primary at 0/2000000 on timeline 1
FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.LOG:  redo starts at 0/2000028
FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.FATAL:  the database system is starting up
.LOG:  consistent recovery state reached at 0/30000F0
LOG:  database system is ready to accept read only connections
 done
server started
2015-10-20 13:24:58,747 INFO: bootstrapped from leader
2015-10-20 13:24:58,773 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:24:58,773 INFO: does not have lock
2015-10-20 13:24:58,774 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:00,557 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:00,558 INFO: does not have lock
2015-10-20 13:25:00,559 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:09,275 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:09,275 INFO: does not have lock
2015-10-20 13:25:09,276 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:19,452 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:19,453 INFO: does not have lock
2015-10-20 13:25:19,454 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:28,973 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:28,974 INFO: does not have lock
2015-10-20 13:25:28,975 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:38,976 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:38,977 INFO: does not have lock
2015-10-20 13:25:38,978 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:48,974 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:48,975 INFO: does not have lock
2015-10-20 13:25:48,975 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:25:58,969 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:25:58,970 INFO: does not have lock
2015-10-20 13:25:58,971 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:08,973 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:08,974 INFO: does not have lock
2015-10-20 13:26:08,975 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:18,975 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:18,976 INFO: does not have lock
2015-10-20 13:26:18,977 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:28,970 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:28,970 INFO: does not have lock
2015-10-20 13:26:28,971 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:38,968 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:38,969 INFO: does not have lock
2015-10-20 13:26:38,970 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:48,961 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:48,962 INFO: does not have lock
2015-10-20 13:26:48,963 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:26:58,961 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:26:58,961 INFO: does not have lock
2015-10-20 13:26:58,962 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:27:08,970 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:27:08,970 INFO: does not have lock
2015-10-20 13:27:08,971 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:27:18,983 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:27:18,983 INFO: does not have lock
2015-10-20 13:27:18,984 INFO: no action.  i am a secondary and i am following a leader
2015-10-20 13:27:28,970 INFO: Lock owner: postgresql_172_17_0_13; I am postgresql_172_17_0_12
2015-10-20 13:27:28,971 INFO: does not have lock
2015-10-20 13:27:28,972 INFO: no action.  i am a secondary and i am following a leader
LOG:  replication terminated by primary server
DETAIL:  End of WAL reached on timeline 1 at 0/8000090.
FATAL:  could not send end-of-streaming message to primary: no COPY in progress

LOG:  record with zero length at 0/8000090
2015-10-20 13:27:39,453 ERROR: Request to server http://172.17.0.11:4001 failed: timeout('timed out',)
2015-10-20 13:27:39,453 INFO: Reconnection allowed, looking for another server.
2015-10-20 13:27:39,454 INFO: Selected new etcd server
waiting for server to shut down...LOG:  received fast shutdown request
.LOG:  aborting any active transactions
FATAL:  terminating connection due to administrator command
LOG:  shutting down
.LOG:  database system is shut down
 done
server stopped
2015-10-20 13:27:41,651 ERROR: Unexpected request failure, re-raising.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 747, in api_execute
    preload_content=False)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 75, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 88, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 145, in urlopen
    conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 119, in connection_from_host
    pool = self._new_pool(scheme, host, port)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 86, in _new_pool
    return pool_cls(host, port, **kwargs)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 232, in __init__
    ConnectionPool.__init__(self, host, port)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 162, in __init__
    host = host.strip('[]')
AttributeError: 'NoneType' object has no attribute 'strip'
Traceback (most recent call last):
  File "/patroni.py", line 6, in <module>
    main()
  File "/patroni/__init__.py", line 99, in main
    patroni.ha.dcs.delete_leader()
  File "/patroni/etcd.py", line 138, in wrapper
    return not func(*args, **kwargs) is None
  File "/patroni/etcd.py", line 240, in delete_leader
    return self.client.delete(self.leader_path, prevValue=self._name)
  File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 529, in delete
    self.key_endpoint + key, self._MDELETE, params=kwds)
  File "/patroni/etcd.py", line 54, in api_execute
    return super(Client, self).api_execute(path, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 747, in api_execute
    preload_content=False)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 75, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 88, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 145, in urlopen
    conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 119, in connection_from_host
    pool = self._new_pool(scheme, host, port)
  File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 86, in _new_pool
    return pool_cls(host, port, **kwargs)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 232, in __init__
    ConnectionPool.__init__(self, host, port)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 162, in __init__
    host = host.strip('[]')
AttributeError: 'NoneType' object has no attribute 'strip'