Skip to content

Instantly share code, notes, and snippets.

@chaporgin
Created January 19, 2022 10:18
Show Gist options
  • Select an option

  • Save chaporgin/69952bfa177fa936e5405e622f209516 to your computer and use it in GitHub Desktop.

Select an option

Save chaporgin/69952bfa177fa936e5405e622f209516 to your computer and use it in GitHub Desktop.
chapson quick paste
console-pageserver-1 | 2022-01-19T10:17:12.194341Z INFO POST /v1/tenant/ 201 Created
console-pageserver-1 | 2022-01-19T10:17:12.216609Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:12.226927Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:12.346646Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:12.495826Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:13.230092Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:13.388767Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:13.497033Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:14.231416Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:14.439837Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:14.498350Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
68b429610745_68b429610745_68b429610745_68b429610745_68b429610745_console-db-1 | 2022-01-19 10:17:14.868 UTC [85] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
console-wal_acceptor2-1 | 2022-01-19T10:17:14.977556Z INFO {tid=23}: creating timeline dir 04bef3374e6c8e4d2fedc5cb9778f860, create is True
console-wal_acceptor3-1 | 2022-01-19T10:17:14.978742Z INFO {tid=19}: creating timeline dir 04bef3374e6c8e4d2fedc5cb9778f860, create is True
console-wal_acceptor1-1 | 2022-01-19T10:17:14.977879Z INFO {tid=27}: creating timeline dir 04bef3374e6c8e4d2fedc5cb9778f860, create is True
console-wal_acceptor2-1 | 2022-01-19T10:17:14.982506Z INFO {tid=23}: loading control file ./04bef3374e6c8e4d2fedc5cb9778f860/safekeeper.control, create=True
console-wal_acceptor3-1 | 2022-01-19T10:17:14.984726Z INFO {tid=19}: loading control file ./04bef3374e6c8e4d2fedc5cb9778f860/safekeeper.control, create=True
console-wal_acceptor1-1 | 2022-01-19T10:17:14.986695Z INFO {tid=27}: loading control file ./04bef3374e6c8e4d2fedc5cb9778f860/safekeeper.control, create=True
console-wal_acceptor3-1 | 2022-01-19T10:17:14.992430Z INFO {tid=19}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:62318 sysid 0 timeline 0
console-wal_acceptor3-1 | 2022-01-19T10:17:14.992993Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:14.991709Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:63218 sysid 0 timeline 0
console-wal_acceptor2-1 | 2022-01-19T10:17:14.992064Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:14.998142Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:60210 sysid 0 timeline 0
console-wal_acceptor1-1 | 2022-01-19T10:17:14.998501Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:15.001335Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [9, 0, 169, 242, 32, 38, 216, 153, 231, 240, 249, 120, 224, 104, 40, 228], sending term 0
console-wal_acceptor3-1 | 2022-01-19T10:17:15.004784Z INFO {tid=19}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [9, 0, 169, 242, 32, 38, 216, 153, 231, 240, 249, 120, 224, 104, 40, 228], sending term 0
console-wal_acceptor1-1 | 2022-01-19T10:17:15.010963Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [9, 0, 169, 242, 32, 38, 216, 153, 231, 240, 249, 120, 224, 104, 40, 228], sending term 0
console-wal_acceptor2-1 | 2022-01-19T10:17:15.019832Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 1: VoteResponse { term: 1, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor3-1 | 2022-01-19T10:17:15.023903Z INFO {tid=19}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 1: VoteResponse { term: 1, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor1-1 | 2022-01-19T10:17:15.028016Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 1: VoteResponse { term: 1, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor3-1 | 2022-01-19T10:17:15.031348Z WARN {tid=19}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:15.031389Z WARN {tid=23}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor1-1 | 2022-01-19T10:17:15.031413Z WARN {tid=27}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-pageserver-1 | 2022-01-19T10:17:15.046810Z INFO basebackup{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: taking basebackup lsn=0/1695D10, prev_lsn=0/1695C98
console-pageserver-1 | 2022-01-19T10:17:15.048025Z INFO basebackup{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5 lsn="0/1695D10"}: list_nonrels called at 0/1695D10
console-pageserver-1 | 2022-01-19T10:17:15.234585Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:15.483844Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:15.499912Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:15.545915Z INFO new call spawned. time Instant { tv_sec: 123159, tv_nsec: 453742033 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:15.546053Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:15.560108Z INFO callmemaybe{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: activating tenant 70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:15.560604Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL receiver thread started
console-pageserver-1 | 2022-01-19T10:17:15.561641Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connecting to "host=wal_acceptor2 port=6500 options='-c ztimelineid=04bef3374e6c8e4d2fedc5cb9778f860 ztenantid=70e94d4b4ebac68cef2325fd3dcbdbc5'"
console-pageserver-1 | 2022-01-19T10:17:15.570038Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connected!
console-pageserver-1 | 2022-01-19T10:17:15.571218Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: IdentifySystem { systemid: 0, timeline: 1, xlogpos: 0/0, dbname: None }
console-pageserver-1 | 2022-01-19T10:17:15.571328Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: last_record_lsn 0/1695D10 starting replication from 0/1695D10, server is at 0/0...
console-wal_acceptor2-1 | 2022-01-19T10:17:15.574613Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/1695D10 till None
console-wal_acceptor2-1 | 2022-01-19T10:17:15.575284Z INFO callmemaybe. thread_main. pause callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:15.590626Z INFO new call spawned. time Instant { tv_sec: 123159, tv_nsec: 498472199 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:15.590706Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:15.668909Z INFO new call spawned. time Instant { tv_sec: 123159, tv_nsec: 576756866 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:15.668991Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:16.116690Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:62356 sysid 7054859081854792284 timeline 0
console-wal_acceptor3-1 | 2022-01-19T10:17:16.116904Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: Some(JoinHandle), last_call_time: Instant { tv_sec: 123159, tv_nsec: 576756866 }, paused: false }
console-wal_acceptor3-1 | 2022-01-19T10:17:16.116990Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:16.116778Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:63256 sysid 7054859081854792284 timeline 0
console-wal_acceptor2-1 | 2022-01-19T10:17:16.118841Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: None, last_call_time: Instant { tv_sec: 123159, tv_nsec: 453742033 }, paused: true }
console-wal_acceptor2-1 | 2022-01-19T10:17:16.118953Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:16.125254Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:60246 sysid 7054859081854792284 timeline 0
console-wal_acceptor1-1 | 2022-01-19T10:17:16.125619Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: Some(JoinHandle), last_call_time: Instant { tv_sec: 123159, tv_nsec: 498472199 }, paused: false }
console-wal_acceptor1-1 | 2022-01-19T10:17:16.127498Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:16.137609Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [74, 171, 110, 62, 81, 193, 25, 73, 82, 211, 50, 164, 93, 24, 90, 249], sending term 1
console-wal_acceptor3-1 | 2022-01-19T10:17:16.136868Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [74, 171, 110, 62, 81, 193, 25, 73, 82, 211, 50, 164, 93, 24, 90, 249], sending term 1
console-wal_acceptor1-1 | 2022-01-19T10:17:16.146718Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [74, 171, 110, 62, 81, 193, 25, 73, 82, 211, 50, 164, 93, 24, 90, 249], sending term 1
console-wal_acceptor3-1 | 2022-01-19T10:17:16.154138Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 2: VoteResponse { term: 2, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor2-1 | 2022-01-19T10:17:16.156605Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 2: VoteResponse { term: 2, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor2-1 | 2022-01-19T10:17:16.161043Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 2, start_streaming_at: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor3-1 | 2022-01-19T10:17:16.161958Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 2, start_streaming_at: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor1-1 | 2022-01-19T10:17:16.163588Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 2: VoteResponse { term: 2, vote_given: 1, flush_lsn: 0/0, truncate_lsn: 0/0, term_history: [] }
console-wal_acceptor1-1 | 2022-01-19T10:17:16.167421Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 2, start_streaming_at: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor2-1 | 2022-01-19T10:17:16.177123Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/1695D10
console-wal_acceptor3-1 | 2022-01-19T10:17:16.176923Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/1695D10
console-wal_acceptor1-1 | 2022-01-19T10:17:16.181582Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/1695D10
console-pageserver-1 | 2022-01-19T10:17:16.235982Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:16.501783Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:16.529057Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-wal_acceptor2-1 | 2022-01-19T10:17:16.545664Z INFO new call spawned. time Instant { tv_sec: 123160, tv_nsec: 453511991 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:16.545745Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:16.590620Z INFO new call spawned. time Instant { tv_sec: 123160, tv_nsec: 498466783 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:16.590724Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:16.668455Z INFO new call spawned. time Instant { tv_sec: 123160, tv_nsec: 576301491 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:16.668530Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:17.237322Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:17.503037Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:17.591270Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:18.238601Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:18.504612Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:18.545888Z INFO new call spawned. time Instant { tv_sec: 123162, tv_nsec: 453734242 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:18.546783Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:18.590389Z INFO new call spawned. time Instant { tv_sec: 123162, tv_nsec: 498234617 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:18.590705Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:18.630049Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-wal_acceptor3-1 | 2022-01-19T10:17:18.668733Z INFO new call spawned. time Instant { tv_sec: 123162, tv_nsec: 576576617 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:18.668860Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:19.239681Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:19.505969Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:19.546096Z INFO new call spawned. time Instant { tv_sec: 123163, tv_nsec: 453942535 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:19.546283Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:19.590700Z INFO new call spawned. time Instant { tv_sec: 123163, tv_nsec: 498547743 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:19.590793Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor2-1 | 2022-01-19T10:17:19.648856Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: restart decoder from 0/0 to 0/1695D10
console-wal_acceptor1-1 | 2022-01-19T10:17:19.654945Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: restart decoder from 0/0 to 0/1695D10
console-wal_acceptor2-1 | 2022-01-19T10:17:19.658730Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/1695D48, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/1695D48, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.663597Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 2680, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.663488Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/1695D48, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/1695D48, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.663799Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.663968Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.664019Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.669764Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/1695D48, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.670932Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 2680, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.671057Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.671097Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.671131Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.671369Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/1695D48, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.674135Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/16967C0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.674953Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/16967C0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:19.675986Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-pageserver-1 | 2022-01-19T10:17:19.680873Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: caught up at LSN 0/1695D48
console-wal_acceptor2-1 | 2022-01-19T10:17:19.680440Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/1695D48
console-wal_acceptor2-1 | 2022-01-19T10:17:19.683941Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/16967C0
console-wal_acceptor3-1 | 2022-01-19T10:17:19.677784Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: restart decoder from 0/0 to 0/1695D10
console-wal_acceptor3-1 | 2022-01-19T10:17:19.690864Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/1695D48, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/1695D48, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.698987Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 2680, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.699316Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.699387Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.699422Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.699473Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/1695D48, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.699507Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/16967C0, commit_lsn=0/16967C0, truncate_lsn=0/1695D10, resp AppendResponse { term: 2, flush_lsn: 0/16967C0, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.859731Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 3304, end_lsn=0/16974A8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16974A8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.859617Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 3304, end_lsn=0/16974A8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16974A8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.859778Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 3304, end_lsn=0/16974A8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16974A8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.866862Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/16994F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16994F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.866692Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/16994F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16994F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.867109Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/16994F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/16994F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.873087Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/169B548, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B548, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.873435Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/169B548, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B548, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.876545Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 8272, end_lsn=0/169B548, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B548, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.879929Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 176, end_lsn=0/169B5F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.880124Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16974A8, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.880484Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16994F8, truncate_lsn=0/16974A8, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.881025Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 176, end_lsn=0/169B5F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.881300Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16974A8, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.881547Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16994F8, truncate_lsn=0/16974A8, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.882632Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 176, end_lsn=0/169B5F8, commit_lsn=0/16967C0, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.882772Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16974A8, truncate_lsn=0/16967C0, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.882818Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B5F8, commit_lsn=0/16994F8, truncate_lsn=0/16974A8, resp AppendResponse { term: 2, flush_lsn: 0/169B5F8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.887604Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 224, end_lsn=0/169B6D8, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B6D8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.888004Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 224, end_lsn=0/169B6D8, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B6D8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.892408Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 224, end_lsn=0/169B6D8, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B6D8, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.896641Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/16994F8
console-wal_acceptor3-1 | 2022-01-19T10:17:19.897660Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 88, end_lsn=0/169B730, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.897897Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B548, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.897983Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B5F8, truncate_lsn=0/169B548, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.898021Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B6D8, truncate_lsn=0/169B5F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.899427Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 88, end_lsn=0/169B730, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.899635Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B548, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.899693Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B5F8, truncate_lsn=0/169B548, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.899729Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B6D8, truncate_lsn=0/169B5F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:19.901862Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B6D8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:19.902319Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B6D8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.901866Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 88, end_lsn=0/169B730, commit_lsn=0/16994F8, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.904062Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B548, truncate_lsn=0/16994F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.904457Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B5F8, truncate_lsn=0/169B548, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.904524Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B6D8, truncate_lsn=0/169B5F8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.905551Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B6D8, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:19.911298Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169B548
console-wal_acceptor2-1 | 2022-01-19T10:17:19.911459Z INFO {tid=25}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169B730
console-wal_acceptor1-1 | 2022-01-19T10:17:20.084642Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:20.084657Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:20.084617Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:20.240944Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:20.507568Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:20.668713Z INFO new call spawned. time Instant { tv_sec: 123164, tv_nsec: 576559118 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:20.668837Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:20.738450Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-wal_acceptor1-1 | 2022-01-19T10:17:21.084390Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:21.084384Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:21.084570Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:21.241938Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:21.508922Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:21.544875Z INFO new call spawned. time Instant { tv_sec: 123165, tv_nsec: 452720494 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:21.545027Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:21.590593Z INFO new call spawned. time Instant { tv_sec: 123165, tv_nsec: 498439619 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:21.590749Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor2-1 | 2022-01-19T10:17:22.085052Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:22.085069Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:22.085356Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:22.242892Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:22.511103Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:22.545029Z INFO new call spawned. time Instant { tv_sec: 123166, tv_nsec: 452872744 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:22.545134Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:22.668723Z INFO new call spawned. time Instant { tv_sec: 123166, tv_nsec: 576570244 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:22.668875Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor2-1 | 2022-01-19T10:17:23.085319Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:23.085341Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:23.085658Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:23.244010Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:23.514528Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:23.545949Z INFO new call spawned. time Instant { tv_sec: 123167, tv_nsec: 453795870 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:23.546065Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:23.590606Z INFO new call spawned. time Instant { tv_sec: 123167, tv_nsec: 498451911 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:23.590697Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:23.668888Z INFO new call spawned. time Instant { tv_sec: 123167, tv_nsec: 576734953 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:23.669050Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:23.757976Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:processing record{lsn=0/169B628}: list_nonrels called at 0/169B5F8
console-pageserver-1 | 2022-01-19T10:17:23.762470Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:processing record{lsn=0/169B628}: Created database 1663/16385, copied 1010 blocks in 243 rels at 0/169B628
console-pageserver-1 | 2022-01-19T10:17:23.763456Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22776 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:23.994875Z INFO GET /v1/branch/70e94d4b4ebac68cef2325fd3dcbdbc5 200 OK
console-wal_acceptor2-1 | 2022-01-19T10:17:24.085347Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:24.085362Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:24.085438Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169B730, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169B730, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:24.104694Z INFO {tid=29}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 1240, end_lsn=0/169BC08, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:24.106928Z INFO {tid=27}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 1240, end_lsn=0/169BC08, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:24.106895Z INFO {tid=21}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 1240, end_lsn=0/169BC08, commit_lsn=0/169B730, truncate_lsn=0/169B730, resp AppendResponse { term: 2, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:24.110952Z WARN {tid=21}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:24.110938Z WARN {tid=27}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor1-1 | 2022-01-19T10:17:24.110879Z WARN {tid=29}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-pageserver-1 | 2022-01-19T10:17:24.245273Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:24.515851Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:24.769076Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:24.914388Z INFO callmemaybe. thread_main. unsubscribe callback. request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:24.914484Z INFO callmemaybe. thread_main. resume callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:24.914513Z WARN {tid=25}: query handler for START_REPLICATION PHYSICAL 0/1695D10 failed: failed to run ReplicationConn: end streaming to Some("pageserver")
console-pageserver-1 | connection error: connection closed
console-pageserver-1 | 2022-01-19T10:17:24.916060Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL streaming connection failed (connection closed)
console-pageserver-1 | 2022-01-19T10:17:25.254993Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:25.526923Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:25.591313Z INFO new call spawned. time Instant { tv_sec: 123169, tv_nsec: 499110121 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:25.591846Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:25.628476Z INFO callmemaybe{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: activating tenant 70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:25.628632Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL receiver thread started
console-pageserver-1 | 2022-01-19T10:17:25.628735Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connecting to "host=wal_acceptor1 port=6500 options='-c ztimelineid=04bef3374e6c8e4d2fedc5cb9778f860 ztenantid=70e94d4b4ebac68cef2325fd3dcbdbc5'"
console-pageserver-1 | 2022-01-19T10:17:25.636088Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connected!
console-pageserver-1 | 2022-01-19T10:17:25.637069Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: IdentifySystem { systemid: 7054859081854792284, timeline: 1, xlogpos: 0/169BC08, dbname: None }
console-pageserver-1 | 2022-01-19T10:17:25.637285Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: last_record_lsn 0/169B730 starting replication from 0/169B730, server is at 0/169BC08...
console-wal_acceptor1-1 | 2022-01-19T10:17:25.639691Z INFO {tid=30}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/169B730 till None
console-wal_acceptor1-1 | 2022-01-19T10:17:25.639948Z INFO callmemaybe. thread_main. pause callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:25.669298Z INFO new call spawned. time Instant { tv_sec: 123169, tv_nsec: 577136287 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:25.669466Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:25.780001Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:26.275757Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:26.530970Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:26.672722Z INFO new call spawned. time Instant { tv_sec: 123170, tv_nsec: 580562580 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:26.672896Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:26.788575Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:27.298446Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:27.538051Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | connection error: connection closed
console-pageserver-1 | 2022-01-19T10:17:27.645333Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL streaming connection failed (connection closed)
console-wal_acceptor1-1 | 2022-01-19T10:17:27.643438Z INFO callmemaybe. thread_main. unsubscribe callback. request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:27.643684Z WARN {tid=30}: query handler for START_REPLICATION PHYSICAL 0/169B730 failed: failed to run ReplicationConn: end streaming to Some("pageserver")
console-wal_acceptor1-1 | 2022-01-19T10:17:27.643921Z INFO callmemaybe. thread_main. resume callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:27.790969Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:27.919269Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:63382 sysid 0 timeline 0
console-wal_acceptor2-1 | 2022-01-19T10:17:27.920098Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:27.920496Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:60374 sysid 0 timeline 0
console-wal_acceptor1-1 | 2022-01-19T10:17:27.920774Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:27.922355Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:62486 sysid 0 timeline 0
console-wal_acceptor3-1 | 2022-01-19T10:17:27.922531Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: Some(JoinHandle), last_call_time: Instant { tv_sec: 123170, tv_nsec: 580562580 }, paused: false }
console-wal_acceptor3-1 | 2022-01-19T10:17:27.922678Z INFO new call spawned. time Instant { tv_sec: 123171, tv_nsec: 830532622 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:27.922702Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:27.924438Z WARN callback task for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5 was cancelled before spawning a new one
console-wal_acceptor3-1 | 2022-01-19T10:17:27.946422Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [196, 55, 100, 22, 75, 244, 184, 153, 40, 103, 78, 110, 253, 103, 34, 88], sending term 2
console-wal_acceptor2-1 | 2022-01-19T10:17:27.952793Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [196, 55, 100, 22, 75, 244, 184, 153, 40, 103, 78, 110, 253, 103, 34, 88], sending term 2
console-wal_acceptor1-1 | 2022-01-19T10:17:27.952504Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [196, 55, 100, 22, 75, 244, 184, 153, 40, 103, 78, 110, 253, 103, 34, 88], sending term 2
console-wal_acceptor1-1 | 2022-01-19T10:17:27.981721Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 3: VoteResponse { term: 3, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor3-1 | 2022-01-19T10:17:27.982769Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 3: VoteResponse { term: 3, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor2-1 | 2022-01-19T10:17:27.984468Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 3: VoteResponse { term: 3, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(2, 0/1695D10)] }
console-wal_acceptor1-1 | 2022-01-19T10:17:27.993079Z INFO {tid=33}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/1695D10 till Some(0/169BC08)
console-wal_acceptor1-1 | 2022-01-19T10:17:28.011420Z INFO {tid=33}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169BC08
console-wal_acceptor1-1 | 2022-01-19T10:17:28.015522Z ERROR connection handler exited: reader taken
console-wal_acceptor1-1 | 2022-01-19T10:17:28.021496Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 3, start_streaming_at: 0/169BC08, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor2-1 | 2022-01-19T10:17:28.021512Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 3, start_streaming_at: 0/169BC08, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor3-1 | 2022-01-19T10:17:28.021868Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 3, start_streaming_at: 0/169BC08, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-pageserver-1 | 2022-01-19T10:17:28.299955Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:28.540557Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:28.545331Z INFO new call spawned. time Instant { tv_sec: 123172, tv_nsec: 453179914 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:28.545438Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:28.554990Z INFO callmemaybe{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: activating tenant 70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:28.555091Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL receiver thread started
console-pageserver-1 | 2022-01-19T10:17:28.555310Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connecting to "host=wal_acceptor2 port=6500 options='-c ztimelineid=04bef3374e6c8e4d2fedc5cb9778f860 ztenantid=70e94d4b4ebac68cef2325fd3dcbdbc5'"
console-pageserver-1 | 2022-01-19T10:17:28.561665Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connected!
console-wal_acceptor1-1 | 2022-01-19T10:17:28.589817Z INFO new call spawned. time Instant { tv_sec: 123172, tv_nsec: 497664830 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:28.590058Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:28.669341Z INFO new call spawned. time Instant { tv_sec: 123172, tv_nsec: 577188706 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:28.669416Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:28.792239Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:29.301248Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:29.542643Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:29.545646Z INFO new call spawned. time Instant { tv_sec: 123173, tv_nsec: 453492581 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:29.545752Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor1-1 | 2022-01-19T10:17:29.590715Z INFO new call spawned. time Instant { tv_sec: 123173, tv_nsec: 498561248 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:29.590842Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:29.793625Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:30.039771Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor3-1 | 2022-01-19T10:17:30.039907Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:30.039998Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:30.040129Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:30.302448Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:30.543940Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:30.591922Z INFO new call spawned. time Instant { tv_sec: 123174, tv_nsec: 499767790 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:30.592632Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:30.668282Z INFO new call spawned. time Instant { tv_sec: 123174, tv_nsec: 576126373 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:30.668481Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:30.794914Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 22984 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:30.906783Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:31.303840Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:31.396883Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor1-1 | 2022-01-19T10:17:31.397027Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:31.397238Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:31.397309Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:31.397359Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:31.407063Z INFO {tid=32}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/169BC08, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:31.407108Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor2-1 | 2022-01-19T10:17:31.407712Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:31.407813Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:31.407853Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/0, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:31.410665Z INFO {tid=22}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/169BC08, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:31.417953Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: IdentifySystem { systemid: 0, timeline: 1, xlogpos: 0/169BC08, dbname: None }
console-pageserver-1 | 2022-01-19T10:17:31.418106Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: last_record_lsn 0/169B730 starting replication from 0/169B730, server is at 0/169BC08...
console-wal_acceptor2-1 | 2022-01-19T10:17:31.417363Z INFO {tid=28}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC08, commit_lsn=0/169BC08, truncate_lsn=0/1695D10, resp AppendResponse { term: 3, flush_lsn: 0/169BC08, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:31.420128Z INFO {tid=29}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/169B730 till None
console-wal_acceptor2-1 | 2022-01-19T10:17:31.420374Z INFO callmemaybe. thread_main. pause callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:31.423778Z WARN {tid=22}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:31.423877Z WARN {tid=28}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor1-1 | 2022-01-19T10:17:31.424450Z WARN {tid=32}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:31.428823Z INFO {tid=29}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169BC08
console-pageserver-1 | 2022-01-19T10:17:31.432088Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: caught up at LSN 0/169BC08
console-pageserver-1 | 2022-01-19T10:17:31.436956Z INFO basebackup{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: taking basebackup lsn=0/169BC08, prev_lsn=0/169B730
console-pageserver-1 | 2022-01-19T10:17:31.437902Z INFO basebackup{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5 lsn="0/169BC08"}: list_nonrels called at 0/169BC08
console-pageserver-1 | 2022-01-19T10:17:31.545404Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:31.668757Z INFO new call spawned. time Instant { tv_sec: 123175, tv_nsec: 576603874 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:31.668920Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:31.796083Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24224 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:32.238592Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:62564 sysid 7054859081854792284 timeline 0
console-wal_acceptor2-1 | 2022-01-19T10:17:32.238494Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:63460 sysid 7054859081854792284 timeline 0
console-wal_acceptor3-1 | 2022-01-19T10:17:32.239238Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: Some(JoinHandle), last_call_time: Instant { tv_sec: 123175, tv_nsec: 576603874 }, paused: false }
console-wal_acceptor2-1 | 2022-01-19T10:17:32.239530Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: None, last_call_time: Instant { tv_sec: 123173, tv_nsec: 453492581 }, paused: true }
console-wal_acceptor2-1 | 2022-01-19T10:17:32.239604Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:32.239848Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:32.240239Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start handshake with wal proposer 172.19.0.1:60452 sysid 7054859081854792284 timeline 0
console-wal_acceptor1-1 | 2022-01-19T10:17:32.241432Z INFO callmemaybe. subscription already exists SubscriptionState { tenantid: ZTenantId(ZId([112, 233, 77, 75, 78, 186, 198, 140, 239, 35, 37, 253, 61, 203, 219, 197])), timelineid: ZTimelineId(ZId([4, 190, 243, 55, 78, 108, 142, 77, 47, 237, 197, 203, 151, 120, 248, 96])), pageserver_connstr: "postgresql://(null):@host.docker.internal:6400", handle: Some(JoinHandle), last_call_time: Instant { tv_sec: 123174, tv_nsec: 499767790 }, paused: false }
console-wal_acceptor1-1 | 2022-01-19T10:17:32.241689Z INFO new call spawned. time Instant { tv_sec: 123176, tv_nsec: 149543832 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:32.241712Z INFO callmemaybe. thread_main. subscribe callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:32.241776Z WARN callback task for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5 was cancelled before spawning a new one
console-wal_acceptor2-1 | 2022-01-19T10:17:32.250038Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [97, 103, 120, 207, 1, 37, 138, 147, 155, 107, 152, 254, 34, 52, 124, 79], sending term 3
console-wal_acceptor3-1 | 2022-01-19T10:17:32.250745Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [97, 103, 120, 207, 1, 37, 138, 147, 155, 107, 152, 254, 34, 52, 124, 79], sending term 3
console-wal_acceptor1-1 | 2022-01-19T10:17:32.251341Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed greeting from proposer [97, 103, 120, 207, 1, 37, 138, 147, 155, 107, 152, 254, 34, 52, 124, 79], sending term 3
console-wal_acceptor3-1 | 2022-01-19T10:17:32.267608Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 4: VoteResponse { term: 4, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor1-1 | 2022-01-19T10:17:32.269546Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 4: VoteResponse { term: 4, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor2-1 | 2022-01-19T10:17:32.269625Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed VoteRequest for term 4: VoteResponse { term: 4, vote_given: 1, flush_lsn: 0/169BC08, truncate_lsn: 0/1695D10, term_history: [(3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor1-1 | 2022-01-19T10:17:32.279009Z INFO {tid=36}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/1695D10 till Some(0/169BC08)
console-wal_acceptor1-1 | 2022-01-19T10:17:32.285623Z INFO {tid=36}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169BC08
console-wal_acceptor1-1 | 2022-01-19T10:17:32.286566Z ERROR connection handler exited: reader taken
console-wal_acceptor1-1 | 2022-01-19T10:17:32.290967Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 4, start_streaming_at: 0/169BC08, term_history: [(4, 0/169BC08), (3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor3-1 | 2022-01-19T10:17:32.291069Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 4, start_streaming_at: 0/169BC08, term_history: [(4, 0/169BC08), (3, 0/169BC08), (2, 0/1695D10)] }
console-wal_acceptor2-1 | 2022-01-19T10:17:32.294509Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: received ProposerElected ProposerElected { term: 4, start_streaming_at: 0/169BC08, term_history: [(4, 0/169BC08), (3, 0/169BC08), (2, 0/1695D10)] }
console-pageserver-1 | 2022-01-19T10:17:32.305079Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:32.429043Z INFO pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: running initdb in "/data/tenants/70e94d4b4ebac68cef2325fd3dcbdbc5/wal-redo-datadir"
console-wal_acceptor2-1 | 2022-01-19T10:17:32.545285Z INFO new call spawned. time Instant { tv_sec: 123176, tv_nsec: 453127624 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:32.545363Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:32.547267Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:32.590784Z INFO new call spawned. time Instant { tv_sec: 123176, tv_nsec: 498627832 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:32.590861Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:32.668360Z INFO new call spawned. time Instant { tv_sec: 123176, tv_nsec: 576187457 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:32.668446Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:32.797500Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24224 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:33.306399Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:33.546837Z INFO new call spawned. time Instant { tv_sec: 123177, tv_nsec: 453897290 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:33.547250Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:33.551401Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:33.800105Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24224 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:34.309187Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:34.453726Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor3-1 | 2022-01-19T10:17:34.461296Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:34.461516Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:34.461561Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:34.553762Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:34.591310Z INFO new call spawned. time Instant { tv_sec: 123178, tv_nsec: 498391957 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:34.591438Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:34.669452Z INFO new call spawned. time Instant { tv_sec: 123178, tv_nsec: 576533957 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:34.669559Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:34.801734Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24224 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:35.220288Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:35.310343Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:35.546547Z INFO new call spawned. time Instant { tv_sec: 123179, tv_nsec: 453630374 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:35.546644Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:35.554869Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:35.669632Z INFO new call spawned. time Instant { tv_sec: 123179, tv_nsec: 576714875 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:35.669732Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:35.804073Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24224 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:36.095895Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor1-1 | 2022-01-19T10:17:36.100746Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:36.100881Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:36.101033Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:36.101078Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:36.104832Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:36.105279Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.107738Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: start receiving WAL since 0/169BC08
console-wal_acceptor2-1 | 2022-01-19T10:17:36.113558Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.113791Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.114069Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.114132Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/0, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.114169Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/1695D10, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.119076Z INFO {tid=29}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169BC40
console-wal_acceptor3-1 | 2022-01-19T10:17:36.219268Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:36.219569Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:36.219753Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:36.311479Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:36.546640Z INFO new call spawned. time Instant { tv_sec: 123180, tv_nsec: 453720625 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:36.546739Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:36.556955Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:36.591090Z INFO new call spawned. time Instant { tv_sec: 123180, tv_nsec: 498172625 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:36.591187Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:36.805337Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:37.220095Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:37.220010Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:37.220070Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:37.312779Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:37.558321Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:37.669409Z INFO new call spawned. time Instant { tv_sec: 123181, tv_nsec: 576488876 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:37.669520Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:37.807955Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:38.221068Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:38.221030Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:38.221329Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:38.313883Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:38.546709Z INFO new call spawned. time Instant { tv_sec: 123182, tv_nsec: 453788501 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:38.546804Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:38.560082Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:38.591450Z INFO new call spawned. time Instant { tv_sec: 123182, tv_nsec: 498534001 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:38.591554Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:38.810101Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:39.221676Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:39.221632Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:39.221589Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:39.315239Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:39.561123Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:39.668557Z INFO new call spawned. time Instant { tv_sec: 123183, tv_nsec: 575638168 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:39.668647Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:39.811253Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:40.221819Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:40.221733Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:40.221813Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:40.316303Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:40.545739Z INFO new call spawned. time Instant { tv_sec: 123184, tv_nsec: 452822752 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:40.545838Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:40.562531Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:40.591405Z INFO new call spawned. time Instant { tv_sec: 123184, tv_nsec: 498488460 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:40.591557Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:40.669380Z INFO new call spawned. time Instant { tv_sec: 123184, tv_nsec: 576462210 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:40.669556Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:40.812663Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:41.224369Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:41.223955Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:41.223864Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:41.317226Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:41.546507Z INFO new call spawned. time Instant { tv_sec: 123185, tv_nsec: 453590461 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:41.546636Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:41.563879Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:41.669635Z INFO new call spawned. time Instant { tv_sec: 123185, tv_nsec: 576716211 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:41.669752Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:41.815012Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:42.224584Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:42.224314Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:42.225434Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:42.318341Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:42.565202Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:42.590843Z INFO new call spawned. time Instant { tv_sec: 123186, tv_nsec: 497925378 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:42.591140Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:42.817585Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:43.225638Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:43.225403Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:43.225493Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:43.319373Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:43.546444Z INFO new call spawned. time Instant { tv_sec: 123187, tv_nsec: 453522462 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:43.546576Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:43.566563Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:43.592024Z INFO new call spawned. time Instant { tv_sec: 123187, tv_nsec: 498965378 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:43.592166Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:43.669340Z INFO new call spawned. time Instant { tv_sec: 123187, tv_nsec: 576421337 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:43.669512Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:43.818824Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:44.224778Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:44.224805Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:44.224905Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:44.320627Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:44.546707Z INFO new call spawned. time Instant { tv_sec: 123188, tv_nsec: 453789670 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:44.546809Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:44.568134Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:44.821086Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:45.224807Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:45.224798Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:45.224881Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:45.322137Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:45.569874Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:45.591734Z INFO new call spawned. time Instant { tv_sec: 123189, tv_nsec: 498777046 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:45.591889Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:45.669345Z INFO new call spawned. time Instant { tv_sec: 123189, tv_nsec: 576428338 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:45.669442Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:45.822422Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:46.225704Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:46.225701Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:46.225687Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:46.323456Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:46.546362Z INFO new call spawned. time Instant { tv_sec: 123190, tv_nsec: 453442296 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:46.546484Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:46.571246Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:46.824082Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24280 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:47.226728Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:47.226697Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:47.226757Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC40, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC40, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:47.259025Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC78, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:47.260026Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC78, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:47.260406Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 56, end_lsn=0/169BC78, commit_lsn=0/169BC40, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:47.263178Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:47.262612Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:47.262650Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC40, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:47.267257Z INFO {tid=29}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: sent WAL up to 0/169BC78
console-pageserver-1 | 2022-01-19T10:17:47.324679Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:47.547712Z INFO new call spawned. time Instant { tv_sec: 123191, tv_nsec: 454501547 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:47.548846Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:47.572619Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:47.590697Z INFO new call spawned. time Instant { tv_sec: 123191, tv_nsec: 497779214 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:47.590860Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:47.669379Z INFO new call spawned. time Instant { tv_sec: 123191, tv_nsec: 576461589 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:47.669482Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:47.826523Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:48.227653Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:48.227891Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:48.227657Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:48.326043Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:48.574079Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:48.591149Z INFO new call spawned. time Instant { tv_sec: 123192, tv_nsec: 498231589 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:48.591264Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:48.828246Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:49.227864Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:49.228062Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:49.228068Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:49.327248Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:49.545797Z INFO new call spawned. time Instant { tv_sec: 123193, tv_nsec: 452880006 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:49.545994Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:49.575165Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:49.668984Z INFO new call spawned. time Instant { tv_sec: 123193, tv_nsec: 575995923 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:49.669103Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:49.829324Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:50.229139Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:50.228886Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:50.229086Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:50.328224Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:50.546444Z INFO new call spawned. time Instant { tv_sec: 123194, tv_nsec: 453527132 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:50.546597Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:50.576679Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:50.591264Z INFO new call spawned. time Instant { tv_sec: 123194, tv_nsec: 498347340 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:50.591437Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:50.669628Z INFO new call spawned. time Instant { tv_sec: 123194, tv_nsec: 576711590 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:50.669737Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:50.830656Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:51.229395Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:51.229585Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:51.229557Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:51.329313Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:51.578054Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:51.831842Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:52.230431Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:52.230287Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor3-1 | 2022-01-19T10:17:52.230544Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:52.330399Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor2-1 | 2022-01-19T10:17:52.546390Z INFO new call spawned. time Instant { tv_sec: 123196, tv_nsec: 453459966 }
console-wal_acceptor2-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor2-1 | 2022-01-19T10:17:52.546521Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:52.579681Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:52.590732Z INFO new call spawned. time Instant { tv_sec: 123196, tv_nsec: 497813674 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:52.590905Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-wal_acceptor3-1 | 2022-01-19T10:17:52.668855Z INFO new call spawned. time Instant { tv_sec: 123196, tv_nsec: 575938299 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:52.669041Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:52.833053Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:53.230299Z INFO {tid=23}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor2-1 | 2022-01-19T10:17:53.230343Z INFO {tid=31}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-wal_acceptor1-1 | 2022-01-19T10:17:53.230271Z INFO {tid=35}:WAL acceptor{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: processed AppendRequest of len 0, end_lsn=0/169BC78, commit_lsn=0/169BC78, truncate_lsn=0/169BC78, resp AppendResponse { term: 4, flush_lsn: 0/169BC78, commit_lsn: 0/169BC08, disk_consistent_lsn: 0/0, hs_feedback: HotStandbyFeedback { ts: 0, xmin: 0, catalog_xmin: 0 } }
console-pageserver-1 | 2022-01-19T10:17:53.331495Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:53.581599Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:53.591335Z INFO new call spawned. time Instant { tv_sec: 123197, tv_nsec: 498415300 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:53.591434Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:53.616514Z INFO pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: launched WAL redo postgres process on "/data/tenants/70e94d4b4ebac68cef2325fd3dcbdbc5/wal-redo-datadir"
console-wal_acceptor3-1 | 2022-01-19T10:17:53.669263Z INFO new call spawned. time Instant { tv_sec: 123197, tv_nsec: 576347216 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:53.669407Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:53.719062Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: wal-redo-postgres: 2022-01-19 10:17:53.718 UTC [810] FATAL: seccomp: failed to load a test filter
console-pageserver-1 |
console-pageserver-1 | 2022-01-19T10:17:53.726315Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: wal-redo-postgres exited with code exit status: 1
console-pageserver-1 | 2022-01-19T10:17:53.726636Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: error reading relation or page version: WAL redo process closed its stderr unexpectedly
console-pageserver-1 |
console-pageserver-1 | Stack backtrace:
console-pageserver-1 | 0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1914:27
console-pageserver-1 | pageserver::layered_repository::LayeredTimeline::reconstruct_page
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:2039:27
console-pageserver-1 | pageserver::layered_repository::LayeredTimeline::materialize_page
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:1981:9
console-pageserver-1 | 1: <pageserver::layered_repository::LayeredTimeline as pageserver::repository::Timeline>::get_page_at_lsn::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:809:46
console-pageserver-1 | prometheus::histogram::Histogram::observe_closure_duration
console-pageserver-1 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/prometheus-0.12.0/src/histogram.rs:715:19
console-pageserver-1 | <pageserver::layered_repository::LayeredTimeline as pageserver::repository::Timeline>::get_page_at_lsn
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:808:13
console-pageserver-1 | 2: pageserver::page_service::PageServerHandler::handle_get_page_at_lsn_request
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:487:20
console-pageserver-1 | 3: pageserver::page_service::PageServerHandler::handle_pagerequests::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:361:37
console-pageserver-1 | prometheus::histogram::Histogram::observe_closure_duration
console-pageserver-1 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/prometheus-0.12.0/src/histogram.rs:715:19
console-pageserver-1 | pageserver::page_service::PageServerHandler::handle_pagerequests
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:358:66
console-pageserver-1 | <pageserver::page_service::PageServerHandler as zenith_utils::postgres_backend::Handler>::process_query
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:597:13
console-pageserver-1 | 4: zenith_utils::postgres_backend::PostgresBackend::process_message
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:434:33
console-pageserver-1 | zenith_utils::postgres_backend::PostgresBackend::run_message_loop
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:283:31
console-pageserver-1 | zenith_utils::postgres_backend::PostgresBackend::run
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:265:19
console-pageserver-1 | 5: pageserver::page_service::page_service_conn_main
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:282:5
console-pageserver-1 | 6: pageserver::page_service::thread_main::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:234:29
console-pageserver-1 | <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panic/unwind_safe.rs:271:9
console-pageserver-1 | std::panicking::try::do_call
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
console-pageserver-1 | std::panicking::try
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
console-pageserver-1 | std::panic::catch_unwind
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
console-pageserver-1 | pageserver::thread_mgr::thread_wrapper
console-pageserver-1 | at /zenith/pageserver/src/thread_mgr.rs:201:18
console-pageserver-1 | 7: pageserver::thread_mgr::spawn::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/thread_mgr.rs:165:24
console-pageserver-1 | std::sys_common::backtrace::__rust_begin_short_backtrace
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:123:18
console-pageserver-1 | 8: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:483:17
console-pageserver-1 | <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panic/unwind_safe.rs:271:9
console-pageserver-1 | std::panicking::try::do_call
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
console-pageserver-1 | std::panicking::try
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
console-pageserver-1 | std::panic::catch_unwind
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
console-pageserver-1 | std::thread::Builder::spawn_unchecked::{{closure}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:482:30
console-pageserver-1 | core::ops::function::FnOnce::call_once{{vtable.shim}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
console-pageserver-1 | 9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
console-pageserver-1 | <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
console-pageserver-1 | std::sys::unix::thread::Thread::new::thread_start
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys/unix/thread.rs:106:17
console-pageserver-1 | 10: start_thread
console-pageserver-1 | 11: clone
console-pageserver-1 | 2022-01-19T10:17:53.733815Z INFO pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: directory "/data/tenants/70e94d4b4ebac68cef2325fd3dcbdbc5/wal-redo-datadir" exists, removing
console-pageserver-1 | 2022-01-19T10:17:53.834243Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:53.909149Z WARN {tid=23}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor1-1 | 2022-01-19T10:17:53.908445Z WARN {tid=35}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:53.908104Z WARN {tid=31}: query handler for START_WAL_PUSH postgresql://(null):@host.docker.internal:6400 failed: failed to run ReceiveWalConn: connection closed unexpectedly
console-wal_acceptor2-1 | 2022-01-19T10:17:54.269611Z INFO callmemaybe. thread_main. unsubscribe callback. request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:54.269677Z INFO callmemaybe. thread_main. resume callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor2-1 | 2022-01-19T10:17:54.269705Z WARN {tid=29}: query handler for START_REPLICATION PHYSICAL 0/169B730 failed: failed to run ReplicationConn: end streaming to Some("pageserver")
console-pageserver-1 | connection error: connection closed
console-pageserver-1 | 2022-01-19T10:17:54.271607Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL streaming connection failed (connection closed)
console-pageserver-1 | 2022-01-19T10:17:54.333260Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:54.583200Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:54.591604Z INFO new call spawned. time Instant { tv_sec: 123198, tv_nsec: 498686217 }
console-wal_acceptor1-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor1-1 | 2022-01-19T10:17:54.591843Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:54.597680Z INFO callmemaybe{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: activating tenant 70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:54.598027Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL receiver thread started
console-pageserver-1 | 2022-01-19T10:17:54.598072Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connecting to "host=wal_acceptor1 port=6500 options='-c ztimelineid=04bef3374e6c8e4d2fedc5cb9778f860 ztenantid=70e94d4b4ebac68cef2325fd3dcbdbc5'"
console-wal_acceptor1-1 | 2022-01-19T10:17:54.606634Z INFO {tid=38}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/169BC78 till None
console-pageserver-1 | 2022-01-19T10:17:54.602828Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connected!
console-pageserver-1 | 2022-01-19T10:17:54.604263Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: IdentifySystem { systemid: 7054859081854792284, timeline: 1, xlogpos: 0/169BC78, dbname: None }
console-pageserver-1 | 2022-01-19T10:17:54.604333Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: last_record_lsn 0/169BC78 starting replication from 0/169BC78, server is at 0/169BC78...
console-wal_acceptor1-1 | 2022-01-19T10:17:54.606856Z INFO callmemaybe. thread_main. pause callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:54.669492Z INFO new call spawned. time Instant { tv_sec: 123198, tv_nsec: 576575467 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:54.669583Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:54.835859Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:55.011347Z INFO pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: running initdb in "/data/tenants/70e94d4b4ebac68cef2325fd3dcbdbc5/wal-redo-datadir"
console-pageserver-1 | 2022-01-19T10:17:55.334877Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:55.586547Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:55.669797Z INFO new call spawned. time Instant { tv_sec: 123199, tv_nsec: 576879467 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:55.670700Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:55.837154Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:56.336475Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:56.589477Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor1-1 | 2022-01-19T10:17:56.607291Z INFO callmemaybe. thread_main. unsubscribe callback. request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:56.607637Z INFO callmemaybe. thread_main. resume callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor1-1 | 2022-01-19T10:17:56.607663Z WARN {tid=38}: query handler for START_REPLICATION PHYSICAL 0/169BC78 failed: failed to run ReplicationConn: end streaming to Some("pageserver")
console-pageserver-1 | connection error: connection closed
console-pageserver-1 | 2022-01-19T10:17:56.610106Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL streaming connection failed (connection closed)
console-pageserver-1 | 2022-01-19T10:17:56.839598Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:57.337665Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:57.590691Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:57.669587Z INFO new call spawned. time Instant { tv_sec: 123201, tv_nsec: 576670927 }
console-wal_acceptor3-1 | tenantid: 70e94d4b4ebac68cef2325fd3dcbdbc5, timelineid: 04bef3374e6c8e4d2fedc5cb9778f860
console-wal_acceptor3-1 | 2022-01-19T10:17:57.669686Z INFO callmemaybe request_callback Connecting to pageserver postgresql://(null):@host.docker.internal:6400
console-pageserver-1 | 2022-01-19T10:17:57.682264Z INFO callmemaybe{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: activating tenant 70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:57.682379Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL receiver thread started
console-pageserver-1 | 2022-01-19T10:17:57.682439Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connecting to "host=wal_acceptor3 port=6500 options='-c ztimelineid=04bef3374e6c8e4d2fedc5cb9778f860 ztenantid=70e94d4b4ebac68cef2325fd3dcbdbc5'"
console-pageserver-1 | 2022-01-19T10:17:57.687994Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: connected!
console-pageserver-1 | 2022-01-19T10:17:57.688407Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: IdentifySystem { systemid: 7054859081854792284, timeline: 1, xlogpos: 0/169BC78, dbname: None }
console-pageserver-1 | 2022-01-19T10:17:57.688463Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: last_record_lsn 0/169BC78 starting replication from 0/169BC78, server is at 0/169BC78...
console-wal_acceptor3-1 | 2022-01-19T10:17:57.692137Z INFO {tid=24}:WAL sender{timeline=04bef3374e6c8e4d2fedc5cb9778f860}: Start replication from 0/169BC78 till None
console-wal_acceptor3-1 | 2022-01-19T10:17:57.692538Z INFO callmemaybe. thread_main. pause callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-pageserver-1 | 2022-01-19T10:17:57.840863Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:58.339093Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:58.592518Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:58.842367Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:59.340571Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:17:59.594053Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-wal_acceptor3-1 | 2022-01-19T10:17:59.693413Z INFO callmemaybe. thread_main. unsubscribe callback. request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:59.693493Z INFO callmemaybe. thread_main. resume callback request for timelineid=04bef3374e6c8e4d2fedc5cb9778f860 tenantid=70e94d4b4ebac68cef2325fd3dcbdbc5
console-wal_acceptor3-1 | 2022-01-19T10:17:59.693525Z WARN {tid=24}: query handler for START_REPLICATION PHYSICAL 0/169BC78 failed: failed to run ReplicationConn: end streaming to Some("pageserver")
console-pageserver-1 | connection error: connection closed
console-pageserver-1 | 2022-01-19T10:17:59.694523Z INFO WAL receiver{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: WAL streaming connection failed (connection closed)
console-pageserver-1 | 2022-01-19T10:17:59.844031Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:00.341656Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:00.597198Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:00.845264Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:01.342844Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:01.598468Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:01.846697Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:02.343996Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:02.599776Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:02.848017Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:03.345138Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:03.600959Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:03.848900Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:04.345750Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:04.603378Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:04.850224Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:05.347307Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:05.604680Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:05.851348Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:06.349183Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:06.606045Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:06.853103Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:07.350487Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:07.607872Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:07.854257Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:08.351752Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:08.609476Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:08.855722Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:09.352989Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:09.611216Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:09.856809Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:10.354344Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:10.615028Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:10.858255Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:11.355621Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:11.616081Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:11.859457Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:12.356789Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:12.617336Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:12.860782Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:13.358165Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:13.618654Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:13.862020Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:14.195090Z INFO pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: launched WAL redo postgres process on "/data/tenants/70e94d4b4ebac68cef2325fd3dcbdbc5/wal-redo-datadir"
console-pageserver-1 | 2022-01-19T10:18:14.288706Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: wal-redo-postgres: 2022-01-19 10:18:14.287 UTC [861] FATAL: seccomp: failed to load a test filter
console-pageserver-1 |
console-pageserver-1 | 2022-01-19T10:18:14.295339Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}:get_page{rel=1664/0/1260 blkno=0 req_lsn=0/169BC08}: wal-redo-postgres exited with code exit status: 1
console-pageserver-1 | 2022-01-19T10:18:14.295776Z ERROR pagestream{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: error reading relation or page version: WAL redo process closed its stderr unexpectedly
console-pageserver-1 |
console-pageserver-1 | Stack backtrace:
console-pageserver-1 | 0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/result.rs:1914:27
console-pageserver-1 | pageserver::layered_repository::LayeredTimeline::reconstruct_page
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:2039:27
console-pageserver-1 | pageserver::layered_repository::LayeredTimeline::materialize_page
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:1981:9
console-pageserver-1 | 1: <pageserver::layered_repository::LayeredTimeline as pageserver::repository::Timeline>::get_page_at_lsn::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:809:46
console-pageserver-1 | prometheus::histogram::Histogram::observe_closure_duration
console-pageserver-1 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/prometheus-0.12.0/src/histogram.rs:715:19
console-pageserver-1 | <pageserver::layered_repository::LayeredTimeline as pageserver::repository::Timeline>::get_page_at_lsn
console-pageserver-1 | at /zenith/pageserver/src/layered_repository.rs:808:13
console-pageserver-1 | 2: pageserver::page_service::PageServerHandler::handle_get_page_at_lsn_request
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:487:20
console-pageserver-1 | 3: pageserver::page_service::PageServerHandler::handle_pagerequests::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:361:37
console-pageserver-1 | prometheus::histogram::Histogram::observe_closure_duration
console-pageserver-1 | at /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/prometheus-0.12.0/src/histogram.rs:715:19
console-pageserver-1 | pageserver::page_service::PageServerHandler::handle_pagerequests
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:358:66
console-pageserver-1 | <pageserver::page_service::PageServerHandler as zenith_utils::postgres_backend::Handler>::process_query
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:597:13
console-pageserver-1 | 4: zenith_utils::postgres_backend::PostgresBackend::process_message
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:434:33
console-pageserver-1 | zenith_utils::postgres_backend::PostgresBackend::run_message_loop
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:283:31
console-pageserver-1 | zenith_utils::postgres_backend::PostgresBackend::run
console-pageserver-1 | at /zenith/zenith_utils/src/postgres_backend.rs:265:19
console-pageserver-1 | 5: pageserver::page_service::page_service_conn_main
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:282:5
console-pageserver-1 | 6: pageserver::page_service::thread_main::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/page_service.rs:234:29
console-pageserver-1 | <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panic/unwind_safe.rs:271:9
console-pageserver-1 | std::panicking::try::do_call
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
console-pageserver-1 | std::panicking::try
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
console-pageserver-1 | std::panic::catch_unwind
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
console-pageserver-1 | pageserver::thread_mgr::thread_wrapper
console-pageserver-1 | at /zenith/pageserver/src/thread_mgr.rs:201:18
console-pageserver-1 | 7: pageserver::thread_mgr::spawn::{{closure}}
console-pageserver-1 | at /zenith/pageserver/src/thread_mgr.rs:165:24
console-pageserver-1 | std::sys_common::backtrace::__rust_begin_short_backtrace
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys_common/backtrace.rs:123:18
console-pageserver-1 | 8: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:483:17
console-pageserver-1 | <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panic/unwind_safe.rs:271:9
console-pageserver-1 | std::panicking::try::do_call
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:403:40
console-pageserver-1 | std::panicking::try
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:367:19
console-pageserver-1 | std::panic::catch_unwind
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panic.rs:133:14
console-pageserver-1 | std::thread::Builder::spawn_unchecked::{{closure}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/thread/mod.rs:482:30
console-pageserver-1 | core::ops::function::FnOnce::call_once{{vtable.shim}}
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
console-pageserver-1 | 9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
console-pageserver-1 | <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/alloc/src/boxed.rs:1691:9
console-pageserver-1 | std::sys::unix::thread::Thread::new::thread_start
console-pageserver-1 | at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/sys/unix/thread.rs:106:17
console-pageserver-1 | 10: start_thread
console-pageserver-1 | 11: clone
console-pageserver-1 | 2022-01-19T10:18:14.360203Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:14.621301Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:14.863139Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:15.365375Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:15.643866Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:15.868796Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:16.389999Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:16.658287Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:16.872745Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:17.406658Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:17.677413Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:17.883859Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:18.416039Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:18.695411Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:18.903140Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:19.430414Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:19.712739Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:19.916345Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:20.443598Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:20.731517Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:20.938808Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:21.456775Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:21.752117Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:21.957155Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:22.475328Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:22.768485Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:22.963875Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:23.490400Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:23.789207Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:23.983309Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:24.517504Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:24.802836Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:24.992413Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:25.525903Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:25.832881Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:26.004159Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:26.533011Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:26.855966Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:27.018415Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:27.548099Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:27.866674Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:28.025783Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:28.566701Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:28.891679Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:29.034886Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:29.579759Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:29.908881Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:30.044309Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:30.591363Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:30.924769Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:31.071032Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:31.601261Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:31.943516Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:32.090389Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:32.609951Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:32.953797Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:33.109052Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:33.618212Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:33.955970Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:34.114781Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:34.625270Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:34.975859Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:35.121906Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:35.642070Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:35.999426Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:36.142164Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:36.661805Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:37.009403Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:37.154136Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:37.676905Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:38.030501Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:38.168486Z INFO checkpoint{timeline=04bef3374e6c8e4d2fedc5cb9778f860 tenant=70e94d4b4ebac68cef2325fd3dcbdbc5}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:38.697793Z INFO checkpoint{timeline=23d2942acd9bddbf253986b91a20423a tenant=d32c66359f09e8b4682a55f67931a446}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
console-pageserver-1 | 2022-01-19T10:18:39.045523Z INFO checkpoint{timeline=4170ddc889f52a235088aaf54f76efb7 tenant=1b21f903831296fa464d461fe2f17200}: the oldest layer is now inmem-pg_control_checkpoint_0_0000000001695D11_FFFFFFFFFFFFFFFF which is 24336 bytes behind last_record_lsn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment