This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| m ~/D/zenithdb kubectl logs zenith-console-periodic-e2e-test-27526284-gm7xm -f | |
| 2022/05/03 11:39:16 Waiting for: https://console.stage.neon.tech/healthz | |
| 2022/05/03 11:39:16 Received 200 from https://console.stage.neon.tech/healthz | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /usr/local/bin/python | |
| cachedir: /tmp/.pytest_cache | |
| rootdir: /tests_e2e, configfile: ../etc/e2e/pytest.ini | |
| plugins: anyio-3.5.0, asyncio-0.18.3, order-1.0.1, timeout-2.1.0 | |
| asyncio: mode=auto | |
| collecting ... collected 57 items / 13 deselected / 44 selected |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| m ~/D/zenithdb kubectl logs zenith-console-periodic-e2e-test-27526284-gm7xm -f | |
| 2022/05/03 11:39:16 Waiting for: https://console.stage.neon.tech/healthz | |
| 2022/05/03 11:39:16 Received 200 from https://console.stage.neon.tech/healthz | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.9.12, pytest-7.1.1, pluggy-1.0.0 -- /usr/local/bin/python | |
| cachedir: /tmp/.pytest_cache | |
| rootdir: /tests_e2e, configfile: ../etc/e2e/pytest.ini | |
| plugins: anyio-3.5.0, asyncio-0.18.3, order-1.0.1, timeout-2.1.0 | |
| asyncio: mode=auto | |
| collecting ... collected 57 items / 13 deselected / 44 selected |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| m ~/D/zenithdb tsh ssh admin@zenith-us-stage-ps-2 'journalctl -u pageserver.service --since "2022-05-03 07:02:03" -b | grep "2022-05-03T07:02:04.435331Z" -m1 -A1000' | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2022-05-03T07:02:04.435331Z INFO version: git-env:local | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2022-05-03T07:02:04.435376Z INFO Starting pageserver http handler on 0.0.0.0:9898 | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2022-05-03T07:02:04.436235Z INFO Starting pageserver pg protocol handler on 0.0.0.0:6400 | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2022-05-03T07:02:04.436293Z INFO Using auth: Trust | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2022-05-03T07:02:04.459834Z INFO Using s3 bucket 'zenith-staging-storage-us-east-1' in region 'us-east-1' as a remote storage, prefix in bucket: 'Some("zenith-us-stage-ps-2")', bucket endpoint: 'None' | |
| May 03 07:02:04 zenith-us-stage-ps-2.local pageserver[18268]: 2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: thread 'Compactor thread' panicked at 'unsorted input', pageserver/src/layered_repository/disk_btree.rs:448:13 | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: stack backtrace: | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: 0: rust_begin_unwind | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5 | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: 1: core::panicking::panic_fmt | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14 | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: 2: pageserver::layered_repository::disk_btree::DiskBtreeBuilder<W,_>::append | |
| May 02 15:03:14 zenith-us-stage-ps-2.local pageserver[13648]: at /home/circleci/project/pageserver/src/layered_r |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| admin@zenith-us-stage-ps-2:~$ journalctl -u pageserver.service --since "2022-04-30 16:43:45" | grep "serving Page Service thread" -m1 -A50 | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: 2022-04-30T16:43:50.873790Z ERROR Thread 'serving Page Service thread' exited with error: Connection reset by peer (os error 104) | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: Stack backtrace: | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: 0: <T as core::convert::Into<U>>::into | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/convert/mod.rs:542:9 | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: utils::pq_proto::FeMessage::read_fut::{{closure}} | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: at /home/circleci/project/libs/utils/src/pq_proto.rs:162:38 | |
| Apr 30 16:43:50 zenith-us-stage-ps-2.local pageserver[17430]: <core::future::from_gener |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| admin@zenith-us-stage-ps-2:~$ journalctl -u pageserver.service --since "2022-04-30 16:43:45" | head -20 | |
| -- Journal begins at Sat 2022-04-30 02:43:37 UTC, ends at Sat 2022-04-30 16:47:16 UTC. -- | |
| Apr 30 16:43:45 zenith-us-stage-ps-2.local systemd[1]: Stopping Zenith pageserver... | |
| Apr 30 16:43:45 zenith-us-stage-ps-2.local pageserver[6090]: 2022-04-30T16:43:45.138955Z INFO Got SIGINT. Terminating gracefully in fast shutdown mode | |
| Apr 30 16:43:45 zenith-us-stage-ps-2.local pageserver[6090]: 2022-04-30T16:43:45.140978Z INFO storage_sync_loop_step: Processing tasks for 1 timelines in batch, more tasks left to process: 0 | |
| Apr 30 16:43:45 zenith-us-stage-ps-2.local pageserver[6090]: 2022-04-30T16:43:45.141027Z INFO storage_sync_loop_step:process_sync_tasks{sync_id=407ea16bac8dd7d4eb972191dd04bbe8-f89660a96807d594cd33c455b80a8737}: Sync task 'upload' processing started, attempt #0 | |
| Apr 30 16:43:45 zenith-us-stage-ps-2.local pageserver[6090]: 2022-04-30T16:43:45.141043Z INFO storage_sync_loop_step:process_sync_tasks{s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Journal begins at Fri 2022-04-29 01:35:33 UTC, ends at Fri 2022-04-29 12:31:41 UTC. -- | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710]: 2022-04-29T06:53:00.390552Z ERROR query handler for 'callmemaybe d8f4b6341b38d06998a9ff421f8f1855 c0d144751e8cec2b46d1b2038bf1c942 host=zenith-us-stage-sk-2.local port=6500 options='-c ztimelineid=c0d144751e8cec2b46d1b2038bf1c942 ztenantid=d8f4b6341b38d06998a9ff421f8f1855 pageserver_connstr=postgresql://(null):@zenith-us-stage-ps-2.local:6400'' failed: Cannot load local timeline | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710]: Caused by: | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710]: deserialize error | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710]: Stack backtrace: | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710]: 0: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual | |
| Apr 29 06:53:00 zenith-us-stage-ps-2.local pageserver[9710] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Journal begins at Fri 2022-04-29 01:35:33 UTC, ends at Fri 2022-04-29 12:21:50 UTC. -- | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local systemd[1]: Started Zenith pageserver. | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 2022-04-29T12:21:14.514471Z INFO version: git-env:local | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 2022-04-29T12:21:14.514506Z INFO Starting pageserver http handler on 0.0.0.0:9898 | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 2022-04-29T12:21:14.514560Z INFO Starting pageserver pg protocol handler on 0.0.0.0:6400 | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 2022-04-29T12:21:14.514614Z INFO Using auth: Trust | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 2022-04-29T12:21:14.521053Z INFO Using s3 bucket 'zenith-staging-storage-us-east-1' in region 'us-east-1' as a remote storage, prefix in bucket: 'Some("zenith-us-stage-ps-2")', bucket endpoint: 'None' | |
| Apr 29 12:21:14 zenith-us-stage-ps-2.local pageserver[3923]: 20 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| m ~/D/zenithdb kubectl logs release-test-8qwq2 > tmp/e2e_logs`date +%s`.log | |
| m ~/D/zenithdb cat tmp/e2e_logs1651225326.log | |
| 2022/04/29 08:13:07 Waiting for: https://console.stage.neon.tech/healthz | |
| 2022/04/29 08:13:07 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:08 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:09 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:10 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:11 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:12 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/29 08:13:13 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2022/04/25 16:18:36 Waiting for: https://console.stage.neon.tech/healthz | |
| 2022/04/25 16:18:36 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:37 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:38 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:39 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:40 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:41 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:42 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:43 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s | |
| 2022/04/25 16:18:44 Received 503 from https://console.stage.neon.tech/healthz. Sleeping 1s |