Skip to content

Instantly share code, notes, and snippets.

@theobolo

theobolo/solana-2.2.3-turbine-panic.log

Created March 19, 2025 16:20
Show Gist options
  • Save theobolo/9e0779f9ba35761eafe5fe427748851d to your computer and use it in GitHub Desktop.
Save theobolo/9e0779f9ba35761eafe5fe427748851d to your computer and use it in GitHub Desktop.
Download ZIP
solana-2.2.3-turbine-panic
Raw
solana-2.2.3-turbine-panic.log
[INFO] Configuring Solana cluster...
Config File: /root/.config/solana/cli/config.yml
RPC URL: https://api.devnet.solana.com
WebSocket URL: wss://api.devnet.solana.com/ (computed)
Keypair Path: /root/.config/solana/id.json
Commitment: confirmed
[INFO] Existing junk identity key found: /root/.config/solana/junk.json
[INFO] Updating package repositories and installing dependencies...
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../00-libnghttp2-14_1.43.0-1ubuntu0.2_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.43.0-1ubuntu0.2) ...
Selecting previously unselected package libpsl5:amd64.
Preparing to unpack .../01-libpsl5_0.21.0-1.2build2_amd64.deb ...
Unpacking libpsl5:amd64 (0.21.0-1.2build2) ...
Selecting previously unselected package publicsuffix.
Preparing to unpack .../02-publicsuffix_20211207.1025-1_all.deb ...
Unpacking publicsuffix (20211207.1025-1) ...
Selecting previously unselected package libbrotli1:amd64.
Preparing to unpack .../03-libbrotli1_1.0.9-2build6_amd64.deb ...
Unpacking libbrotli1:amd64 (1.0.9-2build6) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../04-libsasl2-modules-db_2.1.27+dfsg2-3ubuntu1.2_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../05-libsasl2-2_2.1.27+dfsg2-3ubuntu1.2_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Selecting previously unselected package libldap-2.5-0:amd64.
Preparing to unpack .../06-libldap-2.5-0_2.5.18+dfsg-0ubuntu0.22.04.3_amd64.deb ...
Unpacking libldap-2.5-0:amd64 (2.5.18+dfsg-0ubuntu0.22.04.3) ...
Selecting previously unselected package librtmp1:amd64.
Preparing to unpack .../07-librtmp1_2.4+20151223.gitfa8646d.1-2build4_amd64.deb ...
Unpacking librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build4) ...
Selecting previously unselected package libssh-4:amd64.
Preparing to unpack .../08-libssh-4_0.9.6-2ubuntu0.22.04.3_amd64.deb ...
Unpacking libssh-4:amd64 (0.9.6-2ubuntu0.22.04.3) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../09-libcurl4_7.81.0-1ubuntu1.20_amd64.deb ...
Unpacking libcurl4:amd64 (7.81.0-1ubuntu1.20) ...
Selecting previously unselected package curl.
Preparing to unpack .../10-curl_7.81.0-1ubuntu1.20_amd64.deb ...
Unpacking curl (7.81.0-1ubuntu1.20) ...
Selecting previously unselected package libonig5:amd64.
Preparing to unpack .../11-libonig5_6.9.7.1-2build1_amd64.deb ...
Unpacking libonig5:amd64 (6.9.7.1-2build1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../12-libjq1_1.6-2.1ubuntu3_amd64.deb ...
Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ...
Selecting previously unselected package jq.
Preparing to unpack .../13-jq_1.6-2.1ubuntu3_amd64.deb ...
Unpacking jq (1.6-2.1ubuntu3) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../14-libldap-common_2.5.18+dfsg-0ubuntu0.22.04.3_all.deb ...
Unpacking libldap-common (2.5.18+dfsg-0ubuntu0.22.04.3) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../15-libsasl2-modules_2.1.27+dfsg2-3ubuntu1.2_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Setting up libpsl5:amd64 (0.21.0-1.2build2) ...
Setting up libbrotli1:amd64 (1.0.9-2build6) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Setting up libnghttp2-14:amd64 (1.43.0-1ubuntu0.2) ...
Setting up libldap-common (2.5.18+dfsg-0ubuntu0.22.04.3) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Setting up librtmp1:amd64 (2.4+20151223.gitfa8646d.1-2build4) ...
Setting up libsasl2-2:amd64 (2.1.27+dfsg2-3ubuntu1.2) ...
Setting up libssh-4:amd64 (0.9.6-2ubuntu0.22.04.3) ...
Setting up publicsuffix (20211207.1025-1) ...
Setting up libonig5:amd64 (6.9.7.1-2build1) ...
Setting up libjq1:amd64 (1.6-2.1ubuntu3) ...
Setting up libldap-2.5-0:amd64 (2.5.18+dfsg-0ubuntu0.22.04.3) ...
Setting up jq (1.6-2.1ubuntu3) ...
Setting up libcurl4:amd64 (7.81.0-1ubuntu1.20) ...
Setting up curl (7.81.0-1ubuntu1.20) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
[INFO] Installing kubectl...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
[INFO] kubectl installed successfully.
[INFO] Starting the identity sentinel script...
[INFO] Command-line arguments: agave-validator --entrypoint=entrypoint.devnet.solana.com:8001 --entrypoint=entrypoint2.devnet.solana.com:8001 --entrypoint=entrypoint3.devnet.solana.com:8001 --known-validator=dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 --known-validator=dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV --known-validator=dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB --known-validator=dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS --expected-genesis-hash=EtWTRABZaYq6iMfeYKouRu166VU2xqa1wcaWoxPkrZBG --accounts-index-path=/index/accounts_index --accounts-hash-cache-path=/index/accounts_hash_cache --accounts-index-memory-limit-mb=40000 --account-index=program-id --account-index=spl-token-owner --account-index=spl-token-mint --account-index-include-key=AddressLookupTab1e1111111111111111111111111 --rpc-port=8899 --dynamic-port-range=25000-25100 --identity=/root/.config/solana/junk.json --vote-account=/vault/secrets/vote-account-keypair.json --authorized-voter=/vault/secrets/validator-keypair.json --wal-recovery-mode=skip_any_corrupted_record --private-rpc --rpc-bind-address=0.0.0.0 --accounts=/accounts --ledger=/ledger --snapshots=/snapshots --snapshot-archive-format=lz4 --log=- --block-verification-method=unified-scheduler --disable-banking-trace --limit-ledger-size=50000000 --maximum-snapshot-download-abort=5 --minimal-snapshot-download-speed=50000000 --no-os-network-limits-test=true --replay-slots-concurrently
[SENTINEL.SH] [2025-03-19T16:03:52Z] [INIT] Starting Sentinel...
[SENTINEL.SH] [2025-03-19T16:03:52Z] [INFO] Creating or updating Secret devnet-solana-validator-kiln-tower-secret.
[2025-03-19T16:03:52.746234162Z INFO agave_validator] agave-validator 2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave)
[2025-03-19T16:03:52.746259701Z INFO agave_validator] Starting validator with: ArgsOs {
inner: [
"agave-validator",
"--entrypoint=entrypoint.devnet.solana.com:8001",
"--entrypoint=entrypoint2.devnet.solana.com:8001",
"--entrypoint=entrypoint3.devnet.solana.com:8001",
"--known-validator=dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92",
"--known-validator=dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV",
"--known-validator=dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB",
"--known-validator=dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS",
"--expected-genesis-hash=EtWTRABZaYq6iMfeYKouRu166VU2xqa1wcaWoxPkrZBG",
"--accounts-index-path=/index/accounts_index",
"--accounts-hash-cache-path=/index/accounts_hash_cache",
"--accounts-index-memory-limit-mb=40000",
"--account-index=program-id",
"--account-index=spl-token-owner",
"--account-index=spl-token-mint",
"--account-index-include-key=AddressLookupTab1e1111111111111111111111111",
"--rpc-port=8899",
"--dynamic-port-range=25000-25100",
"--identity=/root/.config/solana/junk.json",
"--vote-account=/vault/secrets/vote-account-keypair.json",
"--authorized-voter=/vault/secrets/validator-keypair.json",
"--wal-recovery-mode=skip_any_corrupted_record",
"--private-rpc",
"--rpc-bind-address=0.0.0.0",
"--accounts=/accounts",
"--ledger=/ledger",
"--snapshots=/snapshots",
"--snapshot-archive-format=lz4",
"--log=-",
"--block-verification-method=unified-scheduler",
"--disable-banking-trace",
"--limit-ledger-size=50000000",
"--maximum-snapshot-download-abort=5",
"--minimal-snapshot-download-speed=50000000",
"--no-os-network-limits-test=true",
"--replay-slots-concurrently",
],
}
[2025-03-19T16:03:52.746291811Z WARN solana_perf] CUDA is disabled
[2025-03-19T16:03:52.746294721Z INFO solana_perf] AVX detected
[2025-03-19T16:03:52.746296781Z INFO solana_perf] AVX2 detected
secret/devnet-solana-validator-kiln-tower-secret unchanged
[SENTINEL.SH] [2025-03-19T16:03:52Z] [SUCCESS] Secret devnet-solana-validator-kiln-tower-secret created or updated.
[SENTINEL.SH] [2025-03-19T16:03:52Z] [INFO] ConfigMap 'devnet-solana-validator-kiln-state' already exists.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] ConfigMap 'devnet-solana-validator-kiln-0-a-ha-config' already exists.
[2025-03-19T16:03:53.252878842Z INFO agave_validator] obtained shred-version 2405 from 145.40.126.137:8001
[2025-03-19T16:03:53.254235491Z INFO agave_validator] Snapshot configuration: full snapshot interval: 25000 slots, incremental snapshot interval: 100 slots
[2025-03-19T16:03:53.254613674Z INFO agave_validator] Contacting 145.40.68.1:8001 to determine the validator's public IP address
[2025-03-19T16:03:53.255208965Z INFO agave_validator::admin_rpc_service] started admin rpc service!
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] current_active: 'true', current_promote_force: 'true', active_validator: ''
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] Attempting to acquire lock 'devnet-solana-validator-kiln-tower-lock'.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] Lock 'devnet-solana-validator-kiln-tower-lock' acquired by 'devnet-solana-validator-kiln-0-a-0'.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [STATE] Forced Promotion.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [ACTION] Starting Promotion
[SENTINEL.SH] [2025-03-19T16:03:53Z] [ACTION] Proceeding with Promotion.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] promoteForce: 'true', tower_ready: 'false'
[SENTINEL.SH] [2025-03-19T16:03:53Z] [WARNING] Force promotion active.
[SENTINEL.SH] [2025-03-19T16:03:53Z] [STEP] Checking validator readiness
[SENTINEL.SH] [2025-03-19T16:03:53Z] [INFO] Waiting for validator readiness (5 consecutive 'ok' responses needed).
[SENTINEL.SH] [2025-03-19T16:03:53Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:03:56Z] [WAIT] Health check failed. Retrying in 3s.
Failed to contact cluster entrypoint 145.40.68.1:8001: deadline has elapsed
[2025-03-19T16:03:58.255957162Z INFO agave_validator] Contacting 145.40.126.137:8001 to determine the validator's public IP address
[2025-03-19T16:03:58.775402093Z INFO solana_metrics::metrics] host id: BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar
[2025-03-19T16:03:58.775435112Z INFO solana_entry::entry] Loading "libpoh-simd.so"
[2025-03-19T16:03:58.775499671Z INFO solana_perf::perf_libs] perf-libs found at "/usr/bin/perf-libs"
[2025-03-19T16:03:58.775503331Z INFO solana_perf::perf_libs] setting ld_library_path to: "/usr/bin/perf-libs"
[2025-03-19T16:03:58.775711138Z INFO solana_net_utils::ip_echo_client] Checking that udp ports [25000, 25001, 25003, 25004, 25005, 25007, 25008, 25009, 25010, 25012, 25014] are reachable from 35.197.53.105:8001
[2025-03-19T16:03:59.361960170Z INFO solana_net_utils::ip_echo_client] checked udp ports: [25000, 25001, 25003, 25004], reachable udp ports: {25003, 25000, 25004, 25001}
[SENTINEL.SH] [2025-03-19T16:03:59Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:03:59.941870723Z INFO solana_net_utils::ip_echo_client] checked udp ports: [25005, 25007, 25008, 25009], reachable udp ports: {25005, 25008, 25007, 25009}
[2025-03-19T16:04:00.513897064Z INFO solana_net_utils::ip_echo_client] checked udp ports: [25010, 25012, 25014], reachable udp ports: {25012, 25014, 25010}
[2025-03-19T16:04:00.515219293Z INFO solana_net_utils::ip_echo_client] Checking that tcp ports [25000] are reachable from 35.197.53.105:8001
[2025-03-19T16:04:00.990901163Z INFO solana_net_utils::ip_echo_client] tcp/25000 is reachable
[2025-03-19T16:04:00.992056645Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:04:01.993730982Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 40 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 153 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 40 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 80 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 40 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 43 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 112 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 379 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 312 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 314 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 193 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 312 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 112 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 784 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 180 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 713 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 40 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 708 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 111 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 314 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 112 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 153 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:04:01.993745732Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405...
[2025-03-19T16:04:01.993758882Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 0 blacklisted
[SENTINEL.SH] [2025-03-19T16:04:02Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:02.993977366Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 312 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 170 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 141 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 113 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 112 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 112 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 170 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 141 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 350 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 511 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 350 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 298 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 298 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 112 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 911 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 171 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 350 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 148 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 658 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 141 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 511 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 112 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:04:02.993996766Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:04:02.994006096Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 0 blacklisted
[2025-03-19T16:04:03.009258344Z INFO solana_metrics::metrics] metrics configuration: host=https://metrics.solana.com:8086 db=devnet username=scratch_writer
[2025-03-19T16:04:03.009444321Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=1i entrypoint2=39i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=6408i purged_values_size=105i failed_inserts_size=5i num_nodes=176i num_nodes_staked=0i num_pubkeys=176i
[2025-03-19T16:04:03.009468931Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=223i new_push_requests2=0i purge=1321i purge_count=0i process_gossip_packets_time=26108i verify_gossip_packets_time=104869i handle_batch_ping_messages_time=1844i handle_batch_pong_messages_time=338i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=0i handle_batch_pull_responses_time=13076i handle_batch_push_messages_time=0i process_pull_resp=10699i filter_pull_resp=2223i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=0i process_pull_resp_count=161i pull_response_fail_insert=5i pull_response_fail_timeout=0i pull_response_success=1087i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:04:03.009482741Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=6342i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=0i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=0i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=6267i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=0i gossip_transmit_loop_time=7827i gossip_transmit_loop_iterations_since_last_report=21i gossip_listen_loop_time=1923297i gossip_listen_loop_iterations_since_last_report=288i
[2025-03-19T16:04:03.009485771Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=515i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=0i pull_from_entrypoint_count=1i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=0i
[2025-03-19T16:04:03.009488091Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=9i packets_received_count=6906i packets_received_ping_messages_count=74i packets_received_pong_messages_count=109i packets_received_prune_messages_count=0i packets_received_pull_requests_count=0i packets_received_pull_responses_count=6722i packets_received_push_messages_count=0i packets_received_unknown_count=1i packets_received_verified_count=6905i packets_sent_gossip_requests_count=337i packets_sent_ping_messages_count=175i packets_sent_pong_messages_count=74i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=88i packets_sent_pull_responses_count=0i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:04:03.009493441Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=0i LegacyContactInfo-pull=33i Vote-push=0i Vote-pull=371i LowestSlot-push=0i LowestSlot-pull=66i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=5i EpochSlots-push=0i EpochSlots-pull=5005i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=33i NodeInstance-push=0i NodeInstance-pull=35i DuplicateShred-push=0i DuplicateShred-pull=649i SnapshotHashes-push=0i SnapshotHashes-pull=65i ContactInfo-push=0i ContactInfo-pull=75i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=6337i
[2025-03-19T16:04:03.009497761Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=0i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=0i
[2025-03-19T16:04:03.994211637Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 169 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 169 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 305 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 192 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 169 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 170 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 982 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 169 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 192 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 778 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 1351 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 776 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 192 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 1113 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 778 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 282 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 169 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 582 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 776 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 982 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 169 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 1113 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 28 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:04:03.994227186Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:04:03.994237166Z INFO agave_validator::bootstrap] Total 23 RPC nodes found. 4 known, 0 blacklisted
[2025-03-19T16:04:03.994261206Z INFO agave_validator::bootstrap] Highest available snapshot slot is 368428352, available from 17 nodes: [dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV, dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS, 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs, CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i, dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB, cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh, FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho, AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC, 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z, DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N, DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu, Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA, G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX, 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8, 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw, 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5, dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92]
[2025-03-19T16:04:03.995237360Z INFO agave_validator::bootstrap] Using RPC service from node 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs: Some(38.46.221.182:8899)
[2025-03-19T16:04:03.995253710Z INFO agave_validator::bootstrap] Using RPC service from node 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8: Some(147.28.133.51:6699)
[2025-03-19T16:04:03.995261930Z INFO agave_validator::bootstrap] Using RPC service from node Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA: Some(38.97.60.6:8899)
[2025-03-19T16:04:03.995258470Z INFO agave_validator::bootstrap] Using RPC service from node cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh: Some(52.13.43.37:8899)
[2025-03-19T16:04:03.995260720Z INFO agave_validator::bootstrap] Using RPC service from node 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z: Some(64.46.114.58:21811)
[2025-03-19T16:04:03.995260860Z INFO agave_validator::bootstrap] Using RPC service from node dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB: Some(147.28.165.81:8899)
[2025-03-19T16:04:03.995262710Z INFO agave_validator::bootstrap] Using RPC service from node DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N: Some(63.251.232.229:21811)
[2025-03-19T16:04:03.995262280Z INFO agave_validator::bootstrap] Using RPC service from node DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu: Some(63.251.106.87:21811)
[2025-03-19T16:04:03.995272650Z INFO agave_validator::bootstrap] Using RPC service from node dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV: Some(147.28.198.57:8899)
[2025-03-19T16:04:03.995262370Z INFO agave_validator::bootstrap] Using RPC service from node FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho: Some(141.94.154.228:8899)
[2025-03-19T16:04:03.995278900Z INFO agave_validator::bootstrap] Using RPC service from node 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5: Some(107.6.94.200:21811)
[2025-03-19T16:04:03.995281969Z INFO agave_validator::bootstrap] Using RPC service from node CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i: Some(34.220.208.242:8899)
[2025-03-19T16:04:03.995294199Z INFO agave_validator::bootstrap] Using RPC service from node G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX: Some(148.113.206.128:8899)
[2025-03-19T16:04:03.995293789Z INFO agave_validator::bootstrap] Using RPC service from node dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92: Some(145.40.126.137:8899)
[2025-03-19T16:04:03.995323889Z INFO agave_validator::bootstrap] Using RPC service from node dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS: Some(147.28.165.119:8899)
[2025-03-19T16:04:03.995260750Z INFO agave_validator::bootstrap] Using RPC service from node 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw: Some(212.232.28.212:8899)
[2025-03-19T16:04:04.021089880Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 6ms
[2025-03-19T16:04:04.021099840Z INFO agave_validator::bootstrap] Using RPC service from node AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC: Some(60.244.89.49:8899)
[2025-03-19T16:04:04.083554020Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 26ms
[2025-03-19T16:04:04.083946384Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 26ms
[2025-03-19T16:04:04.331473670Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://34.220.208.242:8899/): error trying to connect: tcp connect error: Connection refused (os error 111)
[2025-03-19T16:04:04.331485390Z INFO agave_validator::bootstrap] Excluding CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i as a future RPC candidate
[2025-03-19T16:04:04.334780098Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://52.13.43.37:8899/): error trying to connect: tcp connect error: Connection refused (os error 111)
[2025-03-19T16:04:04.334792627Z INFO agave_validator::bootstrap] Excluding cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh as a future RPC candidate
[2025-03-19T16:04:04.470921110Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 157ms
[2025-03-19T16:04:04.501614974Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 167ms
[2025-03-19T16:04:04.646041014Z INFO agave_validator::bootstrap] RPC node version: 2.1.13 Ping: 204ms
[2025-03-19T16:04:04.769911581Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 261ms
[2025-03-19T16:04:04.772194844Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 257ms
[2025-03-19T16:04:05.010592565Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=9183i purged_values_size=171i failed_inserts_size=6i num_nodes=177i num_nodes_staked=0i num_pubkeys=177i
[2025-03-19T16:04:05.010609505Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=134i new_push_requests2=0i purge=1494i purge_count=0i process_gossip_packets_time=15631i verify_gossip_packets_time=45931i handle_batch_ping_messages_time=952i handle_batch_pong_messages_time=0i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=0i handle_batch_pull_responses_time=8731i handle_batch_push_messages_time=0i process_pull_resp=7422i filter_pull_resp=1192i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=0i process_pull_resp_count=126i pull_response_fail_insert=1i pull_response_fail_timeout=0i pull_response_success=394i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:04:05.010628714Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=2842i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=0i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=0i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=3264i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=0i gossip_transmit_loop_time=3985i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2087523i gossip_listen_loop_iterations_since_last_report=159i
[2025-03-19T16:04:05.010631914Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=12i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=0i pull_from_entrypoint_count=0i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=0i
[2025-03-19T16:04:05.010634114Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=7i packets_received_count=2853i packets_received_ping_messages_count=34i packets_received_pong_messages_count=0i packets_received_prune_messages_count=0i packets_received_pull_requests_count=0i packets_received_pull_responses_count=2819i packets_received_push_messages_count=0i packets_received_unknown_count=0i packets_received_verified_count=2853i packets_sent_gossip_requests_count=114i packets_sent_ping_messages_count=0i packets_sent_pong_messages_count=34i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=80i packets_sent_pull_responses_count=0i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:04:05.010637344Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=0i LegacyContactInfo-pull=14i Vote-push=0i Vote-pull=141i LowestSlot-push=0i LowestSlot-pull=24i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=4i EpochSlots-push=0i EpochSlots-pull=2281i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=14i NodeInstance-push=0i NodeInstance-pull=15i DuplicateShred-push=0i DuplicateShred-pull=307i SnapshotHashes-push=0i SnapshotHashes-pull=17i ContactInfo-push=0i ContactInfo-pull=24i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=2841i
[2025-03-19T16:04:05.010640904Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=0i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=0i
[SENTINEL.SH] [2025-03-19T16:04:05Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:07.011174107Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=2i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=10492i purged_values_size=206i failed_inserts_size=6i num_nodes=177i num_nodes_staked=0i num_pubkeys=177i
[2025-03-19T16:04:07.011191166Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=90i new_push_requests2=0i purge=1613i purge_count=0i process_gossip_packets_time=7027i verify_gossip_packets_time=25000i handle_batch_ping_messages_time=598i handle_batch_pong_messages_time=0i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=0i handle_batch_pull_responses_time=3104i handle_batch_push_messages_time=0i process_pull_resp=2155i filter_pull_resp=887i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=0i process_pull_resp_count=63i pull_response_fail_insert=0i pull_response_fail_timeout=0i pull_response_success=200i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:04:07.011195086Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=1344i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=0i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=0i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=3527i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=0i gossip_transmit_loop_time=4239i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=1879597i gossip_listen_loop_iterations_since_last_report=80i
[2025-03-19T16:04:07.011213816Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=0i pull_from_entrypoint_count=1i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=0i
[2025-03-19T16:04:07.011216266Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=1i packets_received_count=1334i packets_received_ping_messages_count=19i packets_received_pong_messages_count=0i packets_received_prune_messages_count=0i packets_received_pull_requests_count=0i packets_received_pull_responses_count=1315i packets_received_push_messages_count=0i packets_received_unknown_count=0i packets_received_verified_count=1334i packets_sent_gossip_requests_count=107i packets_sent_ping_messages_count=0i packets_sent_pong_messages_count=19i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=88i packets_sent_pull_responses_count=0i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:04:07.011220936Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=0i LegacyContactInfo-pull=12i Vote-push=0i Vote-pull=72i LowestSlot-push=0i LowestSlot-pull=12i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=2i EpochSlots-push=0i EpochSlots-pull=1052i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=15i NodeInstance-push=0i NodeInstance-pull=12i DuplicateShred-push=0i DuplicateShred-pull=145i SnapshotHashes-push=0i SnapshotHashes-pull=5i ContactInfo-push=0i ContactInfo-pull=17i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=1344i
[2025-03-19T16:04:07.011224676Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=0i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=0i
[SENTINEL.SH] [2025-03-19T16:04:08Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:09.012516836Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=11142i purged_values_size=307i failed_inserts_size=6i num_nodes=177i num_nodes_staked=0i num_pubkeys=189i
[2025-03-19T16:04:09.012532685Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=315i new_push_requests2=40i purge=1764i purge_count=0i process_gossip_packets_time=11922i verify_gossip_packets_time=23321i handle_batch_ping_messages_time=1366i handle_batch_pong_messages_time=22i handle_batch_prune_messages_time=1i handle_batch_pull_requests_time=1860i handle_batch_pull_responses_time=2011i handle_batch_push_messages_time=2856i process_pull_resp=1204i filter_pull_resp=756i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=5i process_pull_resp_count=50i pull_response_fail_insert=0i pull_response_fail_timeout=0i pull_response_success=151i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:04:09.012536945Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=698i pull_request_ping_pong_check_failed_count=3i generate_pull_responses=1293i process_prune=1i prune_message_timeout=0i bad_prune_destination=0i process_push_message=303i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=3340i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=6330i gossip_transmit_loop_time=5208i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2125265i gossip_listen_loop_iterations_since_last_report=152i
[2025-03-19T16:04:09.012539945Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=102i push_fanout_num_nodes=918i push_message_value_count=52i pull_from_entrypoint_count=0i prune_message_len=4i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=496i
[2025-03-19T16:04:09.012542045Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=3i packets_received_count=806i packets_received_ping_messages_count=38i packets_received_pong_messages_count=3i packets_received_prune_messages_count=4i packets_received_pull_requests_count=25i packets_received_pull_responses_count=684i packets_received_push_messages_count=52i packets_received_unknown_count=0i packets_received_verified_count=806i packets_sent_gossip_requests_count=856i packets_sent_ping_messages_count=3i packets_sent_pong_messages_count=38i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=80i packets_sent_pull_responses_count=15i packets_sent_push_messages_count=720i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:04:09.012544795Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=0i LegacyContactInfo-pull=8i Vote-push=2i Vote-pull=49i LowestSlot-push=0i LowestSlot-pull=8i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=50i EpochSlots-pull=523i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=15i NodeInstance-push=0i NodeInstance-pull=23i DuplicateShred-push=0i DuplicateShred-pull=50i SnapshotHashes-push=0i SnapshotHashes-pull=5i ContactInfo-push=0i ContactInfo-pull=17i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=52i all-pull=698i
[2025-03-19T16:04:09.012547875Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=0i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=0i
[2025-03-19T16:04:11.000526974Z INFO solana_gossip::cluster_info]
IP Address |Age(ms)| Node identifier | Version |Gossip|TPUvote| TPU |TPUfwd| TVU |TVU Q |ServeR|ShredVer
------------------+-------+----------------------------------------------+---------+------+-------+------+------+------+------+------+--------
162.19.43.1 me| 2403 | BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar | 2.2.3 | 25000| none | none | none | none | none | none | 2405
147.28.198.57 | 283 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.172.191.11 | 513 | HMU77m6WSL9Xew9YvVCgz1hLuhzamz74eD9avi4XPdr | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
38.46.221.182 | 3102 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
147.28.165.119 | 79 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
38.46.221.182 | 715 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
5.199.164.171 | 79 | 4mdQGkbfrFvo5eG722xCXik4hn3xzu2EJD47g7Hd4HfH | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
35.194.23.154 | 79 | DGLzpbJ2ckLs4bsUT3Am5CkCCVgPwGumsKw3hEtKDxdz | 2.1.13 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
198.244.253.130 | 281 | 9AyV9DykRupCenmHzvFckXLa2Hg8kvG61c6yX215EZCm | 2.1.14 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
34.220.208.242 | 79 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.115.212.90 | 79 | Cw6X5R68muAyGRCb7W8ZSP2YbaRjwMs1t5sBEPkhdwbM | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
46.166.162.49 | 79 | HzWdnV141bP1PXXce4NJ6NCJgd6jr3kMaySuevbpgwaV | 2.1.13 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
147.28.165.81 | 79 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
67.209.54.163 | 93 | BrX9Z85BbmXYMjvvuAWU8imwsAqutVQiDg9uNfTGkzrJ | 2.1.15 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
188.214.130.100 | 79 | 9qyNwcztJXEbHWpq8YBHAc3gX1Cd9WdeSiaNEtrQMUp7 | 2.2.2 | 25000| 25005| 25003| 25004| 25001| 25002| 25012| 2405
67.209.52.66 | 1030 | FW33UVu8qPN7bvCYgmiFpXrXyDMN9PSUfVhkUAtyC6v4 | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
52.13.43.37 | 2179 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
64.130.34.4 | 678 | 5aNSc9MuJn5JX5jiSZv2YBMRxcdYSiKjv6uyDbRTPUPu | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
45.250.254.67 | 283 | 4wBKoRVAAvs3xBUm92PE3h5x7XQFsvDykSq69Gokh44n | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
64.130.52.151 | 5515 | 2kkTnE2JWek71pTusytU3GJtMbWCf8dnLwQBGxMS7ap5 | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
23.109.159.4 | 1986 | CTy6yhjhzEVZZBbQzCeuKz5XPPL83dPyav1ZKv7DHx6 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
141.98.219.9 | 282 | De2UiLDHRcfSxf2cnAwKu8yNxmq4gNEK4VcifyffiWsj | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
208.91.111.183 | 997 | CSase6DNS3RM1Xeqt3jmwA2cjZnApznZa4W4Dhi1PjRB | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
47.76.150.188 | 521 | 4v3dMU4Gp5EEb1MgyBctoQ6LQ9NZmUC1ZJH94JyaER8G | 2.1.11 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
40.160.1.59 | 423 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
185.26.10.121 | 1360 | FHdLcYmSu5vUQ5UG65teEC5rsNqMeyRtcM3imQAtargp | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
3.214.143.17 | 1692 | EjRGzg256yWQ5QffDc8htmWSabdFPMe2LAEGgtA6XkBF | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
52.19.134.152 | 7712 | 9CFn9Yw3R1HKDHAwpQWdcGaDxgdSPr6u5ebZSLZ9DsVm | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
13.217.130.6 | 1313 | FEceNs7TeZ6doKUC933bvUswmvdEPpbL1Utm8Wu7A4Ye | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
67.213.115.43 | 281 | 6Xd5mg1HbVUcChViLZp8CSfDkex6LTWcFLtVnAJnwf3y | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
208.91.111.147 | 1711 | DW4yMdNwpRm2WjwYHgwiwx2ALz6AooJLz3gCtfMYpdJs | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
141.94.154.228 | 283 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
103.50.32.200 | 796 | 9DAgzGqzG6vUUx9ZZ9yDSgqqusyezEMDqeeMaGAnb1E5 | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
79.137.101.152 | 89 | CL5v3g3Vm1ieqnxprp8342FcrDrhq3Kr21TRQ3qEBrbU | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
104.238.147.73 | 2114 | 2c5yEKTbq8LcWutihtqgZ3yFj7dHgKwLYEygYpyxFdDb | 2.1.11 | 10000| 10005| 10003| 10004| 10001| 10002| 10011| 2405
204.16.247.19 | 5696 | 5UHT2MnFvARGe5mGbKZth5Yh546UWmPtucQYBGTYsi4n | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
162.19.43.1 | 2761 | FDQHfbqgSUk94XKFKWu6E8qidL7bwGEXDPzAoTVTXEDm | 2.1.14 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
103.106.59.111 | 1042 | J7LhLJKkLLRCAzgBpTUK7Zp7duNbcAoUwxdCeLe4CmDG | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
141.98.217.117 | 5515 | EQXyPEDXptUEVzVz4XdJmhKjbUTZyNr7WmXRtz1AZepn | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
186.233.187.64 | 1988 | CyQF5447bJKZLe4dS5j7q8swsgz71sngo5Ay1hSWTb1z | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
60.244.89.49 | 912 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8400 | 8405 | 8403 | 8404 | 8401 | 8402 | 8411 | 2405
3.93.4.50 | 5515 | FrNPkvFUWEd3j2M1uz5u1ZJTKfaCgbHu7r59sD6zhecY | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.91.106.75 | 1125 | 3wyX7mT1AUK46aJrgzZ9cuS2UYbtSYeASx6mAz1hBzqW | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
74.63.235.2 | 281 | 36GpUgtojZTXV5NiNLG4GwfJBLd6s5Qtp5XbZE22S6o8 | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.34.6 | 2199 | Ddeqnfegkm2qA917L2o43sjUny2TSt9wK1BcbH4CQwY4 | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
103.106.59.105 | 230 | 3TCq6rV6TADTpdD19aYhKs2zq2j76mqftWQ3EdTosDqJ | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
141.98.216.89 | 5197 | 39cvwUEpgka9bU7Sn4my82VViMDWaCxi4YoPevfZxLf3 | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.46.114.58 | 79 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
40.160.1.61 | 1916 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
160.202.128.173 | 2188 | 97YUjL2EK42M6jG5VA4fKuVxGXDfxsC5Zawd9haLQJGk | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
79.137.99.55 | 1288 | Bt8yXrWAXH8X3kkZv8kzmaiyWtYwz2FFSZuuiXP6oqR2 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
15.204.47.154 | 111 | Bas9kLa5pr5Pi6T2Pug6PMfXrzT5QunHbrE2EYyAwyBo | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
5.199.164.180 | 230 | 2562dAN2v5tGy9QJxdSFnmKKFND1MducVZLdB8b7P2WK | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
64.130.34.5 | 1507 | 7aTwCz9PBC5nwj1m88NVx3tF8ytJhoAb8uj3FQfZ9TWS | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
15.204.241.62 | 299 | EaHyBMU3fvZMaTn2Jb7qM83gbLCweyJNiETbVB2hkyP9 | 2.1.14 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
67.209.52.78 | 231 | 69UoqUVy96b8zKBECrsNf2KgHkyWqJAmdwQLGWFNrQcH | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
45.250.253.233 | 1277 | 6CjNXemJGFYhJtn1zUqqaiyXMWo3XJQLfFRnvoDm2gY9 | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
204.16.244.26 | 1637 | 3AqQUgfbGoNpStyVCvkvz5MeSMw67HmMdz2KtCLdWK7J | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
63.251.232.229 | 1564 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
160.202.128.135 | 1910 | 6W3aGhPzeTZqmGKrS22SxVeab2kdF5QEBVJQ3k7vwj5Q | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.52.197 | 1323 | QTgZHGRk1aUXSLrgBzgAowbC95c7hES2PWWYyHuu3og | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
52.195.132.188 | 3189 | J3rSdVJimWDjjTS1eypBwgyBo36BYRfGcNQwCduCQD9Y | 2.1.15 | 8800 | 8805 | 8803 | 8804 | 8801 | 8802 | 8811 | 2405
23.109.172.204 | 3573 | 5ddQSb2cVyuGuncX3KyT97mVMbcfLF3vXbAYXBRE6ZJc | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
204.16.247.76 | 1300 | B95quggvWB2yUJM5jrDbSmZwFBL9HFNXqLMnewFoGG5M | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
74.118.143.126 | 42 | 86NTgDwxACHL4W4DbQmHhcmeSzzSyS6NBnmvxWJc1NXP | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.115.193.194 | 1598 | Fr8WCCCBxPtgPcy3R8Uh5sQfuP7af5KpgRT6K8thq5AZ | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
64.130.33.26 | 6787 | Bd9vEWKxyByeYiroT17XHRCkjnj5JwfeMca34CbDfkSD | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
208.115.226.166 | 2115 | CBxqCJrKUvoYjWswgiVVfhCWX32AKSiZABTdBrvfKnEz | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
135.148.171.49 | 2092 | 7D3q1VC1L8djZLjb78619sqLY3RnEU7LQeQRDEYRUhQp | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
194.67.206.120 | 1576 | 5qLppNuoh2HucuUpiYX7NBBdyDAQPVGK2N3Fed6S5c8M | 2.1.15 | 9000 | 9005 | 9003 | 9004 | 9001 | 9002 | 9011 | 2405
145.40.106.183 | 5563 | dw1xbcZQHmzA2zEPEqECp3SswUWdQ5sbdjBrKMEu7gK | 2.2.2 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.8.107.168 | 93 | 6tBs7Eq1Cx6YFtabvLRZeG5Jk7VKGQGNvrTZxsjLGjGG | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.91.110.67 | 93 | 6NNoD2pYEhvF3G4AaA1rhiQ2Ua8ojqGVixWUR5QWM5dP | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
163.114.159.219 | 818 | C8ZAr6Zz5hMF8wz38TMhg4gWfBrrkBA3kzQhp1sKz1JD | 2.2.0 | 11100| 11105| 11103| 11104| 11101| 11102| 11112| 2405
208.97.227.22 | 715 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 30000| 30005| 30003| 30004| 30001| 30002| 30011| 2405
103.50.32.201 | 5118 | 3uQRPJ9uf8dadzCo2GFgDg65Y3qJUTXCUzLfojH2qf9j | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
63.251.106.87 | 1637 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
38.97.60.6 | 63 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
3.16.49.73 | 1089 | 2tb5U6XQYYkZ5jWCRVqoHikQiKF6PmuDnQvNSEuLJXo5 | 2.1.16 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
217.28.63.147 | 1916 | 3THprEXdsRkJdsBvqnMf6RAVenyqW2ax9y3SRHfg3Jj1 | 2.0.15 | 8002 | 8007 | 8005 | 8006 | 8003 | 8004 | 8010 | 2405
20.253.158.75 | 1308 | CmrmEwq4LfuxsbRBpaQnZbo8ojFV5iKdWyGxcFaziCRs | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
148.113.206.128 | 2179 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
64.130.57.83 | 113 | DC7StQXH1n9hrJe9d5ZVXusBYZbW8G6zNKEBCZyam5jw | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
72.46.84.115 | 772 | DkawvKmKVBDcn9R8c76gZ3eqNvrySmVxArz15z6hGcpG | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
103.50.32.203 | 230 | 7Zk5LM4AFBh9S1pMAKXYcAMfoqLAGFZUPmHoCz7BiTYz | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
160.202.128.127 | 715 | HhhhXfqFJtJ835v2nPzTcovUktFAHi7LLnX2u93rSZtz | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.57.78 | 434 | 3wLP3s4GVeMsV3DYGUXBumPbR6njNCUBVPTJ9obQTscw | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
204.16.244.203 | 954 | 2v56rXf2WvNJBHQenasMGJVVTqXeHYqxuctfmBRXrNPQ | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
147.28.133.51 | 426 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6000 | 6005 | 6003 | 6004 | 6001 | 6002 | 6012 | 2405
208.115.212.70 | 133 | Fap57xttR6sj1UpS6XvbVf3n2tLXLzaDvZEFGdpR5Vt5 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
67.209.54.90 | 2188 | 7pbH563fFai2Gm8aXGi27Toj1i7x55rGp7QQ8ZQt6C7i | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
212.232.28.212 | 5330 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
208.91.107.3 | 283 | 8GFyFHMa8oobGzK6Bab1qdExMqjvQoFeHYTpkfJiR8NH | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
107.6.94.200 | 715 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
72.46.84.117 | 1391 | 9KyKWnUDd2DPoGys99HdyLJ553ruUGDov45jUv9pLTsh | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
91.134.8.113 | 79 | 6dYSW74ej3mBhXFTASy8NGLEhYZU4sKqoFwXemxVyJDp | 2.0.19 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8009 | 2405
51.20.239.237 | 1797 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
47.130.39.171 | 5197 | 8UAhoTURfDU5UaL3KK5Dy4W62R3rQNceHs9aUkxKqi8X | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
57.128.141.123 | 79 | sSRDQFY8Gri48fzXCENWgsSRcp5pheZR4TeB1pSoWJs | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
147.28.198.107 | 1507 | dw2gYZBQNisftpoCZ3XrNe9BiCyCG4reMKSejmztVZ8 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.189.44.238 | 111 | 3ne7n82Kqf1zPo4obYTnQA8tJBDuSEYyvKYS89Mbky4q | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
74.118.143.62 | 705 | 8PikNto2ADRtVbduxyEVd7DRUf5kVuRK3oFQ1csDeZF3 | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
141.98.217.57 | 2031 | 2CbEVhecSXz17JtzWtCsLSrnH8YHYtRCYi7co7wNPLXj | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
194.126.172.170 | 1576 | 2ZZkgKcBfp4tW8qCLj2yjxRYh9CuvEVJWb6e2KKS91Mj | 2.1.15 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
103.106.59.87 | 1012 | G29JbGLwxgmtYqkqZntxNgBxyg8NkKSHQrZbJvYxaekf | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
132.145.143.18 | 744 | FsdCY72YcXifByx61gcsvmZ5NVafZkquEqCj94MPx59Z | 2.1.11 | 8001 | none | none | none | none | none | none | 2405
18.117.197.60 | 10008 | APnGAfs4Zh67o1y77LocktHmgAj968BDPAKqQx2oFV8c | 2.1.11 | 8000 | none | none | none | none | none | none | 2405
45.77.252.43 | 10008 | zXNXTerWhz7Mc5JQQgsteJ1E3fz16S9kdPNJue796iH | 2.1.15 | 8000 | 8005 | none | none | 8001 | 8002 | 8011 | 2405
65.109.76.130 | 10008 | FVvc9sktQtTe3h3wHK9Pijkqt28i7EV9WMznV17nqcrQ | 2.2.0 | 14507| none | none | none | none | none | none | 2405
208.91.111.182 | 10008 | TUJAuFA2pVoVwxkz634nd5jc7FoaLpyyaPCdXvWFYwk | 2.2.0 | 8001 | none | none | none | none | none | none | 2405
186.233.187.187 | 10008 | 3d3nZzZ9g2TC7PQfRuv1Y6eTgBxjKeQvua8JRLgGFUMx | 2.0.20 | 8000 | none | none | none | none | none | none | 2405
88.216.198.139 | 938 | A7dsWYJghLoALaHoViXz8bi1MhUBPCpsraCYwbs1puei | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
84.32.32.14 | 1300 | 9dfk2wFcoREXo4pscPPA3zPsaVMuxLaQ1TtEK1s5uyoC | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
93.187.218.219 | 230 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8020 | 8025 | 8023 | 8024 | 8021 | 8022 | 8032 | 2405
88.97.255.77 | 10008 | 9pSiRik1sC7Yex71jVTUnSUeeQpEGoJ38Ji5GKhf55Hz | 2.2.0 | 8000 | none | none | none | none | none | none | 2405
92.38.186.52 | 10008 | 27adQ5Rih7iVMwZHrjo3Hfkga6ZsKk8gRiFSgXmmNFv2 | 2.2.0 | 8000 | none | none | none | none | none | none | 2405
147.28.179.215 | 796 | DbSa4G3AJzPFVftjQowiaKqUsTD1rEDjQp1nUT5E3SyQ | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
147.28.165.21 | 10008 | 5kEbSzXhNM9j73HMhfq3tEi4rHyR1YdYRpUiLxkgV7BV | 2.2.0 | 14521| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | ASptD91PuiYsc1Www492kyABnYN29rdZJBMrv1JbXgrS | 2.2.0 | 14501| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | JDHrDMEsZgn6WmtRrfGtYdfz5VPyX8FgnDmnTwfKXtM3 | 2.2.0 | 14511| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 3PRwGdfiqn5aVMTE4xPZLPLQC7bAdseJTBPY8M8seB4L | 2.2.0 | 14527| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 95efi8SDAWz5eZpsUMu6fkGayfEyTP6HGp3VC22JhxQ2 | 2.2.0 | 14533| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | FQXgQpT1rY9zGMf5LtXRhiPcRSWQpBLzqXQSALhkVxgv | 2.2.0 | 14522| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 2MP6fXk1tcgcqeh4mSDnJLjWpa1kAUnxn6FoS8Zv9rys | 2.2.0 | 14512| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BzxpeHHQfnrr2db6wqsyd9GNtKqVG4KSAo2CWEeekQa1 | 2.2.0 | 14502| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | FRpMYcFKQ8u2Z16ASBrEGixBNsWqyVWbZQqnG65CoQ8S | 2.2.0 | 14532| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 2Y9hELJXQjGEHKFfUJGbcwVyMqqWnZdKPfxPBk1bcVjS | 2.2.0 | 14537| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | GzJqmHczxGyMM64XniBzFhrX7uRuEQ2QrQgSWD55YUUP | 2.2.0 | 14523| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | GPgZC9oBwvvXAfXzj92Tpq6K13CHmimzwbqxcNXG8eQC | 2.2.0 | 14503| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 9KBugAvyB7xVdRF43n6LJXCn15sJvvEjqLCfARk9Ghx8 | 2.2.0 | 14513| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 9MzZnJfqYAbc3JhLdweUEgd5ouV45eJpy2RuUc5rgipS | 2.2.0 | 14529| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BLp7uGyHT3TNrJbi5b5Jubxz71usyBUu8RFqLvvUQkLz | 2.2.0 | 14514| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | D18sA29pNUtgs4cJiQGXUun1fCcUHfHwg9ck3YDjXtGU | 2.2.0 | 14504| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 9Ae9eFw7Lx1ovnLTmFxAvREo8Jzd12jvdjmFUj3iAPju | 2.2.0 | 14530| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | p4oCRoSAoVDMS2JLpvdnrJTYV6QuPZkXEJfUSNg766R | 2.2.0 | 14524| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | F6uZ5Snp1ixFfp66Rs1VkCvk3s4Q76fxTw913Xg6sL2T | 2.2.0 | 14518| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | RHS7qE6JjWbk9W8hsbeQvEW5cyKFvx8uH6MTM7hhJBF | 2.2.0 | 14520| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | EEPM5FzRqRaXWi9LEoUHQp9NzQR7ghiMU1GjhvRuQN26 | 2.2.0 | 14505| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BeT1whVNgaBgawqhLLVyFSAevXP4qZEz6K4844m2NBpY | 2.2.0 | 14515| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 9djoBJW2fV4mJ39ksNVZ5YwEM2gBwJHSp39oheRnTTR6 | 2.2.0 | 14525| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | AHWqqZ6LZVQFTFBVqsrfmavwPK2XHdPkymHaUsm9LdSW | 2.2.0 | 14508| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | H1rVCxKS5nGfxSoancENweaRc6iZDGfr4ip25KEspfgP | 2.2.0 | 14531| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 3aM1RiqXbyohQ3AtgNz66FyQogjCWErYhxgHUWBT9xU5 | 2.2.0 | 14526| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 2rMtb4KRjTCtb6NAsYCEuxLgiwyY35WCiPV1i1JkqcUk | 2.2.0 | 14516| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | Hv4V2ELDNB68KCnj67CR8r3MKxjQcXop8P3sV7JMfbj1 | 2.2.0 | 14506| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 4TTWkhgWGjTzKs4WQThLiPC7UhFXo9F8hqmkxJKaUMqg | 2.2.0 | 14540| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | CjMVxKKdbxpJCtBz6nGzD3hEcU6nhCL4LeBnPN5fHfGQ | 2.2.0 | 14507| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 89qmzF7z67yY7ddSuGNfETtnHBcHwcicVkcJBnBvwaFX | 2.2.0 | 14517| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 5eF3a5Ryyhc6Qiz6B21xKcaToARVm4UQDzYJNhCXRnjD | 2.2.0 | 14509| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | HsDD7sQQidYpf3PWyKbdWzrFafQzFg6g6LaAPt4FZFks | 2.2.0 | 14542| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 6yK64d6jtLvuuZsU2j7b9ANfSRdD4gQQc7APvLcQRawa | 2.2.0 | 14534| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 4FqrZ7MK3DAVdcR4NWPsXBtp23xQmaVMLCuWuN6XWNQJ | 2.2.0 | 14536| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | AWbhMuRKCzkxdnTBvWXqCBj9un6QFZ5VAEYgATLbzURp | 2.2.0 | 14528| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BUYEAcwyNA7bSNeXmndVS6orp7TxFc9JtdGyCLZH5AuP | 2.2.0 | 14500| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | D1DGY57hMNGopiTxTP65cQVYYJJBGw2SyWZis3LzKNVu | 2.2.0 | 14519| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | HULnKLeejyjPQ53UhrwG7JZqFfyKcy8Da8152FYsVj6h | 2.2.0 | 14545| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 2Q2cSDNy6tAPQyx5X3VJESqDXJTFasJ3CrYjbjd9Z8w3 | 2.2.0 | 14535| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 8Kgnpgu4nMhLhCfco7SPXXcY4KwLGrTd8iP6DC2ecGY2 | 2.2.0 | 14510| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BiBTs3eiykKMYceUsPG2vZq1BgWQ5AN6A1pt834kWZ4G | 2.2.0 | 14546| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | Gs95Syj8n8Pa48Q1waAdGW3rdyaND8HuawpafHsHLpC8 | 2.2.0 | 14538| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 2rVdL8VjpCXWaVvpmHXTxSLqJqtD4Tfy2tNy7n2ps4Ys | 2.2.0 | 14539| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | HqbfrXbyGSxFyeon1UWZ8pXp44jkAcdLMsKQZVuM1Q9A | 2.2.0 | 14543| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | J4sc9JoKDMAutT3jypew7593JPvCnsjRSmW6RRtcZ72d | 2.2.0 | 14544| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 3Gz2Zqski8UXoJ4XNSy43t9BiY5PW3JZtNnJp1pBkAVL | 2.2.0 | 14541| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 5Qbn7Y4S8hmV9qxZP5odW6Nfd3FNvgRdJepp8QrRqX75 | 2.2.0 | 14547| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 3GuqvF3MK5hkguVxnaKnwJmVLhPXkVecoFK4brDbuvW2 | 2.2.0 | 14558| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 8VvmLywwYvkrXCUGPJW61ryEWQg5ymGTkWv7hY5CGwmn | 2.2.0 | 14559| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 7e2aqTRJgx7xgV4dUgt7DtQRFtKkpn9SEDWC3DfjMDsD | 2.2.0 | 14625| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 4VzqgFMtVG41NKNniibPqfhf9GvvM6LuncWdSZp2Qu5y | 2.2.0 | 14505| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | 8eMpRcmaVUyvU9ws3YnPxnjzDUn6x9HLytmGswTzK3BC | 2.2.0 | 14525| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | BDgqfmLxprbKurSdirU9ZNNn8idP4cRoTP1yCS29h1CP | 2.2.0 | 14594| none | none | none | none | none | none | 2405
147.28.165.21 | 10008 | D2P2Q6Cnary5NdmBqFEASuhUMsTZkHAubcUApTrrP4N6 | 2.2.0 | 14521| none | none | none | none | none | none | 2405
145.40.117.251 | 2761 | DmCowGH9DUHYCetfGaWzPzYCi455yDxewcycdyWuPLjx | 2.3.0 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
35.197.53.105 | 79 | EBTPS7efmbiRLS4spFdB63q69Vzg1KuZUYwddk4Ec4aV | 1.18.21 | 8001 | none | none | none | none | none | none | 2405
145.40.126.137 | 281 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
Nodes: 110
Spies: 67
Nodes with different shred version: 2
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 283 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 3102 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 79 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 715 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 79 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 79 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 2179 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 423 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 283 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 912 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 79 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 1916 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 1564 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 715 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 1637 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 63 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 2179 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 426 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 5330 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 715 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 1797 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 230 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 281 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:04:11.002934636Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://212.232.28.212:8899/): operation timed out
[2025-03-19T16:04:11.002940246Z INFO agave_validator::bootstrap] Excluding 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw as a future RPC candidate
[2025-03-19T16:04:11.003382919Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://63.251.232.229:21811/): operation timed out
[2025-03-19T16:04:11.003392849Z INFO agave_validator::bootstrap] Excluding DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N as a future RPC candidate
[2025-03-19T16:04:11.003402629Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://141.94.154.228:8899/): operation timed out
[2025-03-19T16:04:11.003405829Z INFO agave_validator::bootstrap] Excluding FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho as a future RPC candidate
[2025-03-19T16:04:11.003853912Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://148.113.206.128:8899/): operation timed out
[2025-03-19T16:04:11.003858912Z INFO agave_validator::bootstrap] Excluding G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX as a future RPC candidate
[2025-03-19T16:04:11.003866842Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://63.251.106.87:21811/): operation timed out
[2025-03-19T16:04:11.003871002Z INFO agave_validator::bootstrap] Excluding DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu as a future RPC candidate
[2025-03-19T16:04:11.005865200Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://38.97.60.6:8899/): operation timed out
[2025-03-19T16:04:11.005871650Z INFO agave_validator::bootstrap] Excluding Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA as a future RPC candidate
[2025-03-19T16:04:11.005881710Z WARN agave_validator::bootstrap] Failed to get RPC node version: error sending request for url (http://64.46.114.58:21811/): operation timed out
[2025-03-19T16:04:11.005884900Z INFO agave_validator::bootstrap] Excluding 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z as a future RPC candidate
[2025-03-19T16:04:11.016035449Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=11201i purged_values_size=1091i failed_inserts_size=20i num_nodes=177i num_nodes_staked=0i num_pubkeys=190i
[2025-03-19T16:04:11.016043639Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=1660i new_push_requests2=330i purge=1331i purge_count=0i process_gossip_packets_time=46416i verify_gossip_packets_time=60427i handle_batch_ping_messages_time=283i handle_batch_pong_messages_time=7i handle_batch_prune_messages_time=101i handle_batch_pull_requests_time=3629i handle_batch_pull_responses_time=1179i handle_batch_push_messages_time=29679i process_pull_resp=906i filter_pull_resp=220i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=103i process_pull_resp_count=72i pull_response_fail_insert=14i pull_response_fail_timeout=0i pull_response_success=286i num_redundant_pull_responses=1i save_contact_info_time=0i
[2025-03-19T16:04:11.016048229Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=316i pull_request_ping_pong_check_failed_count=1i generate_pull_responses=2651i process_prune=66i prune_message_timeout=0i bad_prune_destination=0i process_push_message=2033i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=3257i gossip_pull_request_no_budget=1i gossip_pull_request_dropped_requests=13i gossip_pull_request_sent_bytes=22337i gossip_transmit_loop_time=4786i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2002547i gossip_listen_loop_iterations_since_last_report=633i
[2025-03-19T16:04:11.016051169Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=819i push_fanout_num_nodes=7102i push_message_value_count=542i pull_from_entrypoint_count=0i prune_message_len=255i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=5380i
[2025-03-19T16:04:11.016054379Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=8i packets_received_count=1096i packets_received_ping_messages_count=10i packets_received_pong_messages_count=1i packets_received_prune_messages_count=247i packets_received_pull_requests_count=51i packets_received_pull_responses_count=278i packets_received_push_messages_count=509i packets_received_unknown_count=0i packets_received_verified_count=1096i packets_sent_gossip_requests_count=5653i packets_sent_ping_messages_count=1i packets_sent_pong_messages_count=10i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=80i packets_sent_pull_responses_count=43i packets_sent_push_messages_count=5519i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:04:11.016067689Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=14i LegacyContactInfo-pull=17i Vote-push=21i Vote-pull=180i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=1i EpochSlots-push=468i EpochSlots-pull=24i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=5i NodeInstance-push=14i NodeInstance-pull=25i DuplicateShred-push=0i DuplicateShred-pull=3i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=24i ContactInfo-pull=47i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=541i all-pull=302i
[2025-03-19T16:04:11.016072029Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=1i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=1i all-pull=0i
[2025-03-19T16:04:11.516959445Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[SENTINEL.SH] [2025-03-19T16:04:11Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:11.861891539Z INFO agave_validator::bootstrap] RPC node root slot: 368428424
[2025-03-19T16:04:11.862614158Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:04:11.862658507Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:04:11.862696626Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:04:14Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:17Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:18.391994906Z WARN agave_validator::bootstrap] The snapshot download is too slow, throughput: 4839105 < min speed 50000000 bytes/sec, will abort and try a different node. Abort count: 0, Progress detail: DownloadProgressRecord { elapsed_time: 6.006052825s, last_elapsed_time: 6.006052865s, last_throughput: 4839105.0, total_throughput: 4839105.0, total_bytes: 42982658048, current_bytes: 29063920, percentage_done: 0.06761778, estimated_remaining_time: 8876.351, notification_count: 1 }
[2025-03-19T16:04:18.392012575Z INFO solana_file_download] Download is aborted by the caller
[2025-03-19T16:04:18.393616930Z INFO solana_download_utils] Unable to write "/snapshots/remote/tmp-snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst": Custom { kind: Other, error: "Download is aborted by the caller" }
[2025-03-19T16:04:18.902465217Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://145.40.126.137:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz)
[2025-03-19T16:04:19.412672226Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://145.40.126.137:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2)
[2025-03-19T16:04:19.922589089Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://145.40.126.137:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4)
[2025-03-19T16:04:20.455393483Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://145.40.126.137:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar)
[2025-03-19T16:04:20.455409953Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 145.40.126.137:8899
[2025-03-19T16:04:20.456559625Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[SENTINEL.SH] [2025-03-19T16:04:20Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:20.980195533Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:04:21.320425707Z INFO agave_validator::bootstrap] RPC node root slot: 368428449
[2025-03-19T16:04:21.320578425Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:04:21.320620324Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:04:21.320655654Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:04:23Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:26Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:28.049700668Z WARN agave_validator::bootstrap] The snapshot download is too slow, throughput: 26997450 < min speed 50000000 bytes/sec, will abort and try a different node. Abort count: 1, Progress detail: DownloadProgressRecord { elapsed_time: 6.219883714s, last_elapsed_time: 6.219883774s, last_throughput: 26997450.0, total_throughput: 26997448.0, total_bytes: 42983907328, current_bytes: 167920984, percentage_done: 0.39066017, estimated_remaining_time: 1585.9271, notification_count: 1 }
[2025-03-19T16:04:28.049719657Z INFO solana_file_download] Download is aborted by the caller
[2025-03-19T16:04:28.124168915Z INFO solana_download_utils] Unable to write "/snapshots/remote/tmp-snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst": Custom { kind: Other, error: "Download is aborted by the caller" }
[2025-03-19T16:04:28.632478155Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.198.57:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz)
[2025-03-19T16:04:29.158749663Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.198.57:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2)
[2025-03-19T16:04:29.682711479Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.198.57:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4)
[SENTINEL.SH] [2025-03-19T16:04:29Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:30.202805186Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.198.57:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar)
[2025-03-19T16:04:30.202822026Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 147.28.198.57:8899
[2025-03-19T16:04:30.203805940Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:04:30.625780102Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:04:30.915441476Z INFO agave_validator::bootstrap] RPC node root slot: 368428472
[2025-03-19T16:04:30.915597334Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:04:30.915630753Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:04:30.915662763Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:04:32Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:35Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:37.425126149Z WARN agave_validator::bootstrap] The snapshot download is too slow, throughput: 13513991 < min speed 50000000 bytes/sec, will abort and try a different node. Abort count: 2, Progress detail: DownloadProgressRecord { elapsed_time: 6.08955971s, last_elapsed_time: 6.08955975s, last_throughput: 13513991.0, total_throughput: 13513990.0, total_bytes: 43006246912, current_bytes: 82294256, percentage_done: 0.19135419, estimated_remaining_time: 3176.2605, notification_count: 1 }
[2025-03-19T16:04:37.425143828Z INFO solana_file_download] Download is aborted by the caller
[2025-03-19T16:04:37.465667412Z INFO solana_download_utils] Unable to write "/snapshots/remote/tmp-snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst": Custom { kind: Other, error: "Download is aborted by the caller" }
[2025-03-19T16:04:37.876573642Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://60.244.89.49:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz)
[2025-03-19T16:04:38.286485318Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://60.244.89.49:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2)
[2025-03-19T16:04:38.695028547Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://60.244.89.49:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4)
[SENTINEL.SH] [2025-03-19T16:04:38Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:39.112324798Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://60.244.89.49:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar)
[2025-03-19T16:04:39.112342068Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 60.244.89.49:8899
[2025-03-19T16:04:39.112345058Z INFO agave_validator::bootstrap] Excluding AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC as a future RPC candidate
[2025-03-19T16:04:39.113477570Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:04:39.454803145Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:04:39.687106590Z INFO agave_validator::bootstrap] RPC node root slot: 368428495
[2025-03-19T16:04:39.687262478Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:04:39.687300927Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:04:39.687334527Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:04:41Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:44Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:46.119900263Z WARN agave_validator::bootstrap] The snapshot download is too slow, throughput: 17616888 < min speed 50000000 bytes/sec, will abort and try a different node. Abort count: 3, Progress detail: DownloadProgressRecord { elapsed_time: 6.105018345s, last_elapsed_time: 6.105018396s, last_throughput: 17616888.0, total_throughput: 17616886.0, total_bytes: 43828248576, current_bytes: 107551416, percentage_done: 0.2453929, estimated_remaining_time: 2481.749, notification_count: 1 }
[2025-03-19T16:04:46.119918043Z INFO solana_file_download] Download is aborted by the caller
[2025-03-19T16:04:46.191284746Z INFO solana_download_utils] Unable to write "/snapshots/remote/tmp-snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst": Custom { kind: Other, error: "Download is aborted by the caller" }
[2025-03-19T16:04:46.520402102Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://107.6.94.200:21811/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz)
[2025-03-19T16:04:46.854762287Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://107.6.94.200:21811/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2)
[2025-03-19T16:04:47.178044716Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://107.6.94.200:21811/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4)
[2025-03-19T16:04:47.516098584Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://107.6.94.200:21811/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar)
[2025-03-19T16:04:47.516115384Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 107.6.94.200:21811
[2025-03-19T16:04:47.516118084Z INFO agave_validator::bootstrap] Excluding 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 as a future RPC candidate
[2025-03-19T16:04:47.517319935Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[SENTINEL.SH] [2025-03-19T16:04:47Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:04:47.836950992Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:04:48.080375001Z INFO agave_validator::bootstrap] RPC node root slot: 368428518
[2025-03-19T16:04:48.080526839Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:04:48.080560789Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:04:48.080594818Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:04:50Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:53Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:56Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:04:59Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:02Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:05Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:08Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:11Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:14Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:17Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:05:18.085990137Z INFO solana_download_utils] error sending request for url (http://38.46.221.182:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst): operation timed out
[SENTINEL.SH] [2025-03-19T16:05:20Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:23Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:26Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:29Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:32Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:35Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:38Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:41Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:44Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:47Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:05:48.091462323Z INFO solana_download_utils] error sending request for url (http://38.46.221.182:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz): operation timed out
[SENTINEL.SH] [2025-03-19T16:05:50Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:53Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:56Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:05:59Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:02Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:05Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:08Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:11Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:14Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:17Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:06:18.097056577Z INFO solana_download_utils] error sending request for url (http://38.46.221.182:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2): operation timed out
[SENTINEL.SH] [2025-03-19T16:06:20Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:23Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:30Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:33Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:36Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:39Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:42Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:45Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:48Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:06:48.102536776Z INFO solana_download_utils] error sending request for url (http://38.46.221.182:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4): operation timed out
[SENTINEL.SH] [2025-03-19T16:06:51Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:54Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:06:57Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:00Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:03Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:06Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:09Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:12Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:15Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:07:18.108173380Z INFO solana_download_utils] error sending request for url (http://38.46.221.182:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar): operation timed out
[2025-03-19T16:07:18.108188910Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 38.46.221.182:8899
[2025-03-19T16:07:18.108191650Z INFO agave_validator::bootstrap] Excluding 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs as a future RPC candidate
[2025-03-19T16:07:18.109454070Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[SENTINEL.SH] [2025-03-19T16:07:18Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:07:18.165451177Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:07:18.239260354Z INFO agave_validator::bootstrap] RPC node root slot: 368428905
[2025-03-19T16:07:18.239396642Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:07:18.239432141Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:07:18.239464641Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:07:21Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:24Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:07:24.317633576Z WARN agave_validator::bootstrap] The snapshot download is too slow, throughput: 14760876 < min speed 50000000 bytes/sec, will abort and try a different node. Abort count: 4, Progress detail: DownloadProgressRecord { elapsed_time: 6.019961843s, last_elapsed_time: 6.019961903s, last_throughput: 14760876.0, total_throughput: 14760876.0, total_bytes: 42981359616, current_bytes: 88859912, percentage_done: 0.20674057, estimated_remaining_time: 2905.8235, notification_count: 1 }
[2025-03-19T16:07:24.317651425Z INFO solana_file_download] Download is aborted by the caller
[2025-03-19T16:07:24.327503757Z INFO solana_download_utils] Unable to write "/snapshots/remote/tmp-snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst": Custom { kind: Other, error: "Download is aborted by the caller" }
[2025-03-19T16:07:24.390231882Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.81:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.gz)
[2025-03-19T16:07:24.449164825Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.81:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.bz2)
[2025-03-19T16:07:24.511231771Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.81:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.lz4)
[2025-03-19T16:07:24.573673651Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.81:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar)
[2025-03-19T16:07:24.573687531Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368420648 from 147.28.165.81:8899
[2025-03-19T16:07:24.574865343Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:07:24.634323837Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:07:24.705481686Z INFO agave_validator::bootstrap] RPC node root slot: 368428921
[2025-03-19T16:07:24.705622954Z INFO agave_validator::bootstrap] Local snapshot from slot 368418744 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:07:24.705654104Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:07:24.705686073Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:07:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:30Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:33Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:36Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:39Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:42Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:45Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:48Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:51Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:54Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:07:57Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:00Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:03Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:06Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:09Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:12Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:15Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:18Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:21Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:24Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:30Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:33Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:36Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:39Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:42Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:45Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:48Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:51Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:54Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:08:57Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:00Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:03Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:06Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:09Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:12Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:15Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:18Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:21Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:24Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:30Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:33Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:36Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:39Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:42Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:45Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:48Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:51Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:54Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:09:57Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:00Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:03Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:06Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:09Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:12Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:15Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:18Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:21Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:24Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:30Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:33Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:36Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:39Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:42Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:45Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:48Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:51Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:54Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:10:57Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:00Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:03Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:06Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:09Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:12Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:15Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:18Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:21Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:24Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:27Z] [WAIT] Health check failed. Retrying in 3s.
[SENTINEL.SH] [2025-03-19T16:11:30Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:30.917420488Z INFO solana_file_download] Downloaded http://147.28.165.119:8899/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst (42983690431 bytes) in 246.211666247s
[2025-03-19T16:11:30.918642691Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:11:30.944607037Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[2025-03-19T16:11:31.006989707Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.119:8899/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.zst)
[2025-03-19T16:11:31.065570493Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.119:8899/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.gz)
[2025-03-19T16:11:31.124001630Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.119:8899/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.bz2)
[2025-03-19T16:11:31.182848801Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.119:8899/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.lz4)
[2025-03-19T16:11:31.244919586Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.165.119:8899/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar)
[2025-03-19T16:11:31.244932796Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368428352 from 147.28.165.119:8899
[2025-03-19T16:11:31.246411745Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:11:31.262382915Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:11:31.355204016Z INFO agave_validator::bootstrap] RPC node root slot: 368429560
[2025-03-19T16:11:31.355332635Z INFO agave_validator::bootstrap] Local snapshot from slot 368420648 is too old. Downloading a newer snapshot for slot 368428352.
[2025-03-19T16:11:31.355363174Z INFO agave_validator::bootstrap] Full snapshot archive already exists locally. Skipping download. slot: 368420648, hash: 56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4
[2025-03-19T16:11:31.355396774Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:11:31.355431333Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[2025-03-19T16:11:31.377690982Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.133.51:6699/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.zst)
[2025-03-19T16:11:31.395644813Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.133.51:6699/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.gz)
[2025-03-19T16:11:31.413900320Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.133.51:6699/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.bz2)
[2025-03-19T16:11:31.432069118Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.133.51:6699/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar.lz4)
[2025-03-19T16:11:31.449945070Z INFO solana_download_utils] HTTP status client error (404 Not Found) for url (http://147.28.133.51:6699/incremental-snapshot-368420648-368428352-8eMjiCUG6XhVVDPcqKgj3PkW1qU3jtUUDYwraLU27vz7.tar)
[2025-03-19T16:11:31.449955460Z WARN agave_validator::bootstrap] Failed to download a snapshot archive for slot 368428352 from 147.28.133.51:6699
[2025-03-19T16:11:31.449957870Z INFO agave_validator::bootstrap] Excluding 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 as a future RPC candidate
[2025-03-19T16:11:31.451112464Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:11:32.452961999Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 101 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 101 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 103 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 101 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 25 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 103 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 160 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 101 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 25 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 160 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 25 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 25 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 511 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 103 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 226 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 160 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 101 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 289 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 101 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 314 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 288 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 25 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:32.452979649Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405...
[2025-03-19T16:11:32.453002478Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:33.453196960Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 113 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 108 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 108 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 108 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 108 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 111 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 111 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 488 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 390 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 489 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 1026 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 108 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 310 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 488 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 33 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 569 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 678 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 307 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 310 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 108 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 537 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 32 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:33.453215940Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:33.453228860Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:33.454095017Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=2i entrypoint2=54i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=6084i purged_values_size=116i failed_inserts_size=2i num_nodes=175i num_nodes_staked=0i num_pubkeys=176i
[2025-03-19T16:11:33.454109547Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=197i new_push_requests2=0i purge=1281i purge_count=0i process_gossip_packets_time=31521i verify_gossip_packets_time=106917i handle_batch_ping_messages_time=2663i handle_batch_pong_messages_time=3139i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=3211i handle_batch_pull_responses_time=10725i handle_batch_push_messages_time=0i process_pull_resp=8544i filter_pull_resp=2048i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=90i process_pull_resp_count=157i pull_response_fail_insert=2i pull_response_fail_timeout=0i pull_response_success=1135i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:11:33.454113337Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=6026i pull_request_ping_pong_check_failed_count=1i generate_pull_responses=1848i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=0i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=5964i gossip_pull_request_no_budget=1i gossip_pull_request_dropped_requests=22i gossip_pull_request_sent_bytes=22992i gossip_transmit_loop_time=7047i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=1980767i gossip_listen_loop_iterations_since_last_report=374i
[2025-03-19T16:11:33.454116157Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=430i skip_pull_shred_version=0i num_duplicate_push_messages=0i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=0i pull_from_entrypoint_count=4i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=0i
[2025-03-19T16:11:33.454118927Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=6i packets_received_count=6745i packets_received_ping_messages_count=126i packets_received_pong_messages_count=222i packets_received_prune_messages_count=0i packets_received_pull_requests_count=62i packets_received_pull_responses_count=6335i packets_received_push_messages_count=0i packets_received_unknown_count=0i packets_received_verified_count=6745i packets_sent_gossip_requests_count=461i packets_sent_ping_messages_count=174i packets_sent_pong_messages_count=126i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=104i packets_sent_pull_responses_count=57i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:33.454125067Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=0i LegacyContactInfo-pull=31i Vote-push=0i Vote-pull=352i LowestSlot-push=0i LowestSlot-pull=72i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=3i EpochSlots-push=0i EpochSlots-pull=4717i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=28i NodeInstance-push=0i NodeInstance-pull=35i DuplicateShred-push=0i DuplicateShred-pull=641i SnapshotHashes-push=0i SnapshotHashes-pull=60i ContactInfo-push=0i ContactInfo-pull=85i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=6024i
[2025-03-19T16:11:33.454128667Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=0i LegacyContactInfo-pull=0i Vote-push=0i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=0i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=0i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=0i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=0i all-pull=0i
[SENTINEL.SH] [2025-03-19T16:11:33Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:34.453425823Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 165 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 111 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 40 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 107 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 165 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 111 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 1111 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 165 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 349 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 1489 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 2026 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 107 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 1310 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 159 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 377 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 555 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 111 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 1307 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 555 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 377 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 377 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 349 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:34.453443563Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:34.453456063Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:35.453633958Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 383 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 389 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 389 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 557 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 1165 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 379 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 2111 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 560 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 1349 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 2489 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 3026 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 560 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 2310 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 1159 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 1377 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 379 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 772 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 2307 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 381 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 707 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 99 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 1349 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:35.453651418Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:35.453662508Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:35.458220272Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=8711i purged_values_size=355i failed_inserts_size=76i num_nodes=175i num_nodes_staked=0i num_pubkeys=176i
[2025-03-19T16:11:35.458227312Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=89i new_push_requests2=0i purge=1366i purge_count=0i process_gossip_packets_time=26477i verify_gossip_packets_time=70931i handle_batch_ping_messages_time=434i handle_batch_pong_messages_time=15i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=3895i handle_batch_pull_responses_time=7894i handle_batch_push_messages_time=3685i process_pull_resp=6634i filter_pull_resp=1178i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=129i process_pull_resp_count=100i pull_response_fail_insert=74i pull_response_fail_timeout=1i pull_response_success=351i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:11:35.458231782Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=2734i pull_request_ping_pong_check_failed_count=1i generate_pull_responses=2581i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=863i prune_received_cache=0i epoch_slots_lookup=0i new_pull_requests=3261i gossip_pull_request_no_budget=3i gossip_pull_request_dropped_requests=55i gossip_pull_request_sent_bytes=24568i gossip_transmit_loop_time=3878i gossip_transmit_loop_iterations_since_last_report=21i gossip_listen_loop_time=2020344i gossip_listen_loop_iterations_since_last_report=369i
[2025-03-19T16:11:35.458234612Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=318i skip_pull_shred_version=0i num_duplicate_push_messages=92i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=298i pull_from_entrypoint_count=0i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=2565i
[2025-03-19T16:11:35.458237012Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=2i packets_received_count=3375i packets_received_ping_messages_count=17i packets_received_pong_messages_count=2i packets_received_prune_messages_count=0i packets_received_pull_requests_count=64i packets_received_pull_responses_count=3018i packets_received_push_messages_count=274i packets_received_unknown_count=0i packets_received_verified_count=3375i packets_sent_gossip_requests_count=162i packets_sent_ping_messages_count=2i packets_sent_pong_messages_count=17i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=88i packets_sent_pull_responses_count=55i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:35.458239892Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=6i LegacyContactInfo-pull=16i Vote-push=7i Vote-pull=122i LowestSlot-push=0i LowestSlot-pull=17i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=4i EpochSlots-push=179i EpochSlots-pull=2144i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=24i NodeInstance-push=6i NodeInstance-pull=31i DuplicateShred-push=0i DuplicateShred-pull=264i SnapshotHashes-push=1i SnapshotHashes-pull=19i ContactInfo-push=7i ContactInfo-pull=19i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=206i all-pull=2660i
[2025-03-19T16:11:35.458243362Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=6i LegacyContactInfo-pull=0i Vote-push=1i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=71i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=6i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=1i SnapshotHashes-pull=0i ContactInfo-push=7i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=92i all-pull=0i
[2025-03-19T16:11:36.453852056Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 1383 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 1389 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 1389 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 765 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 2165 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 780 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 3111 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 1560 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 2349 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 3489 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 4026 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 848 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 3310 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 310 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 2377 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 1379 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 1772 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 3307 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 593 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 1707 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 1099 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 2349 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:36.453868236Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:36.453878605Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[SENTINEL.SH] [2025-03-19T16:11:36Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:37.454060835Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 839 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 109 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 839 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 1765 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 183 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 111 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 4111 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 183 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 3349 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 911 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 776 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 315 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 708 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 838 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 3377 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 109 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 839 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 4307 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 150 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 2707 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 2099 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 3349 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:37.454078115Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:37.454089115Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:37.459184731Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=10111i purged_values_size=900i failed_inserts_size=78i num_nodes=176i num_nodes_staked=0i num_pubkeys=176i
[2025-03-19T16:11:38.201323370Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=130i new_push_requests2=0i purge=1369i purge_count=0i process_gossip_packets_time=37462i verify_gossip_packets_time=93195i handle_batch_ping_messages_time=97i handle_batch_pong_messages_time=20i handle_batch_prune_messages_time=0i handle_batch_pull_requests_time=5145i handle_batch_pull_responses_time=2720i handle_batch_push_messages_time=13067i process_pull_resp=2028i filter_pull_resp=632i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=101i process_pull_resp_count=69i pull_response_fail_insert=2i pull_response_fail_timeout=1i pull_response_success=125i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:11:38.201340370Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=1424i pull_request_ping_pong_check_failed_count=2i generate_pull_responses=3582i process_prune=0i prune_message_timeout=0i bad_prune_destination=0i process_push_message=2408i prune_received_cache=1i epoch_slots_lookup=0i new_pull_requests=3009i gossip_pull_request_no_budget=2i gossip_pull_request_dropped_requests=2i gossip_pull_request_sent_bytes=31347i gossip_transmit_loop_time=3563i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2002006i gossip_listen_loop_iterations_since_last_report=762i
[2025-03-19T16:11:38.201343529Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=325i skip_pull_shred_version=0i num_duplicate_push_messages=470i push_fanout_num_entries=0i push_fanout_num_nodes=0i push_message_value_count=993i pull_from_entrypoint_count=0i prune_message_len=0i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=9677i
[2025-03-19T16:11:38.201345900Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=6i packets_received_count=2758i packets_received_ping_messages_count=4i packets_received_pong_messages_count=2i packets_received_prune_messages_count=0i packets_received_pull_requests_count=77i packets_received_pull_responses_count=1741i packets_received_push_messages_count=934i packets_received_unknown_count=0i packets_received_verified_count=2758i packets_sent_gossip_requests_count=154i packets_sent_ping_messages_count=3i packets_sent_pong_messages_count=4i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=80i packets_sent_pull_responses_count=67i packets_sent_push_messages_count=0i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:38.201348849Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=14i LegacyContactInfo-pull=10i Vote-push=19i Vote-pull=52i LowestSlot-push=0i LowestSlot-pull=8i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=2i EpochSlots-push=444i EpochSlots-pull=1145i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=15i NodeInstance-push=14i NodeInstance-pull=11i DuplicateShred-push=0i DuplicateShred-pull=160i SnapshotHashes-push=6i SnapshotHashes-pull=12i ContactInfo-push=26i ContactInfo-pull=7i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=523i all-pull=1422i
[2025-03-19T16:11:38.201352339Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=13i LegacyContactInfo-pull=0i Vote-push=19i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=394i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=13i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=6i SnapshotHashes-pull=0i ContactInfo-push=25i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=470i all-pull=0i
[2025-03-19T16:11:38.454291376Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 705 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 1110 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 345 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 2766 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 692 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 384 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 345 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 704 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 85 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 981 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 1777 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 1316 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 1709 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 1839 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 620 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 1110 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 1840 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 5308 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 1151 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 3708 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 3100 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 4350 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:38.454309496Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:38.454320566Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:39.454516140Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 177 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 2110 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 304 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 171 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 165 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 103 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 1345 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 1704 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 1085 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 1981 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 2777 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 2316 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 176 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 2839 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 1620 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 2110 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 2840 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 6308 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 2151 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 4708 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 4100 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 700 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 22
[2025-03-19T16:11:39.454533849Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:39.454544929Z INFO agave_validator::bootstrap] Total 22 RPC nodes found. 3 known, 13 blacklisted
[2025-03-19T16:11:39.462154309Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=2i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=10357i purged_values_size=1504i failed_inserts_size=79i num_nodes=176i num_nodes_staked=0i num_pubkeys=177i
[2025-03-19T16:11:39.462165999Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=541i new_push_requests2=56i purge=1268i purge_count=0i process_gossip_packets_time=38856i verify_gossip_packets_time=88236i handle_batch_ping_messages_time=142i handle_batch_pong_messages_time=0i handle_batch_prune_messages_time=4i handle_batch_pull_requests_time=5051i handle_batch_pull_responses_time=845i handle_batch_push_messages_time=18791i process_pull_resp=643i filter_pull_resp=158i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=180i process_pull_resp_count=43i pull_response_fail_insert=1i pull_response_fail_timeout=0i pull_response_success=132i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:11:39.462171159Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=313i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=3676i process_prune=4i prune_message_timeout=0i bad_prune_destination=0i process_push_message=3049i prune_received_cache=8i epoch_slots_lookup=0i new_pull_requests=3275i gossip_pull_request_no_budget=3i gossip_pull_request_dropped_requests=11i gossip_pull_request_sent_bytes=29268i gossip_transmit_loop_time=4097i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=1999637i gossip_listen_loop_iterations_since_last_report=803i
[2025-03-19T16:11:39.462175889Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=341i skip_pull_shred_version=0i num_duplicate_push_messages=609i push_fanout_num_entries=175i push_fanout_num_nodes=1573i push_message_value_count=1146i pull_from_entrypoint_count=1i prune_message_len=21i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=9885i
[2025-03-19T16:11:39.462178848Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=3i packets_received_count=1829i packets_received_ping_messages_count=5i packets_received_pong_messages_count=0i packets_received_prune_messages_count=21i packets_received_pull_requests_count=79i packets_received_pull_responses_count=633i packets_received_push_messages_count=1091i packets_received_unknown_count=0i packets_received_verified_count=1829i packets_sent_gossip_requests_count=1347i packets_sent_ping_messages_count=0i packets_sent_pong_messages_count=5i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=88i packets_sent_pull_responses_count=61i packets_sent_push_messages_count=1193i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:39.462182608Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=12i LegacyContactInfo-pull=14i Vote-push=22i Vote-pull=70i LowestSlot-push=1i LowestSlot-pull=4i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=1i EpochSlots-push=467i EpochSlots-pull=166i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=8i NodeInstance-push=12i NodeInstance-pull=8i DuplicateShred-push=0i DuplicateShred-pull=18i SnapshotHashes-push=1i SnapshotHashes-pull=7i ContactInfo-push=22i ContactInfo-pull=16i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=537i all-pull=312i
[2025-03-19T16:11:39.462186259Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=13i LegacyContactInfo-pull=0i Vote-push=24i Vote-pull=0i LowestSlot-push=1i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=534i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=13i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=1i SnapshotHashes-pull=0i ContactInfo-push=23i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=609i all-pull=0i
[SENTINEL.SH] [2025-03-19T16:11:39Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:40.454732895Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 187 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 3110 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 359 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 238 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 174 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 359 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 176 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 2704 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 568 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 2981 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 968 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 3316 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 1176 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 3839 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 2620 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 174 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 3840 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 7308 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 3151 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 5708 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 5100 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 1700 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 28 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:11:40.454750254Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:40.454762804Z INFO agave_validator::bootstrap] Total 23 RPC nodes found. 4 known, 13 blacklisted
[2025-03-19T16:11:41.454948242Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 40 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 4110 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 122 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 107 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 180 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 95 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 1176 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 486 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 1568 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 3981 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 1968 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 4316 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 2176 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 4839 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 3620 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 166 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 4840 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 544 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 4151 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 307 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 6100 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 2700 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 40 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:11:41.454965081Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:41.454977331Z INFO agave_validator::bootstrap] Total 23 RPC nodes found. 4 known, 13 blacklisted
[2025-03-19T16:11:41.459311279Z INFO solana_gossip::cluster_info]
IP Address |Age(ms)| Node identifier | Version |Gossip|TPUvote| TPU |TPUfwd| TVU |TVU Q |ServeR|ShredVer
------------------+-------+----------------------------------------------+---------+------+-------+------+------+------+------+------+--------
162.19.43.1 me| 2501 | BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar | 2.2.3 | 25000| none | none | none | none | none | none | 2405
147.28.198.57 | 44 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.172.191.11 | 44 | HMU77m6WSL9Xew9YvVCgz1hLuhzamz74eD9avi4XPdr | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
38.46.221.182 | 4114 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
147.28.165.119 | 126 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
38.46.221.182 | 111 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
5.199.164.171 | 312 | 4mdQGkbfrFvo5eG722xCXik4hn3xzu2EJD47g7Hd4HfH | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
35.194.23.154 | 99 | DGLzpbJ2ckLs4bsUT3Am5CkCCVgPwGumsKw3hEtKDxdz | 2.1.13 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
198.244.253.130 | 1571 | 9AyV9DykRupCenmHzvFckXLa2Hg8kvG61c6yX215EZCm | 2.1.14 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
34.220.208.242 | 184 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.115.212.90 | 126 | Cw6X5R68muAyGRCb7W8ZSP2YbaRjwMs1t5sBEPkhdwbM | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
46.166.162.49 | 99 | HzWdnV141bP1PXXce4NJ6NCJgd6jr3kMaySuevbpgwaV | 2.1.13 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
147.28.165.81 | 99 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
67.209.54.163 | 99 | BrX9Z85BbmXYMjvvuAWU8imwsAqutVQiDg9uNfTGkzrJ | 2.1.15 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
188.214.130.100 | 90 | 9qyNwcztJXEbHWpq8YBHAc3gX1Cd9WdeSiaNEtrQMUp7 | 2.2.2 | 25000| 25005| 25003| 25004| 25001| 25002| 25012| 2405
67.209.52.66 | 1178 | FW33UVu8qPN7bvCYgmiFpXrXyDMN9PSUfVhkUAtyC6v4 | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
52.13.43.37 | 1180 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
64.130.34.4 | 852 | 5aNSc9MuJn5JX5jiSZv2YBMRxcdYSiKjv6uyDbRTPUPu | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
45.250.254.67 | 1178 | 4wBKoRVAAvs3xBUm92PE3h5x7XQFsvDykSq69Gokh44n | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
64.130.52.151 | 1505 | 2kkTnE2JWek71pTusytU3GJtMbWCf8dnLwQBGxMS7ap5 | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
23.109.159.4 | 3696 | CTy6yhjhzEVZZBbQzCeuKz5XPPL83dPyav1ZKv7DHx6 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
141.98.219.9 | 4114 | De2UiLDHRcfSxf2cnAwKu8yNxmq4gNEK4VcifyffiWsj | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
208.91.111.183 | 1150 | CSase6DNS3RM1Xeqt3jmwA2cjZnApznZa4W4Dhi1PjRB | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
47.76.150.188 | 605 | 4v3dMU4Gp5EEb1MgyBctoQ6LQ9NZmUC1ZJH94JyaER8G | 2.1.11 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
40.160.1.59 | 490 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
185.26.10.121 | 5958 | FHdLcYmSu5vUQ5UG65teEC5rsNqMeyRtcM3imQAtargp | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
3.214.143.17 | 1178 | EjRGzg256yWQ5QffDc8htmWSabdFPMe2LAEGgtA6XkBF | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
52.19.134.152 | 2915 | 9CFn9Yw3R1HKDHAwpQWdcGaDxgdSPr6u5ebZSLZ9DsVm | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
13.217.130.6 | 1178 | FEceNs7TeZ6doKUC933bvUswmvdEPpbL1Utm8Wu7A4Ye | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
67.213.115.43 | 1505 | 6Xd5mg1HbVUcChViLZp8CSfDkex6LTWcFLtVnAJnwf3y | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
208.91.111.147 | 6267 | DW4yMdNwpRm2WjwYHgwiwx2ALz6AooJLz3gCtfMYpdJs | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
141.94.154.228 | 1572 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
103.50.32.200 | 1178 | 9DAgzGqzG6vUUx9ZZ9yDSgqqusyezEMDqeeMaGAnb1E5 | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
79.137.101.152 | 116 | CL5v3g3Vm1ieqnxprp8342FcrDrhq3Kr21TRQ3qEBrbU | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
104.238.147.73 | 516 | 2c5yEKTbq8LcWutihtqgZ3yFj7dHgKwLYEygYpyxFdDb | 2.1.11 | 10000| 10005| 10003| 10004| 10001| 10002| 10011| 2405
204.16.247.19 | 516 | 5UHT2MnFvARGe5mGbKZth5Yh546UWmPtucQYBGTYsi4n | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
162.19.43.1 | 1505 | FDQHfbqgSUk94XKFKWu6E8qidL7bwGEXDPzAoTVTXEDm | 2.1.14 | 25000| 25005| 25003| 25004| 25001| 25002| 25011| 2405
103.106.59.111 | 1125 | J7LhLJKkLLRCAzgBpTUK7Zp7duNbcAoUwxdCeLe4CmDG | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
141.98.217.117 | 1505 | EQXyPEDXptUEVzVz4XdJmhKjbUTZyNr7WmXRtz1AZepn | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
186.233.187.64 | 4472 | CyQF5447bJKZLe4dS5j7q8swsgz71sngo5Ay1hSWTb1z | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
60.244.89.49 | 3985 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8400 | 8405 | 8403 | 8404 | 8401 | 8402 | 8411 | 2405
3.93.4.50 | 1505 | FrNPkvFUWEd3j2M1uz5u1ZJTKfaCgbHu7r59sD6zhecY | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.91.106.75 | 1191 | 3wyX7mT1AUK46aJrgzZ9cuS2UYbtSYeASx6mAz1hBzqW | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
74.63.235.2 | 4468 | 36GpUgtojZTXV5NiNLG4GwfJBLd6s5Qtp5XbZE22S6o8 | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.34.6 | 1506 | Ddeqnfegkm2qA917L2o43sjUny2TSt9wK1BcbH4CQwY4 | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
103.106.59.105 | 3388 | 3TCq6rV6TADTpdD19aYhKs2zq2j76mqftWQ3EdTosDqJ | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
141.98.216.89 | 1505 | 39cvwUEpgka9bU7Sn4my82VViMDWaCxi4YoPevfZxLf3 | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.46.114.58 | 1972 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
40.160.1.61 | 4320 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
160.202.128.173 | 3789 | 97YUjL2EK42M6jG5VA4fKuVxGXDfxsC5Zawd9haLQJGk | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
79.137.99.55 | 1177 | Bt8yXrWAXH8X3kkZv8kzmaiyWtYwz2FFSZuuiXP6oqR2 | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
15.204.47.154 | 1178 | Bas9kLa5pr5Pi6T2Pug6PMfXrzT5QunHbrE2EYyAwyBo | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
5.199.164.180 | 1954 | 2562dAN2v5tGy9QJxdSFnmKKFND1MducVZLdB8b7P2WK | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
64.130.34.5 | 2308 | 7aTwCz9PBC5nwj1m88NVx3tF8ytJhoAb8uj3FQfZ9TWS | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
15.204.241.62 | 392 | EaHyBMU3fvZMaTn2Jb7qM83gbLCweyJNiETbVB2hkyP9 | 2.1.14 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
67.209.52.78 | 2183 | 69UoqUVy96b8zKBECrsNf2KgHkyWqJAmdwQLGWFNrQcH | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
45.250.253.233 | 1461 | 6CjNXemJGFYhJtn1zUqqaiyXMWo3XJQLfFRnvoDm2gY9 | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
204.16.244.26 | 1178 | 3AqQUgfbGoNpStyVCvkvz5MeSMw67HmMdz2KtCLdWK7J | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
63.251.232.229 | 2180 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
160.202.128.135 | 1505 | 6W3aGhPzeTZqmGKrS22SxVeab2kdF5QEBVJQ3k7vwj5Q | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.52.197 | 1361 | QTgZHGRk1aUXSLrgBzgAowbC95c7hES2PWWYyHuu3og | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
52.195.132.188 | 2181 | J3rSdVJimWDjjTS1eypBwgyBo36BYRfGcNQwCduCQD9Y | 2.1.15 | 8800 | 8805 | 8803 | 8804 | 8801 | 8802 | 8811 | 2405
23.109.172.204 | 5710 | 5ddQSb2cVyuGuncX3KyT97mVMbcfLF3vXbAYXBRE6ZJc | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
204.16.247.76 | 4327 | B95quggvWB2yUJM5jrDbSmZwFBL9HFNXqLMnewFoGG5M | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
74.118.143.126 | 2593 | 86NTgDwxACHL4W4DbQmHhcmeSzzSyS6NBnmvxWJc1NXP | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.115.193.194 | 3417 | Fr8WCCCBxPtgPcy3R8Uh5sQfuP7af5KpgRT6K8thq5AZ | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
64.130.33.26 | 5853 | Bd9vEWKxyByeYiroT17XHRCkjnj5JwfeMca34CbDfkSD | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
208.115.226.166 | 2316 | CBxqCJrKUvoYjWswgiVVfhCWX32AKSiZABTdBrvfKnEz | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
135.148.171.49 | 2316 | 7D3q1VC1L8djZLjb78619sqLY3RnEU7LQeQRDEYRUhQp | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
194.67.206.120 | 2301 | 5qLppNuoh2HucuUpiYX7NBBdyDAQPVGK2N3Fed6S5c8M | 2.1.15 | 9000 | 9005 | 9003 | 9004 | 9001 | 9002 | 9011 | 2405
145.40.106.183 | 311 | dw1xbcZQHmzA2zEPEqECp3SswUWdQ5sbdjBrKMEu7gK | 2.2.2 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.8.107.168 | 2316 | 6tBs7Eq1Cx6YFtabvLRZeG5Jk7VKGQGNvrTZxsjLGjGG | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
208.91.110.67 | 3476 | 6NNoD2pYEhvF3G4AaA1rhiQ2Ua8ojqGVixWUR5QWM5dP | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
163.114.159.219 | 532 | C8ZAr6Zz5hMF8wz38TMhg4gWfBrrkBA3kzQhp1sKz1JD | 2.2.0 | 11100| 11105| 11103| 11104| 11101| 11102| 11112| 2405
208.97.227.22 | 4843 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 30000| 30005| 30003| 30004| 30001| 30002| 30011| 2405
103.50.32.201 | 4968 | 3uQRPJ9uf8dadzCo2GFgDg65Y3qJUTXCUzLfojH2qf9j | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
63.251.106.87 | 3624 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
38.97.60.6 | 170 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
3.16.49.73 | 1093 | 2tb5U6XQYYkZ5jWCRVqoHikQiKF6PmuDnQvNSEuLJXo5 | 2.1.16 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
217.28.63.147 | 181 | 3THprEXdsRkJdsBvqnMf6RAVenyqW2ax9y3SRHfg3Jj1 | 2.0.15 | 8002 | 8007 | 8005 | 8006 | 8003 | 8004 | 8010 | 2405
20.253.158.75 | 1420 | CmrmEwq4LfuxsbRBpaQnZbo8ojFV5iKdWyGxcFaziCRs | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
148.113.206.128 | 4844 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
64.130.57.83 | 1769 | DC7StQXH1n9hrJe9d5ZVXusBYZbW8G6zNKEBCZyam5jw | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
72.46.84.115 | 1941 | DkawvKmKVBDcn9R8c76gZ3eqNvrySmVxArz15z6hGcpG | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
103.50.32.203 | 1178 | 7Zk5LM4AFBh9S1pMAKXYcAMfoqLAGFZUPmHoCz7BiTYz | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
160.202.128.127 | 5682 | HhhhXfqFJtJ835v2nPzTcovUktFAHi7LLnX2u93rSZtz | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
64.130.57.78 | 391 | 3wLP3s4GVeMsV3DYGUXBumPbR6njNCUBVPTJ9obQTscw | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
204.16.244.203 | 1138 | 2v56rXf2WvNJBHQenasMGJVVTqXeHYqxuctfmBRXrNPQ | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
147.28.133.51 | 548 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6000 | 6005 | 6003 | 6004 | 6001 | 6002 | 6012 | 2405
208.115.212.70 | 2316 | Fap57xttR6sj1UpS6XvbVf3n2tLXLzaDvZEFGdpR5Vt5 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
67.209.54.90 | 5853 | 7pbH563fFai2Gm8aXGi27Toj1i7x55rGp7QQ8ZQt6C7i | 2.1.13 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
212.232.28.212 | 4155 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
208.91.107.3 | 5095 | 8GFyFHMa8oobGzK6Bab1qdExMqjvQoFeHYTpkfJiR8NH | 2.1.15 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
107.6.94.200 | 311 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21810| 11805| 11803| 11804| 11801| 11802| 11812| 2405
72.46.84.117 | 2316 | 9KyKWnUDd2DPoGys99HdyLJ553ruUGDov45jUv9pLTsh | 2.1.14 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
91.134.8.113 | 6162 | 6dYSW74ej3mBhXFTASy8NGLEhYZU4sKqoFwXemxVyJDp | 2.0.19 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8009 | 2405
51.20.239.237 | 6104 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
47.130.39.171 | 516 | 8UAhoTURfDU5UaL3KK5Dy4W62R3rQNceHs9aUkxKqi8X | 2.1.15 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
57.128.141.123 | 1505 | sSRDQFY8Gri48fzXCENWgsSRcp5pheZR4TeB1pSoWJs | 2.1.11 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8012 | 2405
147.28.198.107 | 1505 | dw2gYZBQNisftpoCZ3XrNe9BiCyCG4reMKSejmztVZ8 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
185.189.44.238 | 516 | 3ne7n82Kqf1zPo4obYTnQA8tJBDuSEYyvKYS89Mbky4q | 2.1.13 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
74.118.143.62 | 764 | 8PikNto2ADRtVbduxyEVd7DRUf5kVuRK3oFQ1csDeZF3 | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
141.98.217.57 | 2316 | 2CbEVhecSXz17JtzWtCsLSrnH8YHYtRCYi7co7wNPLXj | 2.2.0 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
194.126.172.170 | 1505 | 2ZZkgKcBfp4tW8qCLj2yjxRYh9CuvEVJWb6e2KKS91Mj | 2.1.15 | 8001 | 8005 | 8003 | 8004 | 8000 | 8002 | 8011 | 2405
103.106.59.87 | 1039 | G29JbGLwxgmtYqkqZntxNgBxyg8NkKSHQrZbJvYxaekf | 2.1.15 | 8801 | 8004 | 8002 | 8003 | 8000 | 8001 | 8010 | 2405
132.145.143.18 | 2915 | FsdCY72YcXifByx61gcsvmZ5NVafZkquEqCj94MPx59Z | 2.1.11 | 8001 | none | none | none | none | none | none | 2405
18.117.197.60 | 10007 | APnGAfs4Zh67o1y77LocktHmgAj968BDPAKqQx2oFV8c | 2.1.11 | 8000 | none | none | none | none | none | none | 2405
45.77.252.43 | 10007 | zXNXTerWhz7Mc5JQQgsteJ1E3fz16S9kdPNJue796iH | 2.1.15 | 8000 | 8005 | none | none | 8001 | 8002 | 8011 | 2405
65.109.76.130 | 10007 | FVvc9sktQtTe3h3wHK9Pijkqt28i7EV9WMznV17nqcrQ | 2.2.0 | 14507| none | none | none | none | none | none | 2405
208.91.111.182 | 10007 | TUJAuFA2pVoVwxkz634nd5jc7FoaLpyyaPCdXvWFYwk | 2.2.0 | 8001 | none | none | none | none | none | none | 2405
186.233.187.187 | 10007 | 3d3nZzZ9g2TC7PQfRuv1Y6eTgBxjKeQvua8JRLgGFUMx | 2.0.20 | 8000 | none | none | none | none | none | none | 2405
88.216.198.139 | 855 | A7dsWYJghLoALaHoViXz8bi1MhUBPCpsraCYwbs1puei | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
84.32.32.14 | 4844 | 9dfk2wFcoREXo4pscPPA3zPsaVMuxLaQ1TtEK1s5uyoC | 2.1.14 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8011 | 2405
93.187.218.219 | 2704 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8020 | 8025 | 8023 | 8024 | 8021 | 8022 | 8032 | 2405
88.97.255.77 | 8768 | 9pSiRik1sC7Yex71jVTUnSUeeQpEGoJ38Ji5GKhf55Hz | 2.2.0 | 8000 | none | none | none | none | none | none | 2405
92.38.186.52 | 10007 | 27adQ5Rih7iVMwZHrjo3Hfkga6ZsKk8gRiFSgXmmNFv2 | 2.2.0 | 8000 | none | none | none | none | none | none | 2405
147.28.179.215 | 2227 | DbSa4G3AJzPFVftjQowiaKqUsTD1rEDjQp1nUT5E3SyQ | 2.2.3 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
147.28.165.21 | 10007 | 5kEbSzXhNM9j73HMhfq3tEi4rHyR1YdYRpUiLxkgV7BV | 2.2.0 | 14521| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | ASptD91PuiYsc1Www492kyABnYN29rdZJBMrv1JbXgrS | 2.2.0 | 14501| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | JDHrDMEsZgn6WmtRrfGtYdfz5VPyX8FgnDmnTwfKXtM3 | 2.2.0 | 14511| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 3PRwGdfiqn5aVMTE4xPZLPLQC7bAdseJTBPY8M8seB4L | 2.2.0 | 14527| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 95efi8SDAWz5eZpsUMu6fkGayfEyTP6HGp3VC22JhxQ2 | 2.2.0 | 14533| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | FQXgQpT1rY9zGMf5LtXRhiPcRSWQpBLzqXQSALhkVxgv | 2.2.0 | 14522| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 2MP6fXk1tcgcqeh4mSDnJLjWpa1kAUnxn6FoS8Zv9rys | 2.2.0 | 14512| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BzxpeHHQfnrr2db6wqsyd9GNtKqVG4KSAo2CWEeekQa1 | 2.2.0 | 14502| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | FRpMYcFKQ8u2Z16ASBrEGixBNsWqyVWbZQqnG65CoQ8S | 2.2.0 | 14532| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 2Y9hELJXQjGEHKFfUJGbcwVyMqqWnZdKPfxPBk1bcVjS | 2.2.0 | 14537| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | GzJqmHczxGyMM64XniBzFhrX7uRuEQ2QrQgSWD55YUUP | 2.2.0 | 14523| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | GPgZC9oBwvvXAfXzj92Tpq6K13CHmimzwbqxcNXG8eQC | 2.2.0 | 14503| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 9KBugAvyB7xVdRF43n6LJXCn15sJvvEjqLCfARk9Ghx8 | 2.2.0 | 14513| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 9MzZnJfqYAbc3JhLdweUEgd5ouV45eJpy2RuUc5rgipS | 2.2.0 | 14529| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BLp7uGyHT3TNrJbi5b5Jubxz71usyBUu8RFqLvvUQkLz | 2.2.0 | 14514| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | D18sA29pNUtgs4cJiQGXUun1fCcUHfHwg9ck3YDjXtGU | 2.2.0 | 14504| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 9Ae9eFw7Lx1ovnLTmFxAvREo8Jzd12jvdjmFUj3iAPju | 2.2.0 | 14530| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | p4oCRoSAoVDMS2JLpvdnrJTYV6QuPZkXEJfUSNg766R | 2.2.0 | 14524| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | F6uZ5Snp1ixFfp66Rs1VkCvk3s4Q76fxTw913Xg6sL2T | 2.2.0 | 14518| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | RHS7qE6JjWbk9W8hsbeQvEW5cyKFvx8uH6MTM7hhJBF | 2.2.0 | 14520| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | EEPM5FzRqRaXWi9LEoUHQp9NzQR7ghiMU1GjhvRuQN26 | 2.2.0 | 14505| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BeT1whVNgaBgawqhLLVyFSAevXP4qZEz6K4844m2NBpY | 2.2.0 | 14515| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 9djoBJW2fV4mJ39ksNVZ5YwEM2gBwJHSp39oheRnTTR6 | 2.2.0 | 14525| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | AHWqqZ6LZVQFTFBVqsrfmavwPK2XHdPkymHaUsm9LdSW | 2.2.0 | 14508| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | H1rVCxKS5nGfxSoancENweaRc6iZDGfr4ip25KEspfgP | 2.2.0 | 14531| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 3aM1RiqXbyohQ3AtgNz66FyQogjCWErYhxgHUWBT9xU5 | 2.2.0 | 14526| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 2rMtb4KRjTCtb6NAsYCEuxLgiwyY35WCiPV1i1JkqcUk | 2.2.0 | 14516| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | Hv4V2ELDNB68KCnj67CR8r3MKxjQcXop8P3sV7JMfbj1 | 2.2.0 | 14506| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 4TTWkhgWGjTzKs4WQThLiPC7UhFXo9F8hqmkxJKaUMqg | 2.2.0 | 14540| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | CjMVxKKdbxpJCtBz6nGzD3hEcU6nhCL4LeBnPN5fHfGQ | 2.2.0 | 14507| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 89qmzF7z67yY7ddSuGNfETtnHBcHwcicVkcJBnBvwaFX | 2.2.0 | 14517| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 5eF3a5Ryyhc6Qiz6B21xKcaToARVm4UQDzYJNhCXRnjD | 2.2.0 | 14509| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | HsDD7sQQidYpf3PWyKbdWzrFafQzFg6g6LaAPt4FZFks | 2.2.0 | 14542| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 6yK64d6jtLvuuZsU2j7b9ANfSRdD4gQQc7APvLcQRawa | 2.2.0 | 14534| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 4FqrZ7MK3DAVdcR4NWPsXBtp23xQmaVMLCuWuN6XWNQJ | 2.2.0 | 14536| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | AWbhMuRKCzkxdnTBvWXqCBj9un6QFZ5VAEYgATLbzURp | 2.2.0 | 14528| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BUYEAcwyNA7bSNeXmndVS6orp7TxFc9JtdGyCLZH5AuP | 2.2.0 | 14500| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | D1DGY57hMNGopiTxTP65cQVYYJJBGw2SyWZis3LzKNVu | 2.2.0 | 14519| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | HULnKLeejyjPQ53UhrwG7JZqFfyKcy8Da8152FYsVj6h | 2.2.0 | 14545| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 2Q2cSDNy6tAPQyx5X3VJESqDXJTFasJ3CrYjbjd9Z8w3 | 2.2.0 | 14535| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 8Kgnpgu4nMhLhCfco7SPXXcY4KwLGrTd8iP6DC2ecGY2 | 2.2.0 | 14510| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BiBTs3eiykKMYceUsPG2vZq1BgWQ5AN6A1pt834kWZ4G | 2.2.0 | 14546| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | Gs95Syj8n8Pa48Q1waAdGW3rdyaND8HuawpafHsHLpC8 | 2.2.0 | 14538| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 2rVdL8VjpCXWaVvpmHXTxSLqJqtD4Tfy2tNy7n2ps4Ys | 2.2.0 | 14539| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | HqbfrXbyGSxFyeon1UWZ8pXp44jkAcdLMsKQZVuM1Q9A | 2.2.0 | 14543| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | J4sc9JoKDMAutT3jypew7593JPvCnsjRSmW6RRtcZ72d | 2.2.0 | 14544| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 3Gz2Zqski8UXoJ4XNSy43t9BiY5PW3JZtNnJp1pBkAVL | 2.2.0 | 14541| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 5Qbn7Y4S8hmV9qxZP5odW6Nfd3FNvgRdJepp8QrRqX75 | 2.2.0 | 14547| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 3GuqvF3MK5hkguVxnaKnwJmVLhPXkVecoFK4brDbuvW2 | 2.2.0 | 14558| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 8VvmLywwYvkrXCUGPJW61ryEWQg5ymGTkWv7hY5CGwmn | 2.2.0 | 14559| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 7e2aqTRJgx7xgV4dUgt7DtQRFtKkpn9SEDWC3DfjMDsD | 2.2.0 | 14625| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 4VzqgFMtVG41NKNniibPqfhf9GvvM6LuncWdSZp2Qu5y | 2.2.0 | 14505| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | 8eMpRcmaVUyvU9ws3YnPxnjzDUn6x9HLytmGswTzK3BC | 2.2.0 | 14525| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | BDgqfmLxprbKurSdirU9ZNNn8idP4cRoTP1yCS29h1CP | 2.2.0 | 14594| none | none | none | none | none | none | 2405
147.28.165.21 | 10007 | D2P2Q6Cnary5NdmBqFEASuhUMsTZkHAubcUApTrrP4N6 | 2.2.0 | 14521| none | none | none | none | none | none | 2405
145.40.117.251 | 981 | DmCowGH9DUHYCetfGaWzPzYCi455yDxewcycdyWuPLjx | 2.3.0 | 8000 | 8005 | 8003 | 8004 | 8001 | 8002 | 8012 | 2405
35.197.53.105 | 3988 | EBTPS7efmbiRLS4spFdB63q69Vzg1KuZUYwddk4Ec4aV | 1.18.21 | 8001 | none | none | none | none | none | none | 2405
145.40.126.137 | 44 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8001 | 8006 | 8004 | 8005 | 8002 | 8003 | 8013 | 2405
Nodes: 110
Spies: 67
Nodes with different shred version: 2
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 45 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 4115 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 127 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 112 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 185 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 100 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 1181 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 491 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 1573 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 3986 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 1973 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 4321 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 2181 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 4844 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 3625 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 171 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 4845 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 549 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 4156 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 312 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 6105 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 2705 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 45 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:11:41.462754529Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=0i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=10906i purged_values_size=2182i failed_inserts_size=132i num_nodes=177i num_nodes_staked=0i num_pubkeys=178i
[2025-03-19T16:11:41.462769229Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=1702i new_push_requests2=367i purge=1082i purge_count=0i process_gossip_packets_time=59169i verify_gossip_packets_time=91488i handle_batch_ping_messages_time=280i handle_batch_pong_messages_time=0i handle_batch_prune_messages_time=62i handle_batch_pull_requests_time=4099i handle_batch_pull_responses_time=1731i handle_batch_push_messages_time=35351i process_pull_resp=1240i filter_pull_resp=420i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=169i process_pull_resp_count=89i pull_response_fail_insert=53i pull_response_fail_timeout=0i pull_response_success=217i num_redundant_pull_responses=4i save_contact_info_time=0i
[2025-03-19T16:11:41.462774399Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=730i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=2866i process_prune=32i prune_message_timeout=0i bad_prune_destination=0i process_push_message=2649i prune_received_cache=1i epoch_slots_lookup=0i new_pull_requests=2982i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=24913i gossip_transmit_loop_time=3700i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=1999492i gossip_listen_loop_iterations_since_last_report=884i
[2025-03-19T16:11:41.462778869Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=190i skip_pull_shred_version=0i num_duplicate_push_messages=539i push_fanout_num_entries=731i push_fanout_num_nodes=6282i push_message_value_count=1093i pull_from_entrypoint_count=1i prune_message_len=228i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=10192i
[2025-03-19T16:11:41.462789529Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=3i packets_received_count=2229i packets_received_ping_messages_count=12i packets_received_pong_messages_count=0i packets_received_prune_messages_count=221i packets_received_pull_requests_count=62i packets_received_pull_responses_count=907i packets_received_push_messages_count=1027i packets_received_unknown_count=0i packets_received_verified_count=2229i packets_sent_gossip_requests_count=5472i packets_sent_ping_messages_count=0i packets_sent_pong_messages_count=12i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=88i packets_sent_pull_responses_count=51i packets_sent_push_messages_count=5321i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:41.462794199Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=17i LegacyContactInfo-pull=8i Vote-push=25i Vote-pull=139i LowestSlot-push=1i LowestSlot-pull=6i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=1i EpochSlots-push=467i EpochSlots-pull=447i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=5i NodeInstance-push=16i NodeInstance-pull=5i DuplicateShred-push=0i DuplicateShred-pull=48i SnapshotHashes-push=0i SnapshotHashes-pull=6i ContactInfo-push=24i ContactInfo-pull=12i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=550i all-pull=677i
[2025-03-19T16:11:41.462797629Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=16i LegacyContactInfo-pull=0i Vote-push=20i Vote-pull=0i LowestSlot-push=1i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=467i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=16i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=0i SnapshotHashes-pull=0i ContactInfo-push=23i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=543i all-pull=0i
[2025-03-19T16:11:42.455156471Z INFO agave_validator::bootstrap]
RPC Address |Age(ms)| Node identifier | Version | RPC |PubSub|ShredVer
------------------+-------+----------------------------------------------+---------+------+------+--------
147.28.198.57 | 389 | dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 5111 | 88B9r8s7adZZirAxGNQfvT2Zy7vwZtfCATaX27ZLP7Y1 | 2.2.3 | 8899 | 8900 | 2405
147.28.165.119 | 389 | dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS | 2.2.3 | 8899 | 8900 | 2405
38.46.221.182 | 389 | 3zvXem8vqvDYos6BKu66FV84tmDdQwgLxYSempy8tSrs | 2.2.3 | 8899 | 8900 | 2405
34.220.208.242 | 388 | CKMqpoZzrqeobgVMsS9Es8UpRUjdhT3tA7CTPoXC3u6i | 2.1.15 | 8899 | 8900 | 2405
147.28.165.81 | 389 | dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB | 2.2.3 | 8899 | 8900 | 2405
52.13.43.37 | 415 | cpX6HzRZejiSPjHL1y1AP1oy3mVdHLqFh7wWXSvttoh | 2.1.15 | 8899 | 8900 | 2405
40.160.1.59 | 1487 | DjyUQ5ogut25znd62Tiy8xJ95teihrnngsZnkfJ5w3Sp | 2.1.13 | 8899 | 8900 | 2405
141.94.154.228 | 2569 | FtRi287TyF2Bax1y7FtvQRyzmeFoxQc3Jvxy85fTQMho | 2.2.1 | 8899 | 8900 | 2405
60.244.89.49 | 4982 | AFv6MmQhW15M2YTMqUCuR51egdsPB1e79tDFKLiTgNMC | 2.1.13 | 8899 | 8900 | 2405
64.46.114.58 | 2969 | 9T2i5QxiSvZamMxoFPLztxUKUQuXy3UK8NcoJoJ6Vc2Z | 2.2.3 | 21811| 21812| 2405
40.160.1.61 | 5317 | 6a33MLxnqugbrmKr2v67bpatCrAahupkmCmrUwqVcghY | 2.1.13 | 8899 | 8900 | 2405
63.251.232.229 | 3177 | DWYHvyUFxm514djpz9gA3PToE5zMhx3rcL4ZXrETpq7N | 2.2.3 | 21811| 21812| 2405
208.97.227.22 | 5840 | Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ | 2.1.11 | 8899 | 8900 | 2405
63.251.106.87 | 4621 | DiwJZxYKQbWBn59EhUhkiB3hM1TLWiQoGvD5mpUHJscu | 2.2.3 | 21811| 21812| 2405
38.97.60.6 | 1167 | Egedu8XWXw6bKEPSjxPPxWzq2bXBuHniXfXasn8EXYYA | 2.2.1 | 8899 | 8900 | 2405
148.113.206.128 | 5841 | G54PmPWqB2Naaj556wo9PshZR6AXX9yEE9XCKeb6AMLX | 2.2.1 | 8899 | 8900 | 2405
147.28.133.51 | 1545 | 2mobmc71wgFkYPCW33qrVNBMbvXc2LQKAaUTH4AVJgs8 | 2.2.3 | 6699 | 6700 | 2405
212.232.28.212 | 5152 | 7oteHfpDeHsrRFesXZTHrLK5q228Wxc29QisBrxx3xtw | 2.1.11 | 8899 | 8900 | 2405
107.6.94.200 | 372 | 6ZaNrpjNZMmtAaZ4Tjj7GhmgbqpRR9vnipoCgwqjCHR5 | 2.2.3 | 21811| 21812| 2405
51.20.239.237 | 7101 | 5weVtehXQW5HqcTxSyHr8vH3wQKPJsQHPLHLmG8sAs2g | 2.1.11 | 8899 | 8900 | 2405
93.187.218.219 | 3701 | 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu | 2.2.3 | 8899 | 8900 | 2405
145.40.126.137 | 179 | dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 | 2.2.3 | 8899 | 8900 | 2405
RPC Enabled Nodes: 23
[2025-03-19T16:11:42.455174041Z INFO agave_validator::bootstrap] Searching for an RPC service with shred version 2405 (Retrying: No snapshots available)...
[2025-03-19T16:11:42.455186690Z INFO agave_validator::bootstrap] Total 23 RPC nodes found. 4 known, 13 blacklisted
[2025-03-19T16:11:42.455201300Z INFO agave_validator::bootstrap] Highest available snapshot slot is 368429552, available from 6 nodes: [dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV, dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS, dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB, Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ, 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu, dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92]
[2025-03-19T16:11:42.455248199Z INFO agave_validator::bootstrap] Using RPC service from node dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV: Some(147.28.198.57:8899)
[2025-03-19T16:11:42.455262009Z INFO agave_validator::bootstrap] Using RPC service from node Ckjq9PRKgBLjGkvhwRzHmuzTfxV9C5fpi2L9vagneAJJ: Some(208.97.227.22:8899)
[2025-03-19T16:11:42.455274449Z INFO agave_validator::bootstrap] Using RPC service from node dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS: Some(147.28.165.119:8899)
[2025-03-19T16:11:42.455286969Z INFO agave_validator::bootstrap] Using RPC service from node dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB: Some(147.28.165.81:8899)
[2025-03-19T16:11:42.455276959Z INFO agave_validator::bootstrap] Using RPC service from node 7baP94Vgn9Sm9GVA3xZzDXARDhHDi25MoD1ZcheZtiqu: Some(93.187.218.219:8899)
[2025-03-19T16:11:42.455283439Z INFO agave_validator::bootstrap] Using RPC service from node dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92: Some(145.40.126.137:8899)
[2025-03-19T16:11:42.540714959Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 26ms
[2025-03-19T16:11:42.543290802Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 26ms
[SENTINEL.SH] [2025-03-19T16:11:42Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:42.803072662Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 110ms
[2025-03-19T16:11:42.896497987Z INFO agave_validator::bootstrap] RPC node version: 2.1.11 Ping: 144ms
[2025-03-19T16:11:43.222686612Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 254ms
[2025-03-19T16:11:43.224645064Z INFO agave_validator::bootstrap] RPC node version: 2.2.3 Ping: 261ms
[2025-03-19T16:11:43.463375228Z INFO solana_metrics::metrics] datapoint: cluster_info_stats entrypoint=2i entrypoint2=0i push_vote_read=0i get_votes=0i get_votes_count=0i all_tvu_peers=0i tvu_peers=0i table_size=11150i purged_values_size=2884i failed_inserts_size=135i num_nodes=177i num_nodes_staked=0i num_pubkeys=178i
[2025-03-19T16:11:43.463391708Z INFO solana_metrics::metrics] datapoint: cluster_info_stats2 gossip_packets_dropped_count=0i repair_peers=0i new_push_requests=1668i new_push_requests2=271i purge=1105i purge_count=0i process_gossip_packets_time=59313i verify_gossip_packets_time=84642i handle_batch_ping_messages_time=374i handle_batch_pong_messages_time=0i handle_batch_prune_messages_time=63i handle_batch_pull_requests_time=4133i handle_batch_pull_responses_time=788i handle_batch_push_messages_time=34275i process_pull_resp=604i filter_pull_resp=152i filter_crds_values_dropped_requests=0i filter_crds_values_dropped_values=154i process_pull_resp_count=45i pull_response_fail_insert=3i pull_response_fail_timeout=0i pull_response_success=84i num_redundant_pull_responses=0i save_contact_info_time=0i
[2025-03-19T16:11:43.463395827Z INFO solana_metrics::metrics] datapoint: cluster_info_stats3 process_pull_resp_len=296i pull_request_ping_pong_check_failed_count=0i generate_pull_responses=3111i process_prune=39i prune_message_timeout=0i bad_prune_destination=0i process_push_message=2769i prune_received_cache=6i epoch_slots_lookup=0i new_pull_requests=2874i gossip_pull_request_no_budget=0i gossip_pull_request_dropped_requests=0i gossip_pull_request_sent_bytes=29200i gossip_transmit_loop_time=3612i gossip_transmit_loop_iterations_since_last_report=20i gossip_listen_loop_time=2005528i gossip_listen_loop_iterations_since_last_report=941i
[2025-03-19T16:11:43.463398757Z INFO solana_metrics::metrics] datapoint: cluster_info_stats4 skip_push_message_shred_version=0i skip_pull_response_shred_version=0i skip_pull_shred_version=0i num_duplicate_push_messages=622i push_fanout_num_entries=718i push_fanout_num_nodes=5230i push_message_value_count=1275i pull_from_entrypoint_count=0i prune_message_len=332i epoch_slots_filled=0i window_request_loopback=0i generate_prune_messages=11007i
[2025-03-19T16:11:43.463400917Z INFO solana_metrics::metrics] datapoint: cluster_info_stats5 num_unverifed_gossip_addrs=3i packets_received_count=1879i packets_received_ping_messages_count=16i packets_received_pong_messages_count=0i packets_received_prune_messages_count=325i packets_received_pull_requests_count=62i packets_received_pull_responses_count=291i packets_received_push_messages_count=1185i packets_received_unknown_count=0i packets_received_verified_count=1879i packets_sent_gossip_requests_count=4909i packets_sent_ping_messages_count=0i packets_sent_pong_messages_count=16i packets_sent_prune_messages_count=0i packets_sent_pull_requests_count=80i packets_sent_pull_responses_count=51i packets_sent_push_messages_count=4762i trim_crds_table=0i trim_crds_table_failed=0i trim_crds_table_purged_values_count=0i
[2025-03-19T16:11:43.463403667Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats LegacyContactInfo-push=21i LegacyContactInfo-pull=1i Vote-push=29i Vote-pull=58i LowestSlot-push=0i LowestSlot-pull=2i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=1i EpochSlots-push=547i EpochSlots-pull=204i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=21i NodeInstance-pull=1i DuplicateShred-push=0i DuplicateShred-pull=20i SnapshotHashes-push=1i SnapshotHashes-pull=4i ContactInfo-push=34i ContactInfo-pull=2i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=653i all-pull=293i
[2025-03-19T16:11:43.463406797Z INFO solana_metrics::metrics] datapoint: cluster_info_crds_stats_fails LegacyContactInfo-push=22i LegacyContactInfo-pull=0i Vote-push=22i Vote-pull=0i LowestSlot-push=0i LowestSlot-pull=0i LegacySnapshotHashes-push=0i LegacySnapshotHashes-pull=0i AccountsHashes-push=0i AccountsHashes-pull=0i EpochSlots-push=524i EpochSlots-pull=0i LegacyVersion-push=0i LegacyVersion-pull=0i Version-push=0i Version-pull=0i NodeInstance-push=21i NodeInstance-pull=0i DuplicateShred-push=0i DuplicateShred-pull=0i SnapshotHashes-push=1i SnapshotHashes-pull=0i ContactInfo-push=32i ContactInfo-pull=0i RestartLastVotedForkSlots-push=0i RestartLastVotedForkSlots-pull=0i RestartHeaviestFork-push=0i RestartHeaviestFork-pull=0i all-push=622i all-pull=0i
[2025-03-19T16:11:43.476977672Z INFO solana_gossip::cluster_info] Saved contact info for 174 nodes into /ledger/contact-info.bin
[2025-03-19T16:11:43.818377227Z INFO agave_validator::bootstrap] RPC node root slot: 368429592
[2025-03-19T16:11:43.818525925Z INFO agave_validator::bootstrap] Local snapshot from slot 368420648 is too old. Downloading a newer snapshot for slot 368429552.
[2025-03-19T16:11:43.818560615Z INFO agave_validator::bootstrap] Full snapshot archive already exists locally. Skipping download. slot: 368420648, hash: 56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4
[2025-03-19T16:11:43.818609424Z INFO solana_runtime::snapshot_utils] Purging old full snapshot archives in /snapshots, retaining up to 2 full snapshots
[2025-03-19T16:11:43.818643354Z INFO solana_runtime::snapshot_utils] Purging old incremental snapshot archives in /snapshots, retaining up to 4 incremental snapshots
[SENTINEL.SH] [2025-03-19T16:11:45Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:47.147999501Z INFO solana_file_download] Downloaded http://147.28.198.57:8899/incremental-snapshot-368420648-368429552-4rbgbLBCs6jVnv2cTswsNwVkSB2weLo1B7McgySP7iEL.tar.zst (35509900 bytes) in 3.329282858s
[2025-03-19T16:11:47.149600399Z INFO solana_metrics::metrics] datapoint: bootstrap-snapshot-download total_time_secs=466i get_rpc_nodes_time_secs=21i snapshot_download_time_secs=444i download_abort_count=5i blacklisted_nodes_count=13i
[2025-03-19T16:11:47.151580060Z WARN solana_core::validator] Rayon global thread pool already initialized
[2025-03-19T16:11:47.151584760Z INFO solana_core::validator] identity pubkey: BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar
[2025-03-19T16:11:47.151588100Z INFO solana_core::validator] vote account pubkey: FwR3PbjS5iyqzLiLugrBqKSa5EKZ4vK9SKs7eQXtT59f
[2025-03-19T16:11:47.151820637Z WARN solana_core::validator] authorized voter: FDQHfbqgSUk94XKFKWu6E8qidL7bwGEXDPzAoTVTXEDm
[2025-03-19T16:11:47.151824107Z INFO solana_core::validator] entrypoint: ContactInfo { pubkey: 11111111111111111111111111111111, wallclock: 1742400238774, outset: 1742400238774943, shred_version: 0, version: 2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave), addrs: [145.40.68.1], sockets: [SocketEntry { key: 0, index: 0, offset: 8001 }], extensions: [], cache: [145.40.68.1:8001, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0] }
[2025-03-19T16:11:47.151841426Z INFO solana_core::validator] entrypoint: ContactInfo { pubkey: 11111111111111111111111111111111, wallclock: 1742400238774, outset: 1742400238774951, shred_version: 0, version: 2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave), addrs: [145.40.126.137], sockets: [SocketEntry { key: 0, index: 0, offset: 8001 }], extensions: [], cache: [145.40.126.137:8001, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0] }
[2025-03-19T16:11:47.151846376Z INFO solana_core::validator] entrypoint: ContactInfo { pubkey: 11111111111111111111111111111111, wallclock: 1742400238774, outset: 1742400238774952, shred_version: 0, version: 2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave), addrs: [35.197.53.105], sockets: [SocketEntry { key: 0, index: 0, offset: 8001 }], extensions: [], cache: [35.197.53.105:8001, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0, 0.0.0.0:0] }
[2025-03-19T16:11:47.151850746Z INFO solana_core::validator] Initializing sigverify...
[2025-03-19T16:11:47.151851916Z INFO solana_core::validator] Initializing sigverify done.
[2025-03-19T16:11:47.152160722Z INFO solana_core::validator] genesis hash: EtWTRABZaYq6iMfeYKouRu166VU2xqa1wcaWoxPkrZBG
[2025-03-19T16:11:47.152169232Z INFO solana_core::validator] Cleaning accounts paths..
[2025-03-19T16:11:47.152621495Z INFO solana_core::validator] Cleaning accounts paths done. clean_accounts_paths took 450us
[2025-03-19T16:11:47.152816462Z INFO solana_core::validator] Cleaning orphaned account snapshot directories..
[2025-03-19T16:11:47.152924751Z INFO solana_core::validator] Cleaning orphaned account snapshot directories done. clean_orphaned_account_snapshot_dirs took 105us
[2025-03-19T16:11:47.152952340Z INFO solana_core::validator] Geyser plugin: accounts_update_notifier: false, transaction_notifier: false, entry_notifier: false
[2025-03-19T16:11:47.152954670Z INFO solana_core::system_monitor_service] Starting SystemMonitorService
[2025-03-19T16:11:47.153010250Z INFO solana_core::validator] loading ledger from "/ledger"...
[2025-03-19T16:11:47.153035859Z INFO solana_ledger::blockstore] Maximum open file descriptors: 1048576
[2025-03-19T16:11:47.153037479Z INFO solana_ledger::blockstore] Opening blockstore at "/ledger/rocksdb"
[2025-03-19T16:11:47.311022886Z INFO solana_ledger::blockstore] Opening blockstore done; blockstore open took 157ms
[2025-03-19T16:11:47.311208863Z INFO solana_ledger::bank_forks_utils] Initializing bank snapshots dir: /snapshots/snapshots
[2025-03-19T16:11:47.311248713Z INFO solana_runtime::snapshot_bank_utils] Loading bank from full snapshot archive: /snapshots/remote/snapshot-368420648-56YZSbFqKVQ1fHuY7zNuX48XJgci3mgtagagc6A2b2S4.tar.zst, and incremental snapshot archive: Some("/snapshots/remote/incremental-snapshot-368420648-368429552-4rbgbLBCs6jVnv2cTswsNwVkSB2weLo1B7McgySP7iEL.tar.zst")
[SENTINEL.SH] [2025-03-19T16:11:48Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:50.209610411Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 5138/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:11:51Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:52.211042981Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 10469/197898 slots with 0 collisions
[2025-03-19T16:11:53.245621457Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1270i out_datagrams_delta=0i rcvbuf_errors_delta=1270i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=210932i rcvbuf_errors=210932i sndbuf_errors=0i rx_bytes_delta=3227632i rx_packets_delta=2879i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:11:53.245639487Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5036068864i cached_bytes=118191165440i free_percent=34.94202442664209 used_bytes=48217071616i avail_percent=82.15106342504465 buffers_percent=1.8642458102083381 cached_percent=43.75186101012787 swap_free_percent=0
[2025-03-19T16:11:54.155210942Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1371i out_datagrams_delta=8i rcvbuf_errors_delta=1371i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=212303i rcvbuf_errors=212303i sndbuf_errors=0i rx_bytes_delta=3301996i rx_packets_delta=2918i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=20985i tx_packets_delta=24i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:11:54.212520568Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 12071/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:11:54Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:56.213918074Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 15084/197898 slots with 0 collisions
[2025-03-19T16:11:56.655719426Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1668i out_datagrams_delta=0i rcvbuf_errors_delta=1668i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=213971i rcvbuf_errors=213971i sndbuf_errors=0i rx_bytes_delta=2204812i rx_packets_delta=1767i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:11:57.155856759Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5039693824i cached_bytes=124728832000i free_percent=32.03749764639742 used_bytes=49352355840i avail_percent=81.730805300078 buffers_percent=1.865587693465829 cached_percent=46.17196641816606 swap_free_percent=0
[2025-03-19T16:11:57.175101733Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=12.76 average_load_five_minutes=11.02 average_load_fifteen_minutes=10.18 total_num_threads=2317i
[2025-03-19T16:11:57.175474247Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=37127i reads_merged=358786i sectors_read=3175152i time_reading_ms=15143i writes_completed=106008i writes_merged=2090412i sectors_written=17571032i time_writing_ms=17542i io_in_progress=0i time_io_ms=18296i time_io_weighted_ms=32684i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:11:57Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:11:58.215323457Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 16334/197898 slots with 0 collisions
[2025-03-19T16:11:58.675855281Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1095i out_datagrams_delta=0i rcvbuf_errors_delta=1095i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=215066i rcvbuf_errors=215066i sndbuf_errors=0i rx_bytes_delta=1434459i rx_packets_delta=1140i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:00.216759848Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 20000/197898 slots with 0 collisions
[2025-03-19T16:12:00.676324178Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=885i out_datagrams_delta=0i rcvbuf_errors_delta=885i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=215951i rcvbuf_errors=215951i sndbuf_errors=0i rx_bytes_delta=1180283i rx_packets_delta=948i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:00Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:02.176639153Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5041958912i cached_bytes=130438795264i free_percent=29.41529703851975 used_bytes=50576748544i avail_percent=81.27756110701336 buffers_percent=1.8664261809702276 cached_percent=48.285673632824896 swap_free_percent=0
[2025-03-19T16:12:02.218242886Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 23244/197898 slots with 0 collisions
[2025-03-19T16:12:02.936520683Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=22751i reads_merged=378693i sectors_read=3243336i time_reading_ms=11631i writes_completed=90497i writes_merged=2320653i sectors_written=19287958i time_writing_ms=15244i io_in_progress=0i time_io_ms=18104i time_io_weighted_ms=26877i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:02.936534102Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=986i out_datagrams_delta=8i rcvbuf_errors_delta=986i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=216937i rcvbuf_errors=216937i sndbuf_errors=0i rx_bytes_delta=1288194i rx_packets_delta=1036i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6841i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:03Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:04.219655862Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 27280/197898 slots with 0 collisions
[2025-03-19T16:12:05.178008586Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1106i out_datagrams_delta=0i rcvbuf_errors_delta=1106i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=218043i rcvbuf_errors=218043i sndbuf_errors=0i rx_bytes_delta=1459601i rx_packets_delta=1166i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=202i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:06.221233053Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 29563/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:06Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:07.178492926Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=971i out_datagrams_delta=0i rcvbuf_errors_delta=971i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=219014i rcvbuf_errors=219014i sndbuf_errors=0i rx_bytes_delta=1278870i rx_packets_delta=1029i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:07.178560865Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5044224000i cached_bytes=130323226624i free_percent=29.384826432934965 used_bytes=50773622784i avail_percent=81.20468244173506 buffers_percent=1.8672646684746257 cached_percent=48.24289257507336 swap_free_percent=0
[2025-03-19T16:12:07.204598242Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=13.19 average_load_five_minutes=11.17 average_load_fifteen_minutes=10.24 total_num_threads=2315i
[2025-03-19T16:12:07.205092515Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=32818i reads_merged=373837i sectors_read=4364408i time_reading_ms=16243i writes_completed=97483i writes_merged=2239252i sectors_written=18694368i time_writing_ms=17001i io_in_progress=0i time_io_ms=19492i time_io_weighted_ms=33242i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:08.222622255Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 31675/197898 slots with 0 collisions
[2025-03-19T16:12:09.205567393Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1175i out_datagrams_delta=0i rcvbuf_errors_delta=1175i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=220189i rcvbuf_errors=220189i sndbuf_errors=0i rx_bytes_delta=1532771i rx_packets_delta=1227i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:09Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:10.224031013Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 34446/197898 slots with 0 collisions
[2025-03-19T16:12:11.706175916Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1587i out_datagrams_delta=0i rcvbuf_errors_delta=1587i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=221776i rcvbuf_errors=221776i sndbuf_errors=0i rx_bytes_delta=2094622i rx_packets_delta=1675i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:12.206330283Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5046194176i cached_bytes=130287185920i free_percent=29.623610388911732 used_bytes=50173485056i avail_percent=81.42684069158935 buffers_percent=1.8679939858157029 cached_percent=48.22955107136414 swap_free_percent=0
[2025-03-19T16:12:12.226195248Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 37731/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:12Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:12.949594265Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=28220i reads_merged=346384i sectors_read=4562264i time_reading_ms=14889i writes_completed=91280i writes_merged=2128984i sectors_written=17761584i time_writing_ms=14428i io_in_progress=0i time_io_ms=18428i time_io_weighted_ms=29319i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:13.707200323Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=881i out_datagrams_delta=8i rcvbuf_errors_delta=881i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=222657i rcvbuf_errors=222657i sndbuf_errors=0i rx_bytes_delta=1173731i rx_packets_delta=953i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7089i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:14.227719631Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 39744/197898 slots with 0 collisions
[2025-03-19T16:12:15.707640206Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=889i out_datagrams_delta=0i rcvbuf_errors_delta=889i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=223546i rcvbuf_errors=223546i sndbuf_errors=0i rx_bytes_delta=1166299i rx_packets_delta=935i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:15Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:16.229085112Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 40866/197898 slots with 0 collisions
[2025-03-19T16:12:17.207928634Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5048422400i cached_bytes=130260295680i free_percent=30.300877822120743 used_bytes=48401596416i avail_percent=82.08275626035544 buffers_percent=1.868818827049686 cached_percent=48.219596875222415 swap_free_percent=0
[2025-03-19T16:12:17.231113891Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=14.3 average_load_five_minutes=11.48 average_load_fifteen_minutes=10.35 total_num_threads=2318i
[2025-03-19T16:12:17.231491436Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=26958i reads_merged=322567i sectors_read=4329536i time_reading_ms=15174i writes_completed=85824i writes_merged=2166364i sectors_written=18017144i time_writing_ms=14962i io_in_progress=6i time_io_ms=18500i time_io_weighted_ms=30137i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:17.731807447Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=974i out_datagrams_delta=0i rcvbuf_errors_delta=974i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=224520i rcvbuf_errors=224520i sndbuf_errors=0i rx_bytes_delta=1281810i rx_packets_delta=1028i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:18.230460401Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 48261/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:18Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:19.732273714Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=940i out_datagrams_delta=0i rcvbuf_errors_delta=940i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=225460i rcvbuf_errors=225460i sndbuf_errors=0i rx_bytes_delta=1231940i rx_packets_delta=982i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:20.231884926Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 49331/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:21Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:22.232821026Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1164i out_datagrams_delta=0i rcvbuf_errors_delta=1164i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=226624i rcvbuf_errors=226624i sndbuf_errors=0i rx_bytes_delta=1525202i rx_packets_delta=1223i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=248i tx_packets_delta=4i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:22.233273740Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 54900/197898 slots with 0 collisions
[2025-03-19T16:12:22.983825411Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5051379712i cached_bytes=130280943616i free_percent=30.27178245734335 used_bytes=48474390528i avail_percent=82.0558094250423 buffers_percent=1.8699135611874356 cached_percent=48.22724030290718 swap_free_percent=0
[2025-03-19T16:12:22.983842391Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=26242i reads_merged=350925i sectors_read=4670024i time_reading_ms=14758i writes_completed=86341i writes_merged=2246719i sectors_written=18664172i time_writing_ms=13843i io_in_progress=0i time_io_ms=18876i time_io_weighted_ms=28600i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:24.234017949Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=917i out_datagrams_delta=8i rcvbuf_errors_delta=917i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=227541i rcvbuf_errors=227541i sndbuf_errors=0i rx_bytes_delta=1221098i rx_packets_delta=983i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6728i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:24.234667080Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 59731/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:24Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:26.234496850Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=944i out_datagrams_delta=0i rcvbuf_errors_delta=944i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=228485i rcvbuf_errors=228485i sndbuf_errors=0i rx_bytes_delta=1236156i rx_packets_delta=986i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:26.236052838Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 63757/197898 slots with 0 collisions
[2025-03-19T16:12:27.234709554Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5054214144i cached_bytes=130333188096i free_percent=30.290283767521775 used_bytes=48396763136i avail_percent=82.08454543803208 buffers_percent=1.8709628077571345 cached_percent=48.246580100589995 swap_free_percent=0
[2025-03-19T16:12:27.259124434Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3798i average_load_one_minute=15.29 average_load_five_minutes=11.79 average_load_fifteen_minutes=10.47 total_num_threads=2317i
[2025-03-19T16:12:27.259531208Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=30691i reads_merged=379338i sectors_read=5609392i time_reading_ms=16745i writes_completed=87707i writes_merged=2256009i sectors_written=18748928i time_writing_ms=14197i io_in_progress=0i time_io_ms=19296i time_io_weighted_ms=30942i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:12:27Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:28.237447073Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 66251/197898 slots with 0 collisions
[2025-03-19T16:12:28.259886082Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1042i out_datagrams_delta=0i rcvbuf_errors_delta=1042i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=229527i rcvbuf_errors=229527i sndbuf_errors=0i rx_bytes_delta=1356456i rx_packets_delta=1085i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:30.238903324Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 70130/197898 slots with 0 collisions
[2025-03-19T16:12:30.260372717Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=936i out_datagrams_delta=0i rcvbuf_errors_delta=936i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=230463i rcvbuf_errors=230463i sndbuf_errors=0i rx_bytes_delta=1238820i rx_packets_delta=998i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:30Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:32.240284544Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 72944/197898 slots with 0 collisions
[2025-03-19T16:12:32.260906028Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5056577536i cached_bytes=130329272320i free_percent=30.240326287784136 used_bytes=48540418048i avail_percent=82.03136743847648 buffers_percent=1.8718376853159733 cached_percent=48.24513056342145 swap_free_percent=0
[2025-03-19T16:12:33.031790893Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=34782i reads_merged=362005i sectors_read=5333976i time_reading_ms=19479i writes_completed=93615i writes_merged=2236533i sectors_written=18639992i time_writing_ms=18561i io_in_progress=3i time_io_ms=19448i time_io_weighted_ms=38040i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:33.031804422Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1153i out_datagrams_delta=8i rcvbuf_errors_delta=1153i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=231616i rcvbuf_errors=231616i sndbuf_errors=0i rx_bytes_delta=1522892i rx_packets_delta=1234i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6809i tx_packets_delta=18i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:33Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:34.241686330Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 74817/197898 slots with 0 collisions
[2025-03-19T16:12:35.262070180Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1171i out_datagrams_delta=0i rcvbuf_errors_delta=1171i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=232787i rcvbuf_errors=232787i sndbuf_errors=0i rx_bytes_delta=1528959i rx_packets_delta=1220i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:36.243083763Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 76202/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:36Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:37.262539280Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1014i out_datagrams_delta=0i rcvbuf_errors_delta=1014i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=233801i rcvbuf_errors=233801i sndbuf_errors=0i rx_bytes_delta=1328292i rx_packets_delta=1065i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:37.262588789Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5057695744i cached_bytes=130197200896i free_percent=30.289142029563703 used_bytes=48568123392i avail_percent=82.02111150813334 buffers_percent=1.872251622185233 cached_percent=48.19624052528073 swap_free_percent=0
[2025-03-19T16:12:37.287108209Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=15.77 average_load_five_minutes=12.01 average_load_fifteen_minutes=10.55 total_num_threads=2319i
[2025-03-19T16:12:37.287489093Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24657i reads_merged=360773i sectors_read=5039256i time_reading_ms=14242i writes_completed=82989i writes_merged=2303442i sectors_written=19090802i time_writing_ms=12555i io_in_progress=1i time_io_ms=18068i time_io_weighted_ms=26797i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:38.244483294Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 77487/197898 slots with 0 collisions
[2025-03-19T16:12:39.287975780Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=966i out_datagrams_delta=0i rcvbuf_errors_delta=966i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=234767i rcvbuf_errors=234767i sndbuf_errors=0i rx_bytes_delta=1270352i rx_packets_delta=1018i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:39Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:40.245906071Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 79123/197898 slots with 0 collisions
[2025-03-19T16:12:41.288497214Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1345i out_datagrams_delta=0i rcvbuf_errors_delta=1345i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=236112i rcvbuf_errors=236112i sndbuf_errors=0i rx_bytes_delta=1758328i rx_packets_delta=1409i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:42.247302836Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 80203/197898 slots with 0 collisions
[2025-03-19T16:12:42.288720534Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5058834432i cached_bytes=130263019520i free_percent=30.237945771722824 used_bytes=48648953856i avail_percent=81.9911897858697 buffers_percent=1.8726731403158348 cached_percent=48.22060518298087 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:12:42Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:43.050287525Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31580i reads_merged=389249i sectors_read=5225472i time_reading_ms=17586i writes_completed=88184i writes_merged=2227962i sectors_written=18529164i time_writing_ms=15144i io_in_progress=0i time_io_ms=18860i time_io_weighted_ms=32731i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:43.289522298Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1190i out_datagrams_delta=8i rcvbuf_errors_delta=1190i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=237302i rcvbuf_errors=237302i sndbuf_errors=0i rx_bytes_delta=1575255i rx_packets_delta=1269i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7089i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:44.248757317Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 83456/197898 slots with 0 collisions
[2025-03-19T16:12:45.790041346Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1111i out_datagrams_delta=0i rcvbuf_errors_delta=1111i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=238413i rcvbuf_errors=238413i sndbuf_errors=0i rx_bytes_delta=1460769i rx_packets_delta=1161i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:45Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:46.250208155Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 86187/197898 slots with 0 collisions
[2025-03-19T16:12:47.290318034Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5061222400i cached_bytes=130295566336i free_percent=30.35928992450671 used_bytes=48314433536i avail_percent=82.11502210862595 buffers_percent=1.8735571153882837 cached_percent=48.23265332350519 swap_free_percent=0
[2025-03-19T16:12:47.315827270Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=16.67 average_load_five_minutes=12.33 average_load_fifteen_minutes=10.67 total_num_threads=2320i
[2025-03-19T16:12:47.316269584Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24640i reads_merged=370292i sectors_read=5214064i time_reading_ms=14788i writes_completed=81794i writes_merged=2307882i sectors_written=19117454i time_writing_ms=12027i io_in_progress=0i time_io_ms=17852i time_io_weighted_ms=26812i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:47.816533501Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=988i out_datagrams_delta=0i rcvbuf_errors_delta=988i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=239401i rcvbuf_errors=239401i sndbuf_errors=0i rx_bytes_delta=1277447i rx_packets_delta=1023i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:48.251605381Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 89197/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:12:48Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:50.253098871Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 90405/197898 slots with 0 collisions
[2025-03-19T16:12:50.317018129Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1291i out_datagrams_delta=0i rcvbuf_errors_delta=1291i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=240692i rcvbuf_errors=240692i sndbuf_errors=0i rx_bytes_delta=1677786i rx_packets_delta=1348i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:51Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:52.254536231Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 92642/197898 slots with 0 collisions
[2025-03-19T16:12:52.317513702Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1516i out_datagrams_delta=0i rcvbuf_errors_delta=1516i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=242208i rcvbuf_errors=242208i sndbuf_errors=0i rx_bytes_delta=1982432i rx_packets_delta=1586i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:53.087899220Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5062586368i cached_bytes=130253070336i free_percent=30.356111859752243 used_bytes=48368713728i avail_percent=82.09492873356577 buffers_percent=1.8740620273936446 cached_percent=48.21692220622104 swap_free_percent=0
[2025-03-19T16:12:53.087912010Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=27514i reads_merged=357902i sectors_read=4805280i time_reading_ms=15266i writes_completed=83950i writes_merged=2273410i sectors_written=18857284i time_writing_ms=13025i io_in_progress=0i time_io_ms=17772i time_io_weighted_ms=28293i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:12:54.255923647Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 95192/197898 slots with 0 collisions
[2025-03-19T16:12:54.318872709Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=971i out_datagrams_delta=8i rcvbuf_errors_delta=971i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=243179i rcvbuf_errors=243179i sndbuf_errors=0i rx_bytes_delta=1291201i rx_packets_delta=1042i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6737i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:12:54Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:56.257530437Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 96741/197898 slots with 0 collisions
[2025-03-19T16:12:56.319318646Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=912i out_datagrams_delta=0i rcvbuf_errors_delta=912i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=244091i rcvbuf_errors=244091i sndbuf_errors=0i rx_bytes_delta=1191708i rx_packets_delta=956i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:12:57.319543582Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5064007680i cached_bytes=130394402816i free_percent=30.25340547985093 used_bytes=48509972480i avail_percent=82.04263774158713 buffers_percent=1.8745881669307625 cached_percent=48.26924049074051 swap_free_percent=0
[2025-03-19T16:12:57.343643308Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.7 average_load_five_minutes=12.68 average_load_fifteen_minutes=10.8 total_num_threads=2321i
[2025-03-19T16:12:57.344113181Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31060i reads_merged=374252i sectors_read=4873024i time_reading_ms=17382i writes_completed=89901i writes_merged=2237991i sectors_written=18623756i time_writing_ms=16253i io_in_progress=0i time_io_ms=18968i time_io_weighted_ms=33636i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:12:57Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:12:58.258889677Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 98302/197898 slots with 0 collisions
[2025-03-19T16:12:58.344482167Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=901i out_datagrams_delta=0i rcvbuf_errors_delta=901i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=244992i rcvbuf_errors=244992i sndbuf_errors=0i rx_bytes_delta=1174891i rx_packets_delta=947i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:00.260316334Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 100352/197898 slots with 0 collisions
[2025-03-19T16:13:00.845002609Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1214i out_datagrams_delta=0i rcvbuf_errors_delta=1214i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=246206i rcvbuf_errors=246206i sndbuf_errors=0i rx_bytes_delta=1602899i rx_packets_delta=1281i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:00Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:02.261844945Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 101610/197898 slots with 0 collisions
[2025-03-19T16:13:02.345295886Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5065326592i cached_bytes=130429128704i free_percent=30.220525549411736 used_bytes=48556953600i avail_percent=82.02524632806913 buffers_percent=1.8750764001611713 cached_percent=48.28209527747159 swap_free_percent=0
[2025-03-19T16:13:03.124856607Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=27754i reads_merged=385744i sectors_read=4905368i time_reading_ms=15761i writes_completed=85283i writes_merged=2243491i sectors_written=18629226i time_writing_ms=14555i io_in_progress=0i time_io_ms=18788i time_io_weighted_ms=30315i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:03.124869396Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1034i out_datagrams_delta=8i rcvbuf_errors_delta=1034i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=247240i rcvbuf_errors=247240i sndbuf_errors=0i rx_bytes_delta=1358455i rx_packets_delta=1096i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6742i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:04Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:04.263289545Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 103111/197898 slots with 0 collisions
[2025-03-19T16:13:04.846485556Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=909i out_datagrams_delta=0i rcvbuf_errors_delta=909i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=248149i rcvbuf_errors=248149i sndbuf_errors=0i rx_bytes_delta=1201599i rx_packets_delta=963i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:06.264703111Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 104613/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:07Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:07.347018775Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1182i out_datagrams_delta=0i rcvbuf_errors_delta=1182i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=249331i rcvbuf_errors=249331i sndbuf_errors=0i rx_bytes_delta=1572928i rx_packets_delta=1259i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:07.347069624Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5066526720i cached_bytes=130438819840i free_percent=30.025394496240803 used_bytes=49077780480i avail_percent=81.83244727088693 buffers_percent=1.8755206620757983 cached_percent=48.28568273033851 swap_free_percent=0
[2025-03-19T16:13:07.371434597Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=18.63 average_load_five_minutes=13.04 average_load_fifteen_minutes=10.94 total_num_threads=2326i
[2025-03-19T16:13:07.371833262Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=32199i reads_merged=364779i sectors_read=5140152i time_reading_ms=17204i writes_completed=90328i writes_merged=2265527i sectors_written=18846883i time_writing_ms=16302i io_in_progress=1i time_io_ms=18644i time_io_weighted_ms=33506i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:08.266125345Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 107125/197898 slots with 0 collisions
[2025-03-19T16:13:09.372256153Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=961i out_datagrams_delta=0i rcvbuf_errors_delta=961i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=250292i rcvbuf_errors=250292i sndbuf_errors=0i rx_bytes_delta=1270557i rx_packets_delta=1015i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:10Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:10.267540405Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 109166/197898 slots with 0 collisions
[2025-03-19T16:13:11.872824570Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1193i out_datagrams_delta=0i rcvbuf_errors_delta=1193i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=251485i rcvbuf_errors=251485i sndbuf_errors=0i rx_bytes_delta=1554786i rx_packets_delta=1250i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:12.268958972Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 110701/197898 slots with 0 collisions
[2025-03-19T16:13:12.372976711Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5068017664i cached_bytes=130439340032i free_percent=30.13814604741994 used_bytes=48772038656i avail_percent=81.94562640520574 buffers_percent=1.8760725779014782 cached_percent=48.28587529437659 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:13:13Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:13.126993348Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=28500i reads_merged=392384i sectors_read=5258112i time_reading_ms=16531i writes_completed=85599i writes_merged=2278981i sectors_written=18915683i time_writing_ms=14895i io_in_progress=0i time_io_ms=18680i time_io_weighted_ms=31427i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:13.873835678Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=924i out_datagrams_delta=8i rcvbuf_errors_delta=924i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=252409i rcvbuf_errors=252409i sndbuf_errors=0i rx_bytes_delta=1219477i rx_packets_delta=988i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7087i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:14.270566431Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 112991/197898 slots with 0 collisions
[2025-03-19T16:13:15.874311189Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=990i out_datagrams_delta=0i rcvbuf_errors_delta=990i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=253399i rcvbuf_errors=253399i sndbuf_errors=0i rx_bytes_delta=1316084i rx_packets_delta=1054i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:16Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:16.271983361Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 114694/197898 slots with 0 collisions
[2025-03-19T16:13:17.374614503Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5069762560i cached_bytes=130389082112i free_percent=30.16917463383948 used_bytes=48727302144i avail_percent=81.96218691248068 buffers_percent=1.876718501367796 cached_percent=48.26727087904391 swap_free_percent=0
[2025-03-19T16:13:17.399922274Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=18.14 average_load_five_minutes=13.12 average_load_fifteen_minutes=10.99 total_num_threads=2321i
[2025-03-19T16:13:17.400386097Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=33144i reads_merged=367290i sectors_read=5192488i time_reading_ms=18591i writes_completed=90416i writes_merged=2272689i sectors_written=18903540i time_writing_ms=17896i io_in_progress=2i time_io_ms=18796i time_io_weighted_ms=36486i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:17.900618220Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=924i out_datagrams_delta=0i rcvbuf_errors_delta=924i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=254323i rcvbuf_errors=254323i sndbuf_errors=0i rx_bytes_delta=1226881i rx_packets_delta=981i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:18.273381868Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 118972/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:19Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:20.274733462Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 123180/197898 slots with 0 collisions
[2025-03-19T16:13:20.401124964Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1276i out_datagrams_delta=0i rcvbuf_errors_delta=1276i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=255599i rcvbuf_errors=255599i sndbuf_errors=0i rx_bytes_delta=1673446i rx_packets_delta=1341i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:22Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:22.276179241Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 126908/197898 slots with 0 collisions
[2025-03-19T16:13:22.401579588Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=932i out_datagrams_delta=0i rcvbuf_errors_delta=932i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=256531i rcvbuf_errors=256531i sndbuf_errors=0i rx_bytes_delta=1220992i rx_packets_delta=973i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:23.167691365Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5070106624i cached_bytes=130392276992i free_percent=30.205599562082085 used_bytes=48627384320i avail_percent=81.99917437031485 buffers_percent=1.8768458665583374 cached_percent=48.26845355581323 swap_free_percent=0
[2025-03-19T16:13:23.167711125Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=29142i reads_merged=391591i sectors_read=5422096i time_reading_ms=17074i writes_completed=85840i writes_merged=2276674i sectors_written=18900440i time_writing_ms=15018i io_in_progress=0i time_io_ms=18580i time_io_weighted_ms=32092i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:24.277765474Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 129232/197898 slots with 0 collisions
[2025-03-19T16:13:24.402489551Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1022i out_datagrams_delta=8i rcvbuf_errors_delta=1022i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=257553i rcvbuf_errors=257553i sndbuf_errors=0i rx_bytes_delta=1354725i rx_packets_delta=1099i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6805i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:25Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:26.279169606Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 131646/197898 slots with 0 collisions
[2025-03-19T16:13:26.903074000Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1142i out_datagrams_delta=0i rcvbuf_errors_delta=1142i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=258695i rcvbuf_errors=258695i sndbuf_errors=0i rx_bytes_delta=1515502i rx_packets_delta=1217i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:27.403224552Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5071388672i cached_bytes=130370613248i free_percent=30.168860769619933 used_bytes=48742162432i avail_percent=81.9566859492511 buffers_percent=1.8773204535183317 cached_percent=48.26043409756592 swap_free_percent=0
[2025-03-19T16:13:27.423741571Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=18.36 average_load_five_minutes=13.33 average_load_fifteen_minutes=11.08 total_num_threads=2320i
[2025-03-19T16:13:27.424226514Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=26362i reads_merged=377420i sectors_read=5140632i time_reading_ms=15560i writes_completed=84440i writes_merged=2299464i sectors_written=19071236i time_writing_ms=12305i io_in_progress=4i time_io_ms=18264i time_io_weighted_ms=27864i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:13:28Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:28.280546276Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 134183/197898 slots with 0 collisions
[2025-03-19T16:13:28.924696913Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=995i out_datagrams_delta=0i rcvbuf_errors_delta=995i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=259690i rcvbuf_errors=259690i sndbuf_errors=0i rx_bytes_delta=1299178i rx_packets_delta=1039i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:30.281919651Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 136990/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:31Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:31.425301929Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1196i out_datagrams_delta=0i rcvbuf_errors_delta=1196i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=260886i rcvbuf_errors=260886i sndbuf_errors=0i rx_bytes_delta=1552836i rx_packets_delta=1243i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:32.283355523Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 139624/197898 slots with 0 collisions
[2025-03-19T16:13:32.425534264Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5073399808i cached_bytes=130378567680i free_percent=30.15292950703636 used_bytes=48757919744i avail_percent=81.95085292677476 buffers_percent=1.8780649333820918 cached_percent=48.26337865947106 swap_free_percent=0
[2025-03-19T16:13:33.195509229Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31995i reads_merged=365678i sectors_read=5322192i time_reading_ms=17151i writes_completed=88369i writes_merged=2242707i sectors_written=18648142i time_writing_ms=13698i io_in_progress=5i time_io_ms=18408i time_io_weighted_ms=30851i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:33.426260842Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1069i out_datagrams_delta=8i rcvbuf_errors_delta=1069i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=261955i rcvbuf_errors=261955i sndbuf_errors=0i rx_bytes_delta=1401839i rx_packets_delta=1128i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6881i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:34Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:34.284799851Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 141572/197898 slots with 0 collisions
[2025-03-19T16:13:35.926785528Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1735i out_datagrams_delta=0i rcvbuf_errors_delta=1735i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=263690i rcvbuf_errors=263690i sndbuf_errors=0i rx_bytes_delta=2259913i rx_packets_delta=1804i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:36.286259835Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 142362/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:37Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:37.427075770Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5074472960i cached_bytes=130366136320i free_percent=30.161967886808007 used_bytes=48743264256i avail_percent=81.9562780773909 buffers_percent=1.8784621914764 cached_percent=48.25877683383661 swap_free_percent=0
[2025-03-19T16:13:37.447130135Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.92 average_load_five_minutes=13.4 average_load_fifteen_minutes=11.13 total_num_threads=2319i
[2025-03-19T16:13:37.447562619Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=23803i reads_merged=357078i sectors_read=4772168i time_reading_ms=14312i writes_completed=80072i writes_merged=2220816i sectors_written=18406738i time_writing_ms=13038i io_in_progress=14i time_io_ms=17932i time_io_weighted_ms=27348i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:37.948392430Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1021i out_datagrams_delta=0i rcvbuf_errors_delta=1021i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=264711i rcvbuf_errors=264711i sndbuf_errors=0i rx_bytes_delta=1350002i rx_packets_delta=1076i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:38.287678635Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 144435/197898 slots with 0 collisions
[2025-03-19T16:13:39.948828239Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1024i out_datagrams_delta=0i rcvbuf_errors_delta=1024i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=265735i rcvbuf_errors=265735i sndbuf_errors=0i rx_bytes_delta=1346180i rx_packets_delta=1082i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=54i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:40Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:40.289150160Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 146348/197898 slots with 0 collisions
[2025-03-19T16:13:41.949315114Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=976i out_datagrams_delta=0i rcvbuf_errors_delta=976i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=266711i rcvbuf_errors=266711i sndbuf_errors=0i rx_bytes_delta=1278518i rx_packets_delta=1026i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:42.290545694Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 148496/197898 slots with 0 collisions
[2025-03-19T16:13:42.449465719Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5076037632i cached_bytes=130328735744i free_percent=30.139862444987713 used_bytes=48843870208i avail_percent=81.91903588917566 buffers_percent=1.8790413998429103 cached_percent=48.2449319343743 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:13:43Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:43.222354065Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31999i reads_merged=372479i sectors_read=5076744i time_reading_ms=17395i writes_completed=89423i writes_merged=2255134i sectors_written=18755956i time_writing_ms=15555i io_in_progress=0i time_io_ms=18300i time_io_weighted_ms=32950i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:43.950467296Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1033i out_datagrams_delta=8i rcvbuf_errors_delta=1033i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=267744i rcvbuf_errors=267744i sndbuf_errors=0i rx_bytes_delta=1367907i rx_packets_delta=1107i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7163i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:44.291923134Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 152032/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:46Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:46.293347409Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 153972/197898 slots with 0 collisions
[2025-03-19T16:13:46.450983144Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1388i out_datagrams_delta=0i rcvbuf_errors_delta=1388i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=269132i rcvbuf_errors=269132i sndbuf_errors=0i rx_bytes_delta=1849248i rx_packets_delta=1488i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:47.451177391Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5077487616i cached_bytes=130388336640i free_percent=30.087800407101607 used_bytes=48917639168i avail_percent=81.89172818582266 buffers_percent=1.8795781531459066 cached_percent=48.26699492113108 swap_free_percent=0
[2025-03-19T16:13:47.475109752Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.11 average_load_five_minutes=13.37 average_load_fifteen_minutes=11.14 total_num_threads=2320i
[2025-03-19T16:13:47.475490687Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24973i reads_merged=363108i sectors_read=5100544i time_reading_ms=14950i writes_completed=83013i writes_merged=2300752i sectors_written=19069784i time_writing_ms=12557i io_in_progress=11i time_io_ms=18096i time_io_weighted_ms=27509i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:48.294742681Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 157019/197898 slots with 0 collisions
[2025-03-19T16:13:48.475826713Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=929i out_datagrams_delta=0i rcvbuf_errors_delta=929i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=270061i rcvbuf_errors=270061i sndbuf_errors=0i rx_bytes_delta=1242427i rx_packets_delta=991i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:49Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:50.296208588Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 158295/197898 slots with 0 collisions
[2025-03-19T16:13:50.476285785Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=441i out_datagrams_delta=0i rcvbuf_errors_delta=441i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=270502i rcvbuf_errors=270502i sndbuf_errors=0i rx_bytes_delta=635305i rx_packets_delta=545i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:13:52Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:52.297643461Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 160015/197898 slots with 0 collisions
[2025-03-19T16:13:52.476632605Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5079085056i cached_bytes=130279104512i free_percent=30.013936177848873 used_bytes=49226153984i avail_percent=81.77752254871817 buffers_percent=1.8801694915305633 cached_percent=48.22655950563868 swap_free_percent=0
[2025-03-19T16:13:53.258649023Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=29185i reads_merged=363989i sectors_read=5005200i time_reading_ms=16027i writes_completed=85279i writes_merged=2217924i sectors_written=18424332i time_writing_ms=13998i io_in_progress=3i time_io_ms=18048i time_io_weighted_ms=30024i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:13:53.258663193Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1134i out_datagrams_delta=8i rcvbuf_errors_delta=1134i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=271636i rcvbuf_errors=271636i sndbuf_errors=0i rx_bytes_delta=1814267i rx_packets_delta=1463i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6395i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:54.299114690Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 162028/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:13:55Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:55.477792261Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1264i out_datagrams_delta=0i rcvbuf_errors_delta=1264i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=272900i rcvbuf_errors=272900i sndbuf_errors=0i rx_bytes_delta=1698068i rx_packets_delta=1358i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:56.300537176Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 163909/197898 slots with 0 collisions
[2025-03-19T16:13:57.478280914Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=906i out_datagrams_delta=0i rcvbuf_errors_delta=906i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=273806i rcvbuf_errors=273806i sndbuf_errors=0i rx_bytes_delta=1291295i rx_packets_delta=1031i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:13:57.478343243Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5080371200i cached_bytes=130318106624i free_percent=29.72926284697045 used_bytes=49991868416i avail_percent=81.49407131716804 buffers_percent=1.8806455947428258 cached_percent=48.24099725973793 swap_free_percent=0
[2025-03-19T16:13:57.503110492Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.76 average_load_five_minutes=13.63 average_load_fifteen_minutes=11.25 total_num_threads=2301i
[2025-03-19T16:13:57.503502707Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=29843i reads_merged=384392i sectors_read=5265800i time_reading_ms=16999i writes_completed=87818i writes_merged=2268423i sectors_written=18858312i time_writing_ms=14080i io_in_progress=0i time_io_ms=18768i time_io_weighted_ms=31080i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:13:58Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:13:58.302195725Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 164713/197898 slots with 0 collisions
[2025-03-19T16:13:59.503952666Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1031i out_datagrams_delta=0i rcvbuf_errors_delta=1031i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=274837i rcvbuf_errors=274837i sndbuf_errors=0i rx_bytes_delta=1644599i rx_packets_delta=1319i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:00.303636753Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 165349/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:01Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:01.504450031Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=893i out_datagrams_delta=0i rcvbuf_errors_delta=893i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=275730i rcvbuf_errors=275730i sndbuf_errors=0i rx_bytes_delta=1309166i rx_packets_delta=1059i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:02.305077326Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 165982/197898 slots with 0 collisions
[2025-03-19T16:14:02.504703561Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5080731648i cached_bytes=130388275200i free_percent=29.501947823151493 used_bytes=50537533440i avail_percent=81.29207770623054 buffers_percent=1.880779024942441 cached_percent=48.26697217734705 swap_free_percent=0
[2025-03-19T16:14:03.273784264Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=28491i reads_merged=389254i sectors_read=5189552i time_reading_ms=16136i writes_completed=86119i writes_merged=2287644i sectors_written=18989620i time_writing_ms=14652i io_in_progress=4i time_io_ms=18324i time_io_weighted_ms=30787i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:03.505664261Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=945i out_datagrams_delta=8i rcvbuf_errors_delta=945i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=276675i rcvbuf_errors=276675i sndbuf_errors=0i rx_bytes_delta=1236192i rx_packets_delta=1001i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7154i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:04Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:04.306492957Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 166573/197898 slots with 0 collisions
[2025-03-19T16:14:06.006158342Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1268i out_datagrams_delta=0i rcvbuf_errors_delta=1268i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=277943i rcvbuf_errors=277943i sndbuf_errors=0i rx_bytes_delta=1665199i rx_packets_delta=1329i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:06.307909753Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 167194/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:07Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:07.506451194Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5081223168i cached_bytes=130318053376i free_percent=29.363636807484756 used_bytes=50951696384i avail_percent=81.13876337436847 buffers_percent=1.880960975214643 cached_percent=48.24097754845844 swap_free_percent=0
[2025-03-19T16:14:07.527069552Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.82 average_load_five_minutes=13.77 average_load_fifteen_minutes=11.32 total_num_threads=2304i
[2025-03-19T16:14:07.527429247Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=32596i reads_merged=367986i sectors_read=5068352i time_reading_ms=18498i writes_completed=90386i writes_merged=2226633i sectors_written=18535934i time_writing_ms=17653i io_in_progress=0i time_io_ms=18680i time_io_weighted_ms=36151i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:08.027687629Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=977i out_datagrams_delta=0i rcvbuf_errors_delta=977i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=278920i rcvbuf_errors=278920i sndbuf_errors=0i rx_bytes_delta=1292224i rx_packets_delta=1035i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:08.309319325Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 167881/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:10Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:10.310678373Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 168427/197898 slots with 0 collisions
[2025-03-19T16:14:10.528233756Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1170i out_datagrams_delta=0i rcvbuf_errors_delta=1170i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=280090i rcvbuf_errors=280090i sndbuf_errors=0i rx_bytes_delta=1546955i rx_packets_delta=1240i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:12.312030228Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 169035/197898 slots with 0 collisions
[2025-03-19T16:14:12.528568316Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5081370624i cached_bytes=130324480000i free_percent=29.37898128044047 used_bytes=50899791872i avail_percent=81.15797732311303 buffers_percent=1.8810155602963037 cached_percent=48.24335654826748 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:14:13Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:13.311906399Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=25775i reads_merged=367395i sectors_read=4944648i time_reading_ms=14609i writes_completed=80706i writes_merged=2182634i sectors_written=18105906i time_writing_ms=12648i io_in_progress=4i time_io_ms=17940i time_io_weighted_ms=27256i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:13.311919728Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1153i out_datagrams_delta=8i rcvbuf_errors_delta=1153i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=281243i rcvbuf_errors=281243i sndbuf_errors=0i rx_bytes_delta=1511488i rx_packets_delta=1223i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6395i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:14.313531167Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 169642/197898 slots with 0 collisions
[2025-03-19T16:14:15.529731582Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1188i out_datagrams_delta=0i rcvbuf_errors_delta=1188i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=282431i rcvbuf_errors=282431i sndbuf_errors=0i rx_bytes_delta=1569869i rx_packets_delta=1261i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:16Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:16.314958132Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 170217/197898 slots with 0 collisions
[2025-03-19T16:14:17.530059587Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5081800704i cached_bytes=130359201792i free_percent=29.205911697774045 used_bytes=51331100672i avail_percent=80.99831595925563 buffers_percent=1.8811747667844805 cached_percent=48.25620981874629 swap_free_percent=0
[2025-03-19T16:14:17.555702994Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=17.78 average_load_five_minutes=13.9 average_load_fifteen_minutes=11.39 total_num_threads=2301i
[2025-03-19T16:14:17.556136088Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=30014i reads_merged=351700i sectors_read=4802152i time_reading_ms=17065i writes_completed=88003i writes_merged=2191875i sectors_written=18238732i time_writing_ms=17873i io_in_progress=0i time_io_ms=18164i time_io_weighted_ms=34939i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:18.056392218Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1145i out_datagrams_delta=0i rcvbuf_errors_delta=1145i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=283576i rcvbuf_errors=283576i sndbuf_errors=0i rx_bytes_delta=1503178i rx_packets_delta=1199i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:18.316379133Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 170819/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:19Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:20.317856040Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 171426/197898 slots with 0 collisions
[2025-03-19T16:14:20.556905952Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1200i out_datagrams_delta=0i rcvbuf_errors_delta=1200i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=284776i rcvbuf_errors=284776i sndbuf_errors=0i rx_bytes_delta=1560148i rx_packets_delta=1249i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:22Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:22.319295082Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 171995/197898 slots with 0 collisions
[2025-03-19T16:14:22.557400029Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=938i out_datagrams_delta=0i rcvbuf_errors_delta=938i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=285714i rcvbuf_errors=285714i sndbuf_errors=0i rx_bytes_delta=1238985i rx_packets_delta=996i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:23.341776778Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5082292224i cached_bytes=130406223872i free_percent=29.253710034281553 used_bytes=51146846208i avail_percent=81.06652305129515 buffers_percent=1.8813567170566827 cached_percent=48.27361639478697 swap_free_percent=0
[2025-03-19T16:14:23.341788798Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=25039i reads_merged=362860i sectors_read=4926032i time_reading_ms=15497i writes_completed=81091i writes_merged=2220821i sectors_written=18415704i time_writing_ms=12425i io_in_progress=0i time_io_ms=17512i time_io_weighted_ms=27924i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:24.320778660Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 172556/197898 slots with 0 collisions
[2025-03-19T16:14:24.558298335Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=895i out_datagrams_delta=8i rcvbuf_errors_delta=895i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=286609i rcvbuf_errors=286609i sndbuf_errors=0i rx_bytes_delta=1173995i rx_packets_delta=954i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6804i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:25Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:26.322199015Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 173146/197898 slots with 0 collisions
[2025-03-19T16:14:27.059042220Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1257i out_datagrams_delta=0i rcvbuf_errors_delta=1257i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=287866i rcvbuf_errors=287866i sndbuf_errors=0i rx_bytes_delta=1650271i rx_packets_delta=1316i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:27.559185467Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5082742784i cached_bytes=130332704768i free_percent=29.147213023709362 used_bytes=51512086528i avail_percent=80.93131883652627 buffers_percent=1.8815235048062013 cached_percent=48.24640118282233 swap_free_percent=0
[2025-03-19T16:14:27.579178645Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.41 average_load_five_minutes=14 average_load_fifteen_minutes=11.46 total_num_threads=2304i
[2025-03-19T16:14:27.579551180Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=21479i reads_merged=341242i sectors_read=4655296i time_reading_ms=12994i writes_completed=76851i writes_merged=2193037i sectors_written=18158622i time_writing_ms=11279i io_in_progress=0i time_io_ms=17156i time_io_weighted_ms=24269i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:14:28Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:28.323640414Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 173799/197898 slots with 0 collisions
[2025-03-19T16:14:29.079985356Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=854i out_datagrams_delta=0i rcvbuf_errors_delta=854i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=288720i rcvbuf_errors=288720i sndbuf_errors=0i rx_bytes_delta=1134131i rx_packets_delta=913i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:30.325045540Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 174432/197898 slots with 0 collisions
[2025-03-19T16:14:31.080435477Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1011i out_datagrams_delta=0i rcvbuf_errors_delta=1011i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=289731i rcvbuf_errors=289731i sndbuf_errors=0i rx_bytes_delta=1316961i rx_packets_delta=1052i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:31Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:32.326467392Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 175101/197898 slots with 0 collisions
[2025-03-19T16:14:32.580719173Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5083283456i cached_bytes=130404556800i free_percent=29.44916708169018 used_bytes=50617200640i avail_percent=81.26258659961113 buffers_percent=1.8817236501056236 cached_percent=48.27299928011375 swap_free_percent=0
[2025-03-19T16:14:33.345449952Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31761i reads_merged=365035i sectors_read=5217113i time_reading_ms=17416i writes_completed=89271i writes_merged=2269879i sectors_written=18872302i time_writing_ms=14648i io_in_progress=1i time_io_ms=18060i time_io_weighted_ms=32065i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:33.345464172Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=973i out_datagrams_delta=8i rcvbuf_errors_delta=973i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=290704i rcvbuf_errors=290704i sndbuf_errors=0i rx_bytes_delta=1297588i rx_packets_delta=1049i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6743i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:34Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:34.327857509Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 175753/197898 slots with 0 collisions
[2025-03-19T16:14:35.582493977Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1233i out_datagrams_delta=0i rcvbuf_errors_delta=1233i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=291937i rcvbuf_errors=291937i sndbuf_errors=0i rx_bytes_delta=1603909i rx_packets_delta=1287i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:36.329278982Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 176327/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:37Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:37.582849618Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5083455488i cached_bytes=130389872640i free_percent=29.90872798170478 used_bytes=49389486080i avail_percent=81.71706047326539 buffers_percent=1.8817873327008943 cached_percent=48.26756351573171 swap_free_percent=0
[2025-03-19T16:14:37.607522270Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=16.65 average_load_five_minutes=13.95 average_load_fifteen_minutes=11.47 total_num_threads=2300i
[2025-03-19T16:14:37.607972534Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=25853i reads_merged=378070i sectors_read=5121536i time_reading_ms=15380i writes_completed=83165i writes_merged=2274067i sectors_written=18857854i time_writing_ms=12417i io_in_progress=0i time_io_ms=18124i time_io_weighted_ms=27798i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:38.108243078Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1188i out_datagrams_delta=0i rcvbuf_errors_delta=1188i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=293125i rcvbuf_errors=293125i sndbuf_errors=0i rx_bytes_delta=1581839i rx_packets_delta=1264i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:38.330702200Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 176992/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:40Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:40.332105214Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 177583/197898 slots with 0 collisions
[2025-03-19T16:14:40.608808972Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1130i out_datagrams_delta=0i rcvbuf_errors_delta=1130i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=294255i rcvbuf_errors=294255i sndbuf_errors=0i rx_bytes_delta=1481603i rx_packets_delta=1190i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:42.333525723Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 178178/197898 slots with 0 collisions
[2025-03-19T16:14:42.609261606Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=948i out_datagrams_delta=0i rcvbuf_errors_delta=948i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=295203i rcvbuf_errors=295203i sndbuf_errors=0i rx_bytes_delta=1236820i rx_packets_delta=990i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:43Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:43.386670596Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5083971584i cached_bytes=130432352256i free_percent=29.810132161702967 used_bytes=49603932160i avail_percent=81.63767708575587 buffers_percent=1.8819783804867065 cached_percent=48.28328856800678 swap_free_percent=0
[2025-03-19T16:14:43.386687316Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=32231i reads_merged=364251i sectors_read=4961032i time_reading_ms=18126i writes_completed=89771i writes_merged=2217078i sectors_written=18453166i time_writing_ms=18668i io_in_progress=2i time_io_ms=18832i time_io_weighted_ms=36794i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:44.334955188Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 178829/197898 slots with 0 collisions
[2025-03-19T16:14:45.110225311Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1141i out_datagrams_delta=8i rcvbuf_errors_delta=1141i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=296344i rcvbuf_errors=296344i sndbuf_errors=0i rx_bytes_delta=1513190i rx_packets_delta=1229i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6736i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:46Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:46.336388699Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 179487/197898 slots with 0 collisions
[2025-03-19T16:14:47.610816209Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1118i out_datagrams_delta=0i rcvbuf_errors_delta=1118i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=297462i rcvbuf_errors=297462i sndbuf_errors=0i rx_bytes_delta=1457213i rx_packets_delta=1169i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:47.610889658Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5084274688i cached_bytes=130386448384i free_percent=29.73064263653465 used_bytes=49858777088i avail_percent=81.54333890212358 buffers_percent=1.8820905831545647 cached_percent=48.26629592883537 swap_free_percent=0
[2025-03-19T16:14:47.635109916Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=16.98 average_load_five_minutes=14.12 average_load_fifteen_minutes=11.56 total_num_threads=2323i
[2025-03-19T16:14:47.635482351Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24838i reads_merged=364587i sectors_read=4967792i time_reading_ms=14505i writes_completed=82125i writes_merged=2271262i sectors_written=18826926i time_writing_ms=12243i io_in_progress=0i time_io_ms=17924i time_io_weighted_ms=26748i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:48.337852744Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 180152/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:49Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:49.635978044Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1161i out_datagrams_delta=0i rcvbuf_errors_delta=1161i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=298623i rcvbuf_errors=298623i sndbuf_errors=0i rx_bytes_delta=1535583i rx_packets_delta=1235i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:50.339293825Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 180802/197898 slots with 0 collisions
[2025-03-19T16:14:51.636593770Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1066i out_datagrams_delta=0i rcvbuf_errors_delta=1066i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=299689i rcvbuf_errors=299689i sndbuf_errors=0i rx_bytes_delta=1400763i rx_packets_delta=1121i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:52Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:52.340745762Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 181521/197898 slots with 0 collisions
[2025-03-19T16:14:52.636924760Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5084856320i cached_bytes=130405023744i free_percent=29.603362356120165 used_bytes=50148646912i avail_percent=81.43603524534463 buffers_percent=1.8823058909766706 cached_percent=48.27317213287234 swap_free_percent=0
[2025-03-19T16:14:53.396740969Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=29175i reads_merged=362716i sectors_read=5007944i time_reading_ms=16054i writes_completed=87119i writes_merged=2240869i sectors_written=18624411i time_writing_ms=15979i io_in_progress=1i time_io_ms=17972i time_io_weighted_ms=32034i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:53.637628646Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1377i out_datagrams_delta=8i rcvbuf_errors_delta=1377i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=301066i rcvbuf_errors=301066i sndbuf_errors=0i rx_bytes_delta=1799378i rx_packets_delta=1448i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7003i tx_packets_delta=21i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:54.342117575Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 182378/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:14:55Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:56.138082765Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1158i out_datagrams_delta=0i rcvbuf_errors_delta=1158i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=302224i rcvbuf_errors=302224i sndbuf_errors=0i rx_bytes_delta=1531351i rx_packets_delta=1226i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:14:56.343526393Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 183195/197898 slots with 0 collisions
[2025-03-19T16:14:57.638384742Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5085392896i cached_bytes=130427379712i free_percent=29.59094576629464 used_bytes=50159767552i avail_percent=81.43191862043605 buffers_percent=1.8825045200238244 cached_percent=48.281447837753 swap_free_percent=0
[2025-03-19T16:14:57.659091251Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=16.96 average_load_five_minutes=14.21 average_load_fifteen_minutes=11.61 total_num_threads=2322i
[2025-03-19T16:14:57.662094459Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=27112i reads_merged=362726i sectors_read=5072640i time_reading_ms=14737i writes_completed=83541i writes_merged=2233261i sectors_written=18533949i time_writing_ms=12483i io_in_progress=3i time_io_ms=17804i time_io_weighted_ms=27217i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:14:58.162364826Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=962i out_datagrams_delta=0i rcvbuf_errors_delta=962i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=303186i rcvbuf_errors=303186i sndbuf_errors=0i rx_bytes_delta=1256723i rx_packets_delta=1004i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:14:58Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:14:58.344951766Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 183853/197898 slots with 0 collisions
[2025-03-19T16:15:00.346327355Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 184539/197898 slots with 0 collisions
[2025-03-19T16:15:00.662862300Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1246i out_datagrams_delta=0i rcvbuf_errors_delta=1246i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=304432i rcvbuf_errors=304432i sndbuf_errors=0i rx_bytes_delta=1634084i rx_packets_delta=1313i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:15:01Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:02.347739249Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 185131/197898 slots with 0 collisions
[2025-03-19T16:15:02.663291278Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=956i out_datagrams_delta=0i rcvbuf_errors_delta=956i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=305388i rcvbuf_errors=305388i sndbuf_errors=0i rx_bytes_delta=1271925i rx_packets_delta=1016i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:03.426640087Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5085896704i cached_bytes=130453532672i free_percent=29.722286570283767 used_bytes=49775144960i avail_percent=81.57429774093877 buffers_percent=1.8826910190528316 cached_percent=48.29112910848642 swap_free_percent=0
[2025-03-19T16:15:03.426652157Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=28985i reads_merged=377765i sectors_read=5107416i time_reading_ms=16203i writes_completed=84994i writes_merged=2248185i sectors_written=18664694i time_writing_ms=15325i io_in_progress=1i time_io_ms=18504i time_io_weighted_ms=31530i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[SENTINEL.SH] [2025-03-19T16:15:04Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:04.349165248Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 185647/197898 slots with 0 collisions
[2025-03-19T16:15:05.164257830Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1139i out_datagrams_delta=8i rcvbuf_errors_delta=1139i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=306527i rcvbuf_errors=306527i sndbuf_errors=0i rx_bytes_delta=1489590i rx_packets_delta=1211i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6734i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:06.350633362Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 186242/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:07Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:07.664807087Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1242i out_datagrams_delta=0i rcvbuf_errors_delta=1242i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=307769i rcvbuf_errors=307769i sndbuf_errors=0i rx_bytes_delta=1653513i rx_packets_delta=1323i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:07.664858587Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5086326784i cached_bytes=130479480832i free_percent=29.733878318875313 used_bytes=49716215808i avail_percent=81.59611206232354 buffers_percent=1.8828502255410087 cached_percent=48.30073456660643 swap_free_percent=0
[2025-03-19T16:15:07.687311791Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.95 average_load_five_minutes=14.51 average_load_fifteen_minutes=11.74 total_num_threads=2318i
[2025-03-19T16:15:07.687673505Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=32121i reads_merged=389158i sectors_read=5187592i time_reading_ms=18235i writes_completed=89636i writes_merged=2254392i sectors_written=18751536i time_writing_ms=16720i io_in_progress=0i time_io_ms=18440i time_io_weighted_ms=34954i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:08.352119200Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 186971/197898 slots with 0 collisions
[2025-03-19T16:15:09.688183491Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1012i out_datagrams_delta=0i rcvbuf_errors_delta=1012i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=308781i rcvbuf_errors=308781i sndbuf_errors=0i rx_bytes_delta=1325070i rx_packets_delta=1060i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:10.353631834Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 187633/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:10Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:11.688650773Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1092i out_datagrams_delta=0i rcvbuf_errors_delta=1092i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=309873i rcvbuf_errors=309873i sndbuf_errors=0i rx_bytes_delta=1437288i rx_packets_delta=1152i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:12.355043095Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 188267/197898 slots with 0 collisions
[2025-03-19T16:15:12.688856942Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5086846976i cached_bytes=130423054336i free_percent=29.738081370163183 used_bytes=49758314496i avail_percent=81.58052802150944 buffers_percent=1.8830427895790893 cached_percent=48.279846675357625 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:15:13Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:13.467589764Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24328i reads_merged=375352i sectors_read=4957400i time_reading_ms=14413i writes_completed=80044i writes_merged=2240084i sectors_written=18560066i time_writing_ms=12141i io_in_progress=0i time_io_ms=17340i time_io_weighted_ms=26553i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:13.689567134Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1024i out_datagrams_delta=8i rcvbuf_errors_delta=1024i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=310897i rcvbuf_errors=310897i sndbuf_errors=0i rx_bytes_delta=1353038i rx_packets_delta=1094i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6885i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:14.356472250Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 188902/197898 slots with 0 collisions
[2025-03-19T16:15:16.190100669Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1275i out_datagrams_delta=0i rcvbuf_errors_delta=1275i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=312172i rcvbuf_errors=312172i sndbuf_errors=0i rx_bytes_delta=1670272i rx_packets_delta=1336i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:16.358607821Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 189702/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:16Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:17.690377306Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5087367168i cached_bytes=130617487360i free_percent=29.621053987587292 used_bytes=49871167488i avail_percent=81.53875223901181 buffers_percent=1.88323535361717 cached_percent=48.35182165428399 swap_free_percent=0
[2025-03-19T16:15:17.711157224Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=18.14 average_load_five_minutes=14.66 average_load_fifteen_minutes=11.82 total_num_threads=2321i
[2025-03-19T16:15:17.711532238Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=31572i reads_merged=374178i sectors_read=5181520i time_reading_ms=17312i writes_completed=89892i writes_merged=2311080i sectors_written=19207392i time_writing_ms=14076i io_in_progress=8i time_io_ms=17936i time_io_weighted_ms=31390i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:18.211838578Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1114i out_datagrams_delta=0i rcvbuf_errors_delta=1114i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=313286i rcvbuf_errors=313286i sndbuf_errors=0i rx_bytes_delta=1450355i rx_packets_delta=1164i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:18.360116565Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 190352/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:19Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:20.212269733Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1056i out_datagrams_delta=0i rcvbuf_errors_delta=1056i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=314342i rcvbuf_errors=314342i sndbuf_errors=0i rx_bytes_delta=1376673i rx_packets_delta=1102i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:20.361619975Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 191016/197898 slots with 0 collisions
[2025-03-19T16:15:22.363035581Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 192025/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:22Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:22.712785598Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1218i out_datagrams_delta=0i rcvbuf_errors_delta=1218i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=315560i rcvbuf_errors=315560i sndbuf_errors=0i rx_bytes_delta=1600968i rx_packets_delta=1279i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:23.492214162Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5088116736i cached_bytes=130397929472i free_percent=29.458865031198556 used_bytes=50521468928i avail_percent=81.29802444762703 buffers_percent=1.8835128277822781 cached_percent=48.270545983943556 swap_free_percent=0
[2025-03-19T16:15:23.492227252Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=30266i reads_merged=402714i sectors_read=5356496i time_reading_ms=16399i writes_completed=90040i writes_merged=2363754i sectors_written=19629186i time_writing_ms=15653i io_in_progress=4i time_io_ms=18680i time_io_weighted_ms=32051i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:24.364511133Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 194697/197898 slots with 0 collisions
[2025-03-19T16:15:24.713722729Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=944i out_datagrams_delta=8i rcvbuf_errors_delta=944i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=316504i rcvbuf_errors=316504i sndbuf_errors=0i rx_bytes_delta=1240319i rx_packets_delta=999i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6738i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:15:25Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:26.366247374Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 196511/197898 slots with 0 collisions
[2025-03-19T16:15:27.214237299Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1191i out_datagrams_delta=0i rcvbuf_errors_delta=1191i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=317695i rcvbuf_errors=317695i sndbuf_errors=0i rx_bytes_delta=1558035i rx_packets_delta=1244i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:27.714397097Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5089406976i cached_bytes=130450038784i free_percent=29.455189635700073 used_bytes=50479239168i avail_percent=81.31365700851373 buffers_percent=1.883990447246809 cached_percent=48.28983574530152 swap_free_percent=0
[2025-03-19T16:15:27.735163815Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3798i average_load_one_minute=17.66 average_load_five_minutes=14.67 average_load_fifteen_minutes=11.85 total_num_threads=2322i
[2025-03-19T16:15:27.735550900Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=24838i reads_merged=373497i sectors_read=4763584i time_reading_ms=14567i writes_completed=86456i writes_merged=2355704i sectors_written=19537460i time_writing_ms=15706i io_in_progress=3i time_io_ms=17780i time_io_weighted_ms=30274i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:27.987616591Z INFO solana_accounts_db::shared_buffer_reader] reading entire decompressed file took: 220676094 us, bytes: 289757560320, read_us: 157420496, waiting_for_buffer_us: 63063792, largest fetch: 29229056, error: Ok(0)
[2025-03-19T16:15:28.020305052Z INFO solana_accounts_db::hardened_unpack] unpacked 49476 entries total
[2025-03-19T16:15:28.278817803Z INFO solana_accounts_db::hardened_unpack] unpacked 49476 entries total
[2025-03-19T16:15:28.291197469Z INFO solana_accounts_db::hardened_unpack] unpacked 49475 entries total
[2025-03-19T16:15:28.367718405Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 197699/197898 slots with 0 collisions
[SENTINEL.SH] [2025-03-19T16:15:28Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:29.235941677Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=871i out_datagrams_delta=0i rcvbuf_errors_delta=871i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=318566i rcvbuf_errors=318566i sndbuf_errors=0i rx_bytes_delta=1140229i rx_packets_delta=917i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:29.268189815Z INFO solana_accounts_db::hardened_unpack] unpacked 49476 entries total
[2025-03-19T16:15:29.295690809Z INFO solana_runtime::snapshot_utils] snapshot untar took 222.0s
[2025-03-19T16:15:29.459086835Z INFO solana_accounts_db::shared_buffer_reader] reading entire decompressed file took: 163069 us, bytes: 228357632, read_us: 162881, waiting_for_buffer_us: 0, largest fetch: 3801088, error: Ok(0)
[2025-03-19T16:15:29.513752258Z INFO solana_accounts_db::hardened_unpack] unpacked 902 entries total
[2025-03-19T16:15:29.519314909Z INFO solana_accounts_db::hardened_unpack] unpacked 901 entries total
[2025-03-19T16:15:29.519971570Z INFO solana_accounts_db::hardened_unpack] unpacked 901 entries total
[2025-03-19T16:15:29.578709226Z INFO solana_accounts_db::hardened_unpack] unpacked 901 entries total
[2025-03-19T16:15:30.209212536Z INFO solana_runtime::snapshot_utils] incremental snapshot untar took 890ms
[2025-03-19T16:15:30.209656759Z INFO solana_runtime::snapshot_utils] snapshot version: 1.2.0
[2025-03-19T16:15:30.209873187Z INFO solana_runtime::snapshot_utils] snapshot version: 1.2.0
[2025-03-19T16:15:30.209924526Z INFO solana_runtime::snapshot_bank_utils] Rebuilding bank from full snapshot /snapshots/snapshots/tmp-snapshot-archive-qmBZEb/snapshots/368420648/368420648 and incremental snapshot Some("/snapshots/snapshots/tmp-snapshot-archive-oYufwS/snapshots/368429552/368429552")
[2025-03-19T16:15:31.129437640Z INFO solana_accounts_db::read_only_accounts_cache] AccountsReadCacheEvictor has started
[2025-03-19T16:15:31.236460499Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=954i out_datagrams_delta=0i rcvbuf_errors_delta=954i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=319520i rcvbuf_errors=319520i sndbuf_errors=0i rx_bytes_delta=1243565i rx_packets_delta=1009i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:15:31Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:32.168741265Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=1852i num_reverse_index_keys=1852i
[2025-03-19T16:15:32.736728515Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5093588992i cached_bytes=131583733760i free_percent=29.37872200130258 used_bytes=49623216128i avail_percent=81.63053857007647 buffers_percent=1.8855385408127954 cached_percent=48.70950556438806 swap_free_percent=0
[2025-03-19T16:15:33.165727615Z INFO solana_accounts_db::accounts_db] generating index: 7556/201498 slots... (3778/s)
[2025-03-19T16:15:33.533023731Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=96992i reads_merged=359055i sectors_read=7631000i time_reading_ms=29694i writes_completed=90265i writes_merged=2283704i sectors_written=18993492i time_writing_ms=14135i io_in_progress=21i time_io_ms=18076i time_io_weighted_ms=43829i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:33.533048630Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=4018i num_reverse_index_keys=4018i
[2025-03-19T16:15:33.533051460Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=919i out_datagrams_delta=8i rcvbuf_errors_delta=919i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=320439i rcvbuf_errors=320439i sndbuf_errors=0i rx_bytes_delta=1206774i rx_packets_delta=976i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6912i tx_packets_delta=17i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:34.171872787Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=5957i num_reverse_index_keys=5957i
[SENTINEL.SH] [2025-03-19T16:15:34Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:35.176570601Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=7573i num_reverse_index_keys=7573i
[2025-03-19T16:15:35.737986685Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1178i out_datagrams_delta=0i rcvbuf_errors_delta=1178i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=321617i rcvbuf_errors=321617i sndbuf_errors=0i rx_bytes_delta=1551998i rx_packets_delta=1240i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:36.179137176Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=9479i num_reverse_index_keys=9479i
[2025-03-19T16:15:37.181128451Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=12154i num_reverse_index_keys=12154i
[SENTINEL.SH] [2025-03-19T16:15:37Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:37.738603921Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=944i out_datagrams_delta=0i rcvbuf_errors_delta=944i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=322561i rcvbuf_errors=322561i sndbuf_errors=0i rx_bytes_delta=1235501i rx_packets_delta=984i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:37.738621741Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5095927808i cached_bytes=131880349696i free_percent=29.578233507276785 used_bytes=49391489024i avail_percent=81.71631902590617 buffers_percent=1.8864043208580241 cached_percent=48.81930648865299 swap_free_percent=0
[2025-03-19T16:15:37.763558161Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3250i average_load_one_minute=19.04 average_load_five_minutes=15.08 average_load_fifteen_minutes=12.02 total_num_threads=2317i
[2025-03-19T16:15:37.763962525Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=167176i reads_merged=354705i sectors_read=19945368i time_reading_ms=62163i writes_completed=79861i writes_merged=2103771i sectors_written=17468466i time_writing_ms=12588i io_in_progress=1i time_io_ms=20120i time_io_weighted_ms=74751i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:38.188407122Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=14220i num_reverse_index_keys=14220i
[2025-03-19T16:15:39.189030568Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=16550i num_reverse_index_keys=16550i
[2025-03-19T16:15:39.764429469Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=998i out_datagrams_delta=0i rcvbuf_errors_delta=998i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=323559i rcvbuf_errors=323559i sndbuf_errors=0i rx_bytes_delta=1306885i rx_packets_delta=1048i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:40.190063700Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=18785i num_reverse_index_keys=18785i
[SENTINEL.SH] [2025-03-19T16:15:40Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:41.129217421Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=12393589i bg_waiting_percent=60.56019592285156 bg_throttling_wait_percent=45.869163513183594 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=168i max_in_bin_disk=17639i count_from_bins_disk=12393589i median_from_bins_disk=645i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=12393589i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=416i buckets_scanned=200824i flush_scan_us=165i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=45742i disk_index_failed_resizes=1895i disk_index_max_size=33365i disk_index_new_file_us=822796i disk_index_resize_us=22289011i disk_index_flush_file_us=10i disk_index_index_file_size=931742304i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=7566252i disk_index_flush_mmap_us=7122933i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:15:41.191178992Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=20422i num_reverse_index_keys=20422i
[2025-03-19T16:15:42.192035708Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=22024i num_reverse_index_keys=22024i
[2025-03-19T16:15:42.264917766Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1252i out_datagrams_delta=0i rcvbuf_errors_delta=1252i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=324811i rcvbuf_errors=324811i sndbuf_errors=0i rx_bytes_delta=1638493i rx_packets_delta=1311i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:42.765045494Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5095976960i cached_bytes=132211392512i free_percent=29.414253856959128 used_bytes=50183909376i avail_percent=81.42298182956638 buffers_percent=1.8864225158852443 cached_percent=48.94185151323341 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:15:43Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:43.558824656Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=91403i reads_merged=44509i sectors_read=17455200i time_reading_ms=43147i writes_completed=18529i writes_merged=166977i sectors_written=1483836i time_writing_ms=2288i io_in_progress=4i time_io_ms=19208i time_io_weighted_ms=45434i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:43.558839186Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=24149i num_reverse_index_keys=24149i
[2025-03-19T16:15:44.199391526Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=26381i num_reverse_index_keys=26381i
[2025-03-19T16:15:44.265916714Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=962i out_datagrams_delta=8i rcvbuf_errors_delta=962i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=325773i rcvbuf_errors=325773i sndbuf_errors=0i rx_bytes_delta=1271317i rx_packets_delta=1028i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10463i tx_packets_delta=21i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:45.205403694Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=27434i num_reverse_index_keys=27434i
[2025-03-19T16:15:46.207006965Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=29165i num_reverse_index_keys=29165i
[2025-03-19T16:15:46.266396093Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=982i out_datagrams_delta=0i rcvbuf_errors_delta=982i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=326755i rcvbuf_errors=326755i sndbuf_errors=0i rx_bytes_delta=1290498i rx_packets_delta=1033i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:15:46Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:47.207126328Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=30738i num_reverse_index_keys=30738i
[2025-03-19T16:15:47.766672037Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5094129664i cached_bytes=132490084352i free_percent=29.584597218047055 used_bytes=50422050816i avail_percent=81.33482692268444 buffers_percent=1.8857386861122176 cached_percent=49.04501731757203 swap_free_percent=0
[2025-03-19T16:15:47.787379256Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=19.56 average_load_five_minutes=15.32 average_load_fifteen_minutes=12.13 total_num_threads=2313i
[2025-03-19T16:15:47.787769501Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=70029i reads_merged=48826i sectors_read=15049896i time_reading_ms=34751i writes_completed=6748i writes_merged=128432i sectors_written=1081078i time_writing_ms=744i io_in_progress=23i time_io_ms=19460i time_io_weighted_ms=35496i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:48.209662298Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=32543i num_reverse_index_keys=32543i
[2025-03-19T16:15:48.288050080Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1016i out_datagrams_delta=0i rcvbuf_errors_delta=1016i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=327771i rcvbuf_errors=327771i sndbuf_errors=0i rx_bytes_delta=1336499i rx_packets_delta=1070i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:49.210197018Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=33967i num_reverse_index_keys=33967i
[SENTINEL.SH] [2025-03-19T16:15:49Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:50.211352620Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=35763i num_reverse_index_keys=35763i
[2025-03-19T16:15:50.288558428Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=948i out_datagrams_delta=0i rcvbuf_errors_delta=948i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=328719i rcvbuf_errors=328719i sndbuf_errors=0i rx_bytes_delta=1250416i rx_packets_delta=1009i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:51.130238127Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=26102505i entries_reused=0i
[2025-03-19T16:15:51.130253876Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=26102505i bg_waiting_percent=43.399410247802734 bg_throttling_wait_percent=35.57144546508789 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=325i max_in_bin_disk=17800i count_from_bins_disk=26102505i median_from_bins_disk=1396i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=13708916i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=204247i flush_scan_us=195i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=29653i disk_index_failed_resizes=5329i disk_index_max_size=33706i disk_index_new_file_us=553040i disk_index_resize_us=37562041i disk_index_flush_file_us=6i disk_index_index_file_size=2062649424i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=11679835i disk_index_flush_mmap_us=3832304i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:15:51.213255963Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=37604i num_reverse_index_keys=37604i
[2025-03-19T16:15:52.215586682Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=39134i num_reverse_index_keys=39134i
[SENTINEL.SH] [2025-03-19T16:15:52Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:52.789053228Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1164i out_datagrams_delta=0i rcvbuf_errors_delta=1164i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=329883i rcvbuf_errors=329883i sndbuf_errors=0i rx_bytes_delta=1541093i rx_packets_delta=1231i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:53.567268268Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5093761024i cached_bytes=132728422400i free_percent=29.54476981971427 used_bytes=50689015808i avail_percent=81.23600214859012 buffers_percent=1.885602223408066 cached_percent=49.13324500456286 swap_free_percent=0
[2025-03-19T16:15:53.567283238Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=94945i reads_merged=50698i sectors_read=20146536i time_reading_ms=46906i writes_completed=11583i writes_merged=150757i sectors_written=1298268i time_writing_ms=1075i io_in_progress=5i time_io_ms=19876i time_io_weighted_ms=47979i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:53.567287988Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=40653i num_reverse_index_keys=40653i
[2025-03-19T16:15:54.223196921Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=42271i num_reverse_index_keys=42271i
[2025-03-19T16:15:54.790122081Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=934i out_datagrams_delta=8i rcvbuf_errors_delta=934i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=330817i rcvbuf_errors=330817i sndbuf_errors=0i rx_bytes_delta=1236962i rx_packets_delta=1003i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10189i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:55.227792671Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=43796i num_reverse_index_keys=43796i
[SENTINEL.SH] [2025-03-19T16:15:55Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:56.228055372Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=45995i num_reverse_index_keys=45995i
[2025-03-19T16:15:56.790616335Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1032i out_datagrams_delta=0i rcvbuf_errors_delta=1032i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=331849i rcvbuf_errors=331849i sndbuf_errors=0i rx_bytes_delta=1362255i rx_packets_delta=1087i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:57.230311738Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=47160i num_reverse_index_keys=47160i
[2025-03-19T16:15:57.790837999Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5090566144i cached_bytes=132944736256i free_percent=29.01956429399367 used_bytes=53011468288i avail_percent=80.3762795311736 buffers_percent=1.8844195466387523 cached_percent=49.21331980310676 swap_free_percent=0
[2025-03-19T16:15:57.815889929Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=19 average_load_five_minutes=15.35 average_load_fifteen_minutes=12.17 total_num_threads=2308i
[2025-03-19T16:15:57.815954378Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=100014i reads_merged=50514i sectors_read=20919432i time_reading_ms=50979i writes_completed=11904i writes_merged=165180i sectors_written=1416622i time_writing_ms=1373i io_in_progress=1i time_io_ms=19992i time_io_weighted_ms=52355i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:15:58.232203000Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=48780i num_reverse_index_keys=48780i
[SENTINEL.SH] [2025-03-19T16:15:58Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:15:58.816210763Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1042i out_datagrams_delta=0i rcvbuf_errors_delta=1042i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=332891i rcvbuf_errors=332891i sndbuf_errors=0i rx_bytes_delta=1378978i rx_packets_delta=1104i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:15:59.233047857Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=50776i num_reverse_index_keys=50776i
[2025-03-19T16:16:00.241493899Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=52273i num_reverse_index_keys=52273i
[2025-03-19T16:16:01.131320563Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=12876565i entries_reused=0i
[2025-03-19T16:16:01.131341072Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=38979070i bg_waiting_percent=48.23517608642578 bg_throttling_wait_percent=40.502201080322266 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=473i max_in_bin_disk=18029i count_from_bins_disk=38979053i median_from_bins_disk=2242i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=12876565i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=416i buckets_scanned=205681i flush_scan_us=87i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=21111i disk_index_failed_resizes=4308i disk_index_max_size=36181i disk_index_new_file_us=414454i disk_index_resize_us=34959727i disk_index_flush_file_us=10i disk_index_index_file_size=3162444720i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=10183627i disk_index_flush_mmap_us=2943654i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:01.243130557Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=53950i num_reverse_index_keys=53950i
[2025-03-19T16:16:01.317068492Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1334i out_datagrams_delta=0i rcvbuf_errors_delta=1334i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=334225i rcvbuf_errors=334225i sndbuf_errors=0i rx_bytes_delta=1758635i rx_packets_delta=1410i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:16:01Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:02.247077035Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=55520i num_reverse_index_keys=55520i
[2025-03-19T16:16:02.817390033Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5086298112i cached_bytes=133228003328i free_percent=29.116675703024768 used_bytes=52644802560i avail_percent=80.51201140173185 buffers_percent=1.8828396117751303 cached_percent=49.318179261229126 swap_free_percent=0
[2025-03-19T16:16:03.594707274Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=91959i reads_merged=26754i sectors_read=19486976i time_reading_ms=45582i writes_completed=4876i writes_merged=71479i sectors_written=622428i time_writing_ms=555i io_in_progress=9i time_io_ms=19952i time_io_weighted_ms=46138i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:03.594728523Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=57240i num_reverse_index_keys=57240i
[2025-03-19T16:16:03.594731223Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=955i out_datagrams_delta=8i rcvbuf_errors_delta=955i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=335180i rcvbuf_errors=335180i sndbuf_errors=0i rx_bytes_delta=1261962i rx_packets_delta=1026i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10253i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:04.256309695Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=58816i num_reverse_index_keys=58816i
[SENTINEL.SH] [2025-03-19T16:16:04Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:05.257620553Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=60594i num_reverse_index_keys=60594i
[2025-03-19T16:16:05.319702994Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1040i out_datagrams_delta=0i rcvbuf_errors_delta=1040i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=336220i rcvbuf_errors=336220i sndbuf_errors=0i rx_bytes_delta=1364287i rx_packets_delta=1090i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:06.264287667Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=62110i num_reverse_index_keys=62110i
[2025-03-19T16:16:07.266019932Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=63665i num_reverse_index_keys=63665i
[2025-03-19T16:16:07.320111285Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1327i out_datagrams_delta=0i rcvbuf_errors_delta=1327i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=337547i rcvbuf_errors=337547i sndbuf_errors=0i rx_bytes_delta=1732537i rx_packets_delta=1388i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:16:07Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:07.820298737Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4980723712i cached_bytes=133386477568i free_percent=29.141346643683114 used_bytes=53151481856i avail_percent=80.32444947988454 buffers_percent=1.8437582095583753 cached_percent=49.37684306149164 swap_free_percent=0
[2025-03-19T16:16:07.840601763Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=19.29 average_load_five_minutes=15.53 average_load_fifteen_minutes=12.27 total_num_threads=2311i
[2025-03-19T16:16:07.841072736Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=101746i reads_merged=46302i sectors_read=20932408i time_reading_ms=51812i writes_completed=19351i writes_merged=289722i sectors_written=2472256i time_writing_ms=2126i io_in_progress=1i time_io_ms=19964i time_io_weighted_ms=53935i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:08.267062387Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=65106i num_reverse_index_keys=65106i
[2025-03-19T16:16:09.268360560Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=66712i num_reverse_index_keys=66712i
[2025-03-19T16:16:09.341762893Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1093i out_datagrams_delta=0i rcvbuf_errors_delta=1093i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=338640i rcvbuf_errors=338640i sndbuf_errors=0i rx_bytes_delta=1439442i rx_packets_delta=1153i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:10.269190121Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=68260i num_reverse_index_keys=68260i
[SENTINEL.SH] [2025-03-19T16:16:10Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:11.132432977Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=12725223i entries_reused=0i
[2025-03-19T16:16:11.132454546Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=51704293i bg_waiting_percent=52.547245025634766 bg_throttling_wait_percent=43.19192886352539 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=577i max_in_bin_disk=18169i count_from_bins_disk=51704293i median_from_bins_disk=4799i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=12725223i deletes=0i active_threads=5i items=0i keys=0i ms_per_age=400i buckets_scanned=210326i flush_scan_us=74i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=15586i disk_index_failed_resizes=3437i disk_index_max_size=35072i disk_index_new_file_us=321430i disk_index_resize_us=32100349i disk_index_flush_file_us=2i disk_index_index_file_size=4240046496i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=9194979i disk_index_flush_mmap_us=2428555i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:11.270370687Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=69790i num_reverse_index_keys=69790i
[2025-03-19T16:16:11.842265359Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1310i out_datagrams_delta=0i rcvbuf_errors_delta=1310i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=339950i rcvbuf_errors=339950i sndbuf_errors=0i rx_bytes_delta=1710742i rx_packets_delta=1372i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:12.274420526Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=71527i num_reverse_index_keys=71527i
[2025-03-19T16:16:12.842476871Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4980035584i cached_bytes=133492940800i free_percent=29.171400279894105 used_bytes=53549801472i avail_percent=80.17700001179644 buffers_percent=1.8435034791772924 cached_percent=49.41625349045063 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:16:13Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:13.616474958Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=99758i reads_merged=27005i sectors_read=20924336i time_reading_ms=49921i writes_completed=5456i writes_merged=62398i sectors_written=542752i time_writing_ms=699i io_in_progress=16i time_io_ms=19916i time_io_weighted_ms=50622i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:13.616491998Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=73156i num_reverse_index_keys=73156i
[2025-03-19T16:16:13.843381245Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=958i out_datagrams_delta=8i rcvbuf_errors_delta=958i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=340908i rcvbuf_errors=340908i sndbuf_errors=0i rx_bytes_delta=1268789i rx_packets_delta=1025i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10685i tx_packets_delta=22i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:14.279884061Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=74778i num_reverse_index_keys=74778i
[2025-03-19T16:16:15.287728100Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=76560i num_reverse_index_keys=76560i
[2025-03-19T16:16:16.289781030Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=77887i num_reverse_index_keys=77887i
[2025-03-19T16:16:16.343856694Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1283i out_datagrams_delta=0i rcvbuf_errors_delta=1283i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=342191i rcvbuf_errors=342191i sndbuf_errors=0i rx_bytes_delta=1701021i rx_packets_delta=1359i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:16:16Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:17.290031707Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=80221i num_reverse_index_keys=80221i
[2025-03-19T16:16:17.844193451Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5008277504i cached_bytes=133503631360i free_percent=29.13488740901994 used_bytes=54210822144i avail_percent=79.93230418822537 buffers_percent=1.8539580385675745 cached_percent=49.42021090887103 swap_free_percent=0
[2025-03-19T16:16:17.864615745Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=19.36 average_load_five_minutes=15.68 average_load_fifteen_minutes=12.35 total_num_threads=2310i
[2025-03-19T16:16:17.864628264Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=113879i reads_merged=72753i sectors_read=20987440i time_reading_ms=52288i writes_completed=18347i writes_merged=171600i sectors_written=1519132i time_writing_ms=2149i io_in_progress=17i time_io_ms=20096i time_io_weighted_ms=54434i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:18.291591506Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=82120i num_reverse_index_keys=82120i
[2025-03-19T16:16:18.364875052Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=963i out_datagrams_delta=0i rcvbuf_errors_delta=963i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=343154i rcvbuf_errors=343154i sndbuf_errors=0i rx_bytes_delta=1266676i rx_packets_delta=1017i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:19.292486657Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=83683i num_reverse_index_keys=83683i
[SENTINEL.SH] [2025-03-19T16:16:19Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:20.294074388Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=86308i num_reverse_index_keys=86308i
[2025-03-19T16:16:20.365292533Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1050i out_datagrams_delta=0i rcvbuf_errors_delta=1050i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=344204i rcvbuf_errors=344204i sndbuf_errors=0i rx_bytes_delta=1379131i rx_packets_delta=1111i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:21.133255361Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=12232643i entries_reused=0i
[2025-03-19T16:16:21.133269190Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=63936936i bg_waiting_percent=53.989601135253906 bg_throttling_wait_percent=44.27182388305664 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=699i max_in_bin_disk=18300i count_from_bins_disk=63936472i median_from_bins_disk=9302i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=12232643i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=210375i flush_scan_us=85i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=12961i disk_index_failed_resizes=3205i disk_index_max_size=37120i disk_index_new_file_us=272979i disk_index_resize_us=31597710i disk_index_flush_file_us=7i disk_index_index_file_size=5324665104i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=8631042i disk_index_flush_mmap_us=1855939i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:21.295682241Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=87957i num_reverse_index_keys=87957i
[2025-03-19T16:16:22.296162290Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=89259i num_reverse_index_keys=89259i
[SENTINEL.SH] [2025-03-19T16:16:22Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:22.865865490Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1086i out_datagrams_delta=0i rcvbuf_errors_delta=1086i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=345290i rcvbuf_errors=345290i sndbuf_errors=0i rx_bytes_delta=1416175i rx_packets_delta=1137i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:23.639389292Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5008220160i cached_bytes=133411868672i free_percent=29.22192180547557 used_bytes=54861758464i avail_percent=79.691341377739 buffers_percent=1.8539368110358179 cached_percent=49.38624230930315 swap_free_percent=0
[2025-03-19T16:16:23.639406561Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=108179i reads_merged=56390i sectors_read=21224216i time_reading_ms=51981i writes_completed=11663i writes_merged=146805i sectors_written=1267518i time_writing_ms=1517i io_in_progress=0i time_io_ms=20012i time_io_weighted_ms=53500i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:23.639410331Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=91029i num_reverse_index_keys=91029i
[2025-03-19T16:16:24.301283414Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=92125i num_reverse_index_keys=92125i
[2025-03-19T16:16:24.866840102Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=924i out_datagrams_delta=8i rcvbuf_errors_delta=924i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=346214i rcvbuf_errors=346214i sndbuf_errors=0i rx_bytes_delta=1220473i rx_packets_delta=991i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10188i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:25.302857661Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=93624i num_reverse_index_keys=93624i
[SENTINEL.SH] [2025-03-19T16:16:25Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:26.305366457Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=95371i num_reverse_index_keys=95371i
[2025-03-19T16:16:26.867303657Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=963i out_datagrams_delta=0i rcvbuf_errors_delta=963i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=347177i rcvbuf_errors=347177i sndbuf_errors=0i rx_bytes_delta=1271085i rx_packets_delta=1013i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:27.313707092Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=96462i num_reverse_index_keys=96462i
[2025-03-19T16:16:27.867510877Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5006254080i cached_bytes=133541908480i free_percent=28.88787323573056 used_bytes=55927803904i avail_percent=79.29671398111653 buffers_percent=1.8532090099470089 cached_percent=49.43438028631877 swap_free_percent=0
[2025-03-19T16:16:27.887110903Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=19.4 average_load_five_minutes=15.8 average_load_fifteen_minutes=12.43 total_num_threads=2309i
[2025-03-19T16:16:27.887487368Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=99708i reads_merged=62257i sectors_read=20125088i time_reading_ms=48658i writes_completed=16629i writes_merged=219069i sectors_written=1885218i time_writing_ms=2226i io_in_progress=5i time_io_ms=20024i time_io_weighted_ms=50882i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:28.315185135Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=98163i num_reverse_index_keys=98163i
[SENTINEL.SH] [2025-03-19T16:16:28Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:28.887805337Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=928i out_datagrams_delta=0i rcvbuf_errors_delta=928i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=348105i rcvbuf_errors=348105i sndbuf_errors=0i rx_bytes_delta=1216085i rx_packets_delta=971i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:29.317231070Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=99430i num_reverse_index_keys=99430i
[2025-03-19T16:16:30.319112789Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=100416i num_reverse_index_keys=100416i
[2025-03-19T16:16:30.888219162Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=852i out_datagrams_delta=0i rcvbuf_errors_delta=852i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=348957i rcvbuf_errors=348957i sndbuf_errors=0i rx_bytes_delta=1124782i rx_packets_delta=899i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:31.134376354Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=11423671i entries_reused=0i
[2025-03-19T16:16:31.134389464Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=75360607i bg_waiting_percent=60.60268783569336 bg_throttling_wait_percent=50.25372314453125 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=738i max_in_bin_disk=18482i count_from_bins_disk=75360607i median_from_bins_disk=11070i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=11423671i deletes=0i active_threads=3i items=0i keys=0i ms_per_age=400i buckets_scanned=209697i flush_scan_us=55i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=9847i disk_index_failed_resizes=2430i disk_index_max_size=38400i disk_index_new_file_us=212908i disk_index_resize_us=27134745i disk_index_flush_file_us=1i disk_index_index_file_size=6327945168i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6987973i disk_index_flush_mmap_us=1304863i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:31.324964233Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=102228i num_reverse_index_keys=102228i
[SENTINEL.SH] [2025-03-19T16:16:31Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:32.330526243Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=103192i num_reverse_index_keys=103192i
[2025-03-19T16:16:32.888549772Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5004754944i cached_bytes=133487955968i free_percent=29.22229935229039 used_bytes=55834558464i avail_percent=79.33123146400554 buffers_percent=1.8526540616167926 cached_percent=49.41440821144004 swap_free_percent=0
[2025-03-19T16:16:33.665483835Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=89901i reads_merged=16020i sectors_read=19278528i time_reading_ms=44518i writes_completed=5885i writes_merged=37479i sectors_written=349180i time_writing_ms=358i io_in_progress=1i time_io_ms=19956i time_io_weighted_ms=44879i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:33.665498984Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=104356i num_reverse_index_keys=104356i
[2025-03-19T16:16:33.665501784Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1163i out_datagrams_delta=8i rcvbuf_errors_delta=1163i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=350120i rcvbuf_errors=350120i sndbuf_errors=0i rx_bytes_delta=1531865i rx_packets_delta=1241i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10258i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:34.337508483Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=106056i num_reverse_index_keys=106056i
[SENTINEL.SH] [2025-03-19T16:16:34Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:35.338671018Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=106881i num_reverse_index_keys=106881i
[2025-03-19T16:16:35.889747776Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1192i out_datagrams_delta=0i rcvbuf_errors_delta=1192i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=351312i rcvbuf_errors=351312i sndbuf_errors=0i rx_bytes_delta=1581703i rx_packets_delta=1269i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:36.346010317Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=108450i num_reverse_index_keys=108450i
[2025-03-19T16:16:37.352257343Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=110057i num_reverse_index_keys=110057i
[SENTINEL.SH] [2025-03-19T16:16:37Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:37.890047637Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=5003911168i cached_bytes=133515091968i free_percent=29.23446727674391 used_bytes=55988551680i avail_percent=79.27422644372461 buffers_percent=1.8523417136495122 cached_percent=49.42445338271787 swap_free_percent=0
[2025-03-19T16:16:37.915433233Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=18.55 average_load_five_minutes=15.74 average_load_fifteen_minutes=12.44 total_num_threads=2309i
[2025-03-19T16:16:37.915900406Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=88989i reads_merged=26493i sectors_read=19312512i time_reading_ms=44714i writes_completed=4298i writes_merged=65472i sectors_written=557828i time_writing_ms=447i io_in_progress=5i time_io_ms=20012i time_io_weighted_ms=45160i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:38.356354510Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=111366i num_reverse_index_keys=111366i
[2025-03-19T16:16:38.416169765Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1237i out_datagrams_delta=0i rcvbuf_errors_delta=1237i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=352549i rcvbuf_errors=352549i sndbuf_errors=0i rx_bytes_delta=1617808i rx_packets_delta=1294i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:39.360332939Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=112669i num_reverse_index_keys=112669i
[2025-03-19T16:16:40.363933446Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=114622i num_reverse_index_keys=114622i
[2025-03-19T16:16:40.416631060Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=940i out_datagrams_delta=0i rcvbuf_errors_delta=940i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=353489i rcvbuf_errors=353489i sndbuf_errors=0i rx_bytes_delta=1235575i rx_packets_delta=998i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:16:40Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:41.135451190Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=10100532i entries_reused=0i
[2025-03-19T16:16:41.135466420Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=85461139i bg_waiting_percent=63.45115280151367 bg_throttling_wait_percent=52.38351058959961 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=836i max_in_bin_disk=18537i count_from_bins_disk=85460783i median_from_bins_disk=12519i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=10100532i deletes=0i active_threads=8i items=0i keys=0i ms_per_age=400i buckets_scanned=215255i flush_scan_us=41i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=8716i disk_index_failed_resizes=2246i disk_index_max_size=37034i disk_index_new_file_us=193687i disk_index_resize_us=25130439i disk_index_flush_file_us=0i disk_index_index_file_size=7216580544i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6520409i disk_index_flush_mmap_us=1177670i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:41.364166749Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=115591i num_reverse_index_keys=115591i
[2025-03-19T16:16:42.366003392Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=116993i num_reverse_index_keys=116993i
[2025-03-19T16:16:42.917186373Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1275i out_datagrams_delta=0i rcvbuf_errors_delta=1275i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=354764i rcvbuf_errors=354764i sndbuf_errors=0i rx_bytes_delta=1654328i rx_packets_delta=1315i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:16:43Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:43.667655824Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4993269760i cached_bytes=133519880192i free_percent=28.983613952711064 used_bytes=57123897344i avail_percent=78.85394557497392 buffers_percent=1.8484024902563356 cached_percent=49.42622588161957 swap_free_percent=0
[2025-03-19T16:16:43.667668803Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=98183i reads_merged=23541i sectors_read=20145800i time_reading_ms=47605i writes_completed=12412i writes_merged=108651i sectors_written=968498i time_writing_ms=1197i io_in_progress=8i time_io_ms=19912i time_io_weighted_ms=48802i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:43.667673454Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=118556i num_reverse_index_keys=118556i
[2025-03-19T16:16:44.371610894Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=119536i num_reverse_index_keys=119536i
[2025-03-19T16:16:44.918114111Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=829i out_datagrams_delta=8i rcvbuf_errors_delta=829i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=355593i rcvbuf_errors=355593i sndbuf_errors=0i rx_bytes_delta=1104855i rx_packets_delta=897i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10269i tx_packets_delta=21i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:45.372559273Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=120938i num_reverse_index_keys=120938i
[2025-03-19T16:16:46.373879367Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=122098i num_reverse_index_keys=122098i
[SENTINEL.SH] [2025-03-19T16:16:46Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:46.918560250Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=979i out_datagrams_delta=0i rcvbuf_errors_delta=979i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=356572i rcvbuf_errors=356572i sndbuf_errors=0i rx_bytes_delta=1285172i rx_packets_delta=1029i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:47.376205938Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=123071i num_reverse_index_keys=123071i
[2025-03-19T16:16:47.918761002Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4989497344i cached_bytes=133473939456i free_percent=29.023975071842305 used_bytes=57760989184i avail_percent=78.61810769715457 buffers_percent=1.847006021917184 cached_percent=49.40921959617774 swap_free_percent=0
[2025-03-19T16:16:47.943130092Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=18.45 average_load_five_minutes=15.82 average_load_fifteen_minutes=12.5 total_num_threads=2307i
[2025-03-19T16:16:47.943514717Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=92299i reads_merged=49110i sectors_read=19790304i time_reading_ms=46287i writes_completed=7586i writes_merged=124486i sectors_written=1056196i time_writing_ms=811i io_in_progress=15i time_io_ms=20032i time_io_weighted_ms=47098i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:48.379097283Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=124267i num_reverse_index_keys=124267i
[2025-03-19T16:16:48.943883317Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=999i out_datagrams_delta=0i rcvbuf_errors_delta=999i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=357571i rcvbuf_errors=357571i sndbuf_errors=0i rx_bytes_delta=1314714i rx_packets_delta=1055i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:49.380143184Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=125409i num_reverse_index_keys=125409i
[SENTINEL.SH] [2025-03-19T16:16:49Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:50.381670651Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=126542i num_reverse_index_keys=126542i
[2025-03-19T16:16:50.944345416Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=890i out_datagrams_delta=0i rcvbuf_errors_delta=890i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=358461i rcvbuf_errors=358461i sndbuf_errors=0i rx_bytes_delta=1173442i rx_packets_delta=940i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:51.137370034Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=9870540i entries_reused=0i
[2025-03-19T16:16:51.137387364Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=95331679i bg_waiting_percent=64.2583999633789 bg_throttling_wait_percent=53.909217834472656 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=878i max_in_bin_disk=18583i count_from_bins_disk=95331679i median_from_bins_disk=13765i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=9870540i deletes=0i active_threads=8i items=0i keys=0i ms_per_age=416i buckets_scanned=210573i flush_scan_us=77i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=7570i disk_index_failed_resizes=1960i disk_index_max_size=38058i disk_index_new_file_us=172622i disk_index_resize_us=24650434i disk_index_flush_file_us=0i disk_index_index_file_size=8104184208i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6564322i disk_index_flush_mmap_us=1114236i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:16:51.383546972Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=127810i num_reverse_index_keys=127810i
[2025-03-19T16:16:52.384317451Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=129005i num_reverse_index_keys=129005i
[SENTINEL.SH] [2025-03-19T16:16:52Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:52.944706399Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4964118528i cached_bytes=133656162304i free_percent=29.183944234910175 used_bytes=57350696960i avail_percent=78.76998917062305 buffers_percent=1.8376113228624793 cached_percent=49.47667462709216 swap_free_percent=0
[2025-03-19T16:16:53.718132908Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=94540i reads_merged=33523i sectors_read=19686096i time_reading_ms=46232i writes_completed=11409i writes_merged=139370i sectors_written=1206092i time_writing_ms=1142i io_in_progress=6i time_io_ms=19984i time_io_weighted_ms=47375i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:53.718149518Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=130270i num_reverse_index_keys=130270i
[2025-03-19T16:16:53.718155258Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1251i out_datagrams_delta=8i rcvbuf_errors_delta=1251i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=359712i rcvbuf_errors=359712i sndbuf_errors=0i rx_bytes_delta=1658310i rx_packets_delta=1343i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10286i tx_packets_delta=19i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:54.391594282Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=131315i num_reverse_index_keys=131315i
[2025-03-19T16:16:55.396068311Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=132447i num_reverse_index_keys=132447i
[SENTINEL.SH] [2025-03-19T16:16:55Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:55.946029177Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1168i out_datagrams_delta=0i rcvbuf_errors_delta=1168i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=360880i rcvbuf_errors=360880i sndbuf_errors=0i rx_bytes_delta=1544401i rx_packets_delta=1233i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:56.400732490Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=133625i num_reverse_index_keys=133625i
[2025-03-19T16:16:57.402522700Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=134807i num_reverse_index_keys=134807i
[2025-03-19T16:16:57.946517699Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=996i out_datagrams_delta=0i rcvbuf_errors_delta=996i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=361876i rcvbuf_errors=361876i sndbuf_errors=0i rx_bytes_delta=1313826i rx_packets_delta=1050i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:16:57.946579268Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4965572608i cached_bytes=133622022144i free_percent=29.233411965165136 used_bytes=57670955008i avail_percent=78.6514364382652 buffers_percent=1.838149592417744 cached_percent=49.464036664435454 swap_free_percent=0
[2025-03-19T16:16:57.972015024Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=18.96 average_load_five_minutes=16.02 average_load_fifteen_minutes=12.61 total_num_threads=2310i
[2025-03-19T16:16:57.972066603Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=88717i reads_merged=24568i sectors_read=19143856i time_reading_ms=44799i writes_completed=4048i writes_merged=53633i sectors_written=461134i time_writing_ms=428i io_in_progress=0i time_io_ms=19992i time_io_weighted_ms=45225i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:16:58.404834273Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=135877i num_reverse_index_keys=135877i
[SENTINEL.SH] [2025-03-19T16:16:58Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:16:59.408168914Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=137081i num_reverse_index_keys=137081i
[2025-03-19T16:16:59.972524661Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=931i out_datagrams_delta=0i rcvbuf_errors_delta=931i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=362807i rcvbuf_errors=362807i sndbuf_errors=0i rx_bytes_delta=1217142i rx_packets_delta=978i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:00.409500273Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=138226i num_reverse_index_keys=138226i
[2025-03-19T16:17:01.138260470Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=9530484i entries_reused=0i
[2025-03-19T16:17:01.138277680Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=104862163i bg_waiting_percent=63.18368148803711 bg_throttling_wait_percent=53.65338134765625 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=958i max_in_bin_disk=18771i count_from_bins_disk=104862163i median_from_bins_disk=14793i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=9530484i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=216393i flush_scan_us=49i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=7105i disk_index_failed_resizes=1881i disk_index_max_size=39850i disk_index_new_file_us=170061i disk_index_resize_us=25511420i disk_index_flush_file_us=0i disk_index_index_file_size=8975402016i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6852643i disk_index_flush_mmap_us=1033229i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:01.410524227Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=139441i num_reverse_index_keys=139441i
[SENTINEL.SH] [2025-03-19T16:17:01Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:02.413954210Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=140502i num_reverse_index_keys=140502i
[2025-03-19T16:17:02.473055877Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1195i out_datagrams_delta=0i rcvbuf_errors_delta=1195i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=364002i rcvbuf_errors=364002i sndbuf_errors=0i rx_bytes_delta=1571953i rx_packets_delta=1259i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:02.973352587Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4967976960i cached_bytes=133667491840i free_percent=29.194968905153356 used_bytes=58176905216i avail_percent=78.46414440932163 buffers_percent=1.8390396324992662 cached_percent=49.480868580866414 swap_free_percent=0
[2025-03-19T16:17:03.745968686Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=87894i reads_merged=19344i sectors_read=18961944i time_reading_ms=43426i writes_completed=3249i writes_merged=41584i sectors_written=358296i time_writing_ms=342i io_in_progress=1i time_io_ms=19908i time_io_weighted_ms=43768i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:03.745983856Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=141628i num_reverse_index_keys=141628i
[2025-03-19T16:17:04.418294632Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=142603i num_reverse_index_keys=142603i
[2025-03-19T16:17:04.475011822Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=976i out_datagrams_delta=8i rcvbuf_errors_delta=976i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=364978i rcvbuf_errors=364978i sndbuf_errors=0i rx_bytes_delta=1282658i rx_packets_delta=1041i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10687i tx_packets_delta=22i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:04Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:05.419732875Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=143683i num_reverse_index_keys=143683i
[2025-03-19T16:17:06.421671872Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=144660i num_reverse_index_keys=144660i
[2025-03-19T16:17:06.475462843Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=900i out_datagrams_delta=0i rcvbuf_errors_delta=900i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=365878i rcvbuf_errors=365878i sndbuf_errors=0i rx_bytes_delta=1185806i rx_packets_delta=946i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:07.424789244Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=145796i num_reverse_index_keys=145796i
[SENTINEL.SH] [2025-03-19T16:17:07Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:07.975740273Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4969361408i cached_bytes=133669040128i free_percent=29.226917856699785 used_bytes=58474463232i avail_percent=78.35399474703499 buffers_percent=1.839552125765969 cached_percent=49.481441724223856 swap_free_percent=0
[2025-03-19T16:17:07.999072988Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.57 average_load_five_minutes=15.82 average_load_fifteen_minutes=12.58 total_num_threads=2308i
[2025-03-19T16:17:07.999438103Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=89346i reads_merged=24621i sectors_read=18309056i time_reading_ms=43616i writes_completed=12422i writes_merged=112553i sectors_written=999502i time_writing_ms=1248i io_in_progress=0i time_io_ms=19940i time_io_weighted_ms=44868i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:08.428246232Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=146891i num_reverse_index_keys=146891i
[2025-03-19T16:17:08.499675648Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1013i out_datagrams_delta=0i rcvbuf_errors_delta=1013i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=366891i rcvbuf_errors=366891i sndbuf_errors=0i rx_bytes_delta=1320167i rx_packets_delta=1057i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:09.430823394Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=147906i num_reverse_index_keys=147906i
[2025-03-19T16:17:10.433397966Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=148988i num_reverse_index_keys=148988i
[SENTINEL.SH] [2025-03-19T16:17:10Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:11.000202016Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1210i out_datagrams_delta=0i rcvbuf_errors_delta=1210i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=368101i rcvbuf_errors=368101i sndbuf_errors=0i rx_bytes_delta=1576670i rx_packets_delta=1266i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:11.140152698Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=8913922i entries_reused=0i
[2025-03-19T16:17:11.140168497Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=113776085i bg_waiting_percent=65.50970458984375 bg_throttling_wait_percent=54.563297271728516 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=1056i max_in_bin_disk=19490i count_from_bins_disk=113776085i median_from_bins_disk=15735i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=8913922i deletes=0i active_threads=8i items=0i keys=0i ms_per_age=400i buckets_scanned=212360i flush_scan_us=56i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=5999i disk_index_failed_resizes=1508i disk_index_max_size=41898i disk_index_new_file_us=155360i disk_index_resize_us=23849494i disk_index_flush_file_us=0i disk_index_index_file_size=9775178352i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6761111i disk_index_flush_mmap_us=942979i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:11.434675288Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=150053i num_reverse_index_keys=150053i
[2025-03-19T16:17:12.436374334Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=150964i num_reverse_index_keys=150964i
[2025-03-19T16:17:13.000616931Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=993i out_datagrams_delta=0i rcvbuf_errors_delta=993i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=369094i rcvbuf_errors=369094i sndbuf_errors=0i rx_bytes_delta=1294971i rx_packets_delta=1037i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:13Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:13.760047611Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4970774528i cached_bytes=133673742336i free_percent=29.20041831580581 used_bytes=58936209408i avail_percent=78.18306611257148 buffers_percent=1.8400752327985503 cached_percent=49.48318238182792 swap_free_percent=0
[2025-03-19T16:17:13.760061821Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=86477i reads_merged=57438i sectors_read=18633704i time_reading_ms=43241i writes_completed=7618i writes_merged=139458i sectors_written=1176260i time_writing_ms=804i io_in_progress=19i time_io_ms=19924i time_io_weighted_ms=44041i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:13.760066451Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=152197i num_reverse_index_keys=152197i
[2025-03-19T16:17:14.441849352Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=153191i num_reverse_index_keys=153191i
[2025-03-19T16:17:15.001539432Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=961i out_datagrams_delta=8i rcvbuf_errors_delta=961i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=370055i rcvbuf_errors=370055i sndbuf_errors=0i rx_bytes_delta=1275595i rx_packets_delta=1032i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10202i tx_packets_delta=20i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:15.444510879Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=154212i num_reverse_index_keys=154212i
[2025-03-19T16:17:16.451021893Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=155357i num_reverse_index_keys=155357i
[SENTINEL.SH] [2025-03-19T16:17:16Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:17.451673409Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=156391i num_reverse_index_keys=156391i
[2025-03-19T16:17:17.502102067Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1087i out_datagrams_delta=0i rcvbuf_errors_delta=1087i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=371142i rcvbuf_errors=371142i sndbuf_errors=0i rx_bytes_delta=1436993i rx_packets_delta=1144i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:18.002274568Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4971913216i cached_bytes=133681885184i free_percent=29.19688848052509 used_bytes=59312930816i avail_percent=78.0436118426944 buffers_percent=1.8404967509291517 cached_percent=49.486196691337405 swap_free_percent=0
[2025-03-19T16:17:18.023308245Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.52 average_load_five_minutes=15.86 average_load_fifteen_minutes=12.63 total_num_threads=2307i
[2025-03-19T16:17:18.023687230Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=91199i reads_merged=11336i sectors_read=18669104i time_reading_ms=44331i writes_completed=9462i writes_merged=22190i sectors_written=253076i time_writing_ms=835i io_in_progress=1i time_io_ms=19948i time_io_weighted_ms=45165i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:18.452153469Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=157644i num_reverse_index_keys=157644i
[2025-03-19T16:17:19.455895514Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=158663i num_reverse_index_keys=158663i
[2025-03-19T16:17:19.524098215Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1014i out_datagrams_delta=0i rcvbuf_errors_delta=1014i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=372156i rcvbuf_errors=372156i sndbuf_errors=0i rx_bytes_delta=1332401i rx_packets_delta=1069i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:19Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:20.456572293Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=159640i num_reverse_index_keys=159640i
[2025-03-19T16:17:21.140224623Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=8577360i entries_reused=0i
[2025-03-19T16:17:21.140238393Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=122353445i bg_waiting_percent=66.93580627441406 bg_throttling_wait_percent=56.95555114746094 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=1122i max_in_bin_disk=23082i count_from_bins_disk=122353439i median_from_bins_disk=16529i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=8577360i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=217929i flush_scan_us=71i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=5421i disk_index_failed_resizes=1402i disk_index_max_size=39594i disk_index_new_file_us=142185i disk_index_resize_us=23013094i disk_index_flush_file_us=0i disk_index_index_file_size=10546592208i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6450989i disk_index_flush_mmap_us=860913i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:21.458557266Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=160817i num_reverse_index_keys=160817i
[2025-03-19T16:17:21.524552007Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=936i out_datagrams_delta=0i rcvbuf_errors_delta=936i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=373092i rcvbuf_errors=373092i sndbuf_errors=0i rx_bytes_delta=1215095i rx_packets_delta=983i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:22.464462794Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=161919i num_reverse_index_keys=161919i
[SENTINEL.SH] [2025-03-19T16:17:22Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:23.024841520Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4973694976i cached_bytes=133689462784i free_percent=29.17735460255192 used_bytes=59756531712i avail_percent=77.87940020577969 buffers_percent=1.8411563206658845 cached_percent=49.48900175803386 swap_free_percent=0
[2025-03-19T16:17:23.795012008Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=86089i reads_merged=7086i sectors_read=18742432i time_reading_ms=42910i writes_completed=1182i writes_merged=10951i sectors_written=96960i time_writing_ms=81i io_in_progress=1i time_io_ms=19864i time_io_weighted_ms=42993i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:23.795027228Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=162635i num_reverse_index_keys=162635i
[2025-03-19T16:17:23.795029558Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=899i out_datagrams_delta=8i rcvbuf_errors_delta=899i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=373991i rcvbuf_errors=373991i sndbuf_errors=0i rx_bytes_delta=1183237i rx_packets_delta=960i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10211i tx_packets_delta=18i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:24.467045302Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=163639i num_reverse_index_keys=163639i
[2025-03-19T16:17:25.468979339Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=164665i num_reverse_index_keys=164665i
[SENTINEL.SH] [2025-03-19T16:17:25Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:26.026082352Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1117i out_datagrams_delta=0i rcvbuf_errors_delta=1117i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=375108i rcvbuf_errors=375108i sndbuf_errors=0i rx_bytes_delta=1471339i rx_packets_delta=1177i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=186i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:26.477526425Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=165418i num_reverse_index_keys=165418i
[2025-03-19T16:17:27.478902902Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=166562i num_reverse_index_keys=166562i
[2025-03-19T16:17:28.026657146Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=911i out_datagrams_delta=0i rcvbuf_errors_delta=911i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=376019i rcvbuf_errors=376019i sndbuf_errors=0i rx_bytes_delta=1202193i rx_packets_delta=960i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:28.026737314Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4973465600i cached_bytes=133741883392i free_percent=29.016848686181056 used_bytes=60510486528i avail_percent=77.60030213449201 buffers_percent=1.841071410538857 cached_percent=49.50840675456421 swap_free_percent=0
[2025-03-19T16:17:28.047585615Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=18.08 average_load_five_minutes=16.03 average_load_fifteen_minutes=12.72 total_num_threads=2309i
[2025-03-19T16:17:28.048040208Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=89938i reads_merged=74404i sectors_read=18794048i time_reading_ms=44156i writes_completed=12491i writes_merged=235018i sectors_written=1979634i time_writing_ms=1451i io_in_progress=13i time_io_ms=20068i time_io_weighted_ms=45606i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:28.485222010Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=167751i num_reverse_index_keys=167751i
[SENTINEL.SH] [2025-03-19T16:17:28Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:29.491330453Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=168458i num_reverse_index_keys=168458i
[2025-03-19T16:17:30.048492248Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=940i out_datagrams_delta=0i rcvbuf_errors_delta=940i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=376959i rcvbuf_errors=376959i sndbuf_errors=0i rx_bytes_delta=1243463i rx_packets_delta=998i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=42i tx_packets_delta=1i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:30.493254985Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=169621i num_reverse_index_keys=169621i
[2025-03-19T16:17:31.141256497Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=8547297i entries_reused=0i
[2025-03-19T16:17:31.141275647Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=130900742i bg_waiting_percent=67.48574829101563 bg_throttling_wait_percent=57.57924270629883 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=1182i max_in_bin_disk=23605i count_from_bins_disk=130900741i median_from_bins_disk=17213i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=8547297i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=416i buckets_scanned=214752i flush_scan_us=33i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=4990i disk_index_failed_resizes=1282i disk_index_max_size=44373i disk_index_new_file_us=137814i disk_index_resize_us=22520028i disk_index_flush_file_us=6i disk_index_index_file_size=11309061600i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6601837i disk_index_flush_mmap_us=821940i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:31.501912275Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=171065i num_reverse_index_keys=171065i
[SENTINEL.SH] [2025-03-19T16:17:31Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:32.507374729Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=171754i num_reverse_index_keys=171754i
[2025-03-19T16:17:32.549123539Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1234i out_datagrams_delta=0i rcvbuf_errors_delta=1234i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=378193i rcvbuf_errors=378193i sndbuf_errors=0i rx_bytes_delta=1610515i rx_packets_delta=1295i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:33.049279997Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4972544000i cached_bytes=133662121984i free_percent=28.987230214371078 used_bytes=61041025024i avail_percent=77.40390804318153 buffers_percent=1.840730253778478 cached_percent=49.47888077414261 swap_free_percent=0
[2025-03-19T16:17:33.821330947Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=91208i reads_merged=22591i sectors_read=18584248i time_reading_ms=44609i writes_completed=9906i writes_merged=49233i sectors_written=472864i time_writing_ms=996i io_in_progress=8i time_io_ms=19832i time_io_weighted_ms=45605i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:33.821344207Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=173046i num_reverse_index_keys=173046i
[2025-03-19T16:17:34.511846571Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=174296i num_reverse_index_keys=174296i
[2025-03-19T16:17:34.550071160Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=838i out_datagrams_delta=8i rcvbuf_errors_delta=838i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=379031i rcvbuf_errors=379031i sndbuf_errors=0i rx_bytes_delta=1105694i rx_packets_delta=894i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10615i tx_packets_delta=21i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:34Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:35.521602279Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=175006i num_reverse_index_keys=175006i
[2025-03-19T16:17:36.527950306Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=176212i num_reverse_index_keys=176212i
[2025-03-19T16:17:36.550521042Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=896i out_datagrams_delta=0i rcvbuf_errors_delta=896i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=379927i rcvbuf_errors=379927i sndbuf_errors=0i rx_bytes_delta=1175297i rx_packets_delta=944i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:37.530071422Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=177311i num_reverse_index_keys=177311i
[SENTINEL.SH] [2025-03-19T16:17:37Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:38.050846886Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4973084672i cached_bytes=133723570176i free_percent=29.03220225665038 used_bytes=61243150336i avail_percent=77.3290855424952 buffers_percent=1.8409303990779002 cached_percent=49.50162759067241 swap_free_percent=0
[2025-03-19T16:17:38.075058690Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.13 average_load_five_minutes=15.9 average_load_fifteen_minutes=12.71 total_num_threads=2306i
[2025-03-19T16:17:38.075442205Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=85846i reads_merged=22795i sectors_read=18523296i time_reading_ms=42848i writes_completed=3454i writes_merged=49826i sectors_written=426096i time_writing_ms=403i io_in_progress=23i time_io_ms=19992i time_io_weighted_ms=43249i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:38.535750499Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=178039i num_reverse_index_keys=178039i
[2025-03-19T16:17:38.575730104Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=974i out_datagrams_delta=0i rcvbuf_errors_delta=974i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=380901i rcvbuf_errors=380901i sndbuf_errors=0i rx_bytes_delta=1249514i rx_packets_delta=1006i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:39.539195239Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=179448i num_reverse_index_keys=179448i
[2025-03-19T16:17:40.541299938Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=180479i num_reverse_index_keys=180479i
[SENTINEL.SH] [2025-03-19T16:17:40Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:41.076221998Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1116i out_datagrams_delta=0i rcvbuf_errors_delta=1116i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=382017i rcvbuf_errors=382017i sndbuf_errors=0i rx_bytes_delta=1453199i rx_packets_delta=1164i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:41.142250451Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=8263994i entries_reused=0i
[2025-03-19T16:17:41.142264431Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=139164736i bg_waiting_percent=67.19203186035156 bg_throttling_wait_percent=57.41425704956055 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=1226i max_in_bin_disk=23756i count_from_bins_disk=139164736i median_from_bins_disk=17883i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=8263994i deletes=0i active_threads=2i items=0i keys=0i ms_per_age=400i buckets_scanned=222986i flush_scan_us=74i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=4619i disk_index_failed_resizes=1278i disk_index_max_size=44714i disk_index_new_file_us=132422i disk_index_resize_us=22846348i disk_index_flush_file_us=0i disk_index_index_file_size=12052904304i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6497991i disk_index_flush_mmap_us=811181i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:41.550268583Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=181247i num_reverse_index_keys=181247i
[2025-03-19T16:17:42.553647147Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=182436i num_reverse_index_keys=182436i
[2025-03-19T16:17:43.076553485Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4973174784i cached_bytes=133718642688i free_percent=29.02391745425611 used_bytes=61640339456i avail_percent=77.18205456003318 buffers_percent=1.840963756627804 cached_percent=49.499803539193586 swap_free_percent=0
[SENTINEL.SH] [2025-03-19T16:17:43Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:43.848244367Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=90951i reads_merged=12276i sectors_read=18547920i time_reading_ms=44916i writes_completed=10062i writes_merged=75036i sectors_written=680508i time_writing_ms=1279i io_in_progress=11i time_io_ms=19888i time_io_weighted_ms=46196i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:43.848260477Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=183400i num_reverse_index_keys=183400i
[2025-03-19T16:17:43.848263107Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=1212i out_datagrams_delta=8i rcvbuf_errors_delta=1212i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=383229i rcvbuf_errors=383229i sndbuf_errors=0i rx_bytes_delta=1601240i rx_packets_delta=1290i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=9856i tx_packets_delta=18i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:44.559027797Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=184202i num_reverse_index_keys=184202i
[2025-03-19T16:17:45.562251085Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=185378i num_reverse_index_keys=185378i
[2025-03-19T16:17:45.577641032Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=897i out_datagrams_delta=0i rcvbuf_errors_delta=897i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=384126i rcvbuf_errors=384126i sndbuf_errors=0i rx_bytes_delta=1180325i rx_packets_delta=940i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=132i tx_packets_delta=2i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:46.563370934Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=186264i num_reverse_index_keys=186264i
[SENTINEL.SH] [2025-03-19T16:17:46Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:47.565526569Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=187194i num_reverse_index_keys=187194i
[2025-03-19T16:17:48.078221342Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1260i out_datagrams_delta=0i rcvbuf_errors_delta=1260i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=385386i rcvbuf_errors=385386i sndbuf_errors=0i rx_bytes_delta=1659580i rx_packets_delta=1326i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:48.078279971Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4973244416i cached_bytes=133743579136i free_percent=29.06076086812478 used_bytes=61886242816i avail_percent=77.0910263551027 buffers_percent=1.8409895329163661 cached_percent=49.50903448300331 swap_free_percent=0
[2025-03-19T16:17:48.099477166Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.37 average_load_five_minutes=16 average_load_fifteen_minutes=12.78 total_num_threads=2310i
[2025-03-19T16:17:48.099925830Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=83166i reads_merged=7385i sectors_read=18028096i time_reading_ms=42022i writes_completed=1536i writes_merged=10571i sectors_written=96704i time_writing_ms=112i io_in_progress=8i time_io_ms=19888i time_io_weighted_ms=42136i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:48.568841830Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=188370i num_reverse_index_keys=188370i
[2025-03-19T16:17:49.570912288Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=189492i num_reverse_index_keys=189492i
[SENTINEL.SH] [2025-03-19T16:17:49Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:50.100381578Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=925i out_datagrams_delta=0i rcvbuf_errors_delta=925i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=386311i rcvbuf_errors=386311i sndbuf_errors=0i rx_bytes_delta=1208157i rx_packets_delta=972i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:50.575900267Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=190214i num_reverse_index_keys=190214i
[2025-03-19T16:17:51.143835504Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=7880653i entries_reused=0i
[2025-03-19T16:17:51.143853834Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=0i flush_should_evict_us=0i count_in_mem=0i count=147045389i bg_waiting_percent=66.73332214355469 bg_throttling_wait_percent=55.551944732666016 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=0i count_from_bins_mem=0i median_from_bins_mem=0i min_in_bin_disk=1295i max_in_bin_disk=24336i count_from_bins_disk=147045389i median_from_bins_disk=18399i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=0i entry_mem_us=0i load_disk_found_count=0i load_disk_found_us=0i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=0i entry_missing_us=0i failed_to_evict=0i updates_in_mem=0i get_range_us=0i inserts=7880653i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=219013i flush_scan_us=80i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=4417i disk_index_failed_resizes=1195i disk_index_max_size=44800i disk_index_new_file_us=133272i disk_index_resize_us=23255352i disk_index_flush_file_us=0i disk_index_index_file_size=12786619776i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=6803611i disk_index_flush_mmap_us=798799i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:17:51.584781393Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=191372i num_reverse_index_keys=191372i
[2025-03-19T16:17:52.585464875Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=192308i num_reverse_index_keys=192308i
[2025-03-19T16:17:52.600961859Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1229i out_datagrams_delta=0i rcvbuf_errors_delta=1229i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=387540i rcvbuf_errors=387540i sndbuf_errors=0i rx_bytes_delta=1626545i rx_packets_delta=1308i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:52Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:53.101111547Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4972097536i cached_bytes=133755195392i free_percent=28.906909782959705 used_bytes=62663655424i avail_percent=76.80324470707416 buffers_percent=1.8405649822812276 cached_percent=49.51333457443636 swap_free_percent=0
[2025-03-19T16:17:53.853881570Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=92175i reads_merged=67224i sectors_read=18459184i time_reading_ms=44883i writes_completed=16455i writes_merged=199659i sectors_written=1728428i time_writing_ms=1868i io_in_progress=12i time_io_ms=19952i time_io_weighted_ms=46751i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:53.853894880Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=192969i num_reverse_index_keys=192969i
[2025-03-19T16:17:54.594463413Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=194111i num_reverse_index_keys=194111i
[2025-03-19T16:17:54.602101037Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=907i out_datagrams_delta=8i rcvbuf_errors_delta=907i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=388447i rcvbuf_errors=388447i sndbuf_errors=0i rx_bytes_delta=1188068i rx_packets_delta=970i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=10613i tx_packets_delta=21i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:55.598282293Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=194922i num_reverse_index_keys=194922i
[SENTINEL.SH] [2025-03-19T16:17:55Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:56.602573807Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=889i out_datagrams_delta=0i rcvbuf_errors_delta=889i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=389336i rcvbuf_errors=389336i sndbuf_errors=0i rx_bytes_delta=1171362i rx_packets_delta=938i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:17:56.603909098Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=195563i num_reverse_index_keys=195563i
[2025-03-19T16:17:57.608691247Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=196371i num_reverse_index_keys=196371i
[2025-03-19T16:17:58.102794662Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4972728320i cached_bytes=133694214144i free_percent=28.709649911604007 used_bytes=63560982528i avail_percent=76.47107325763737 buffers_percent=1.8407984851305539 cached_percent=49.49076061066514 swap_free_percent=0
[2025-03-19T16:17:58.127147694Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=64i cpu0_freq_mhz=3799i average_load_one_minute=17.9 average_load_five_minutes=16.16 average_load_fifteen_minutes=12.86 total_num_threads=2302i
[2025-03-19T16:17:58.127527029Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=67586i reads_merged=40702i sectors_read=14362320i time_reading_ms=33021i writes_completed=5874i writes_merged=93565i sectors_written=795320i time_writing_ms=598i io_in_progress=7i time_io_ms=19652i time_io_weighted_ms=33618i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:17:58.611932118Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=196858i num_reverse_index_keys=196858i
[2025-03-19T16:17:58.627777848Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=967i out_datagrams_delta=0i rcvbuf_errors_delta=967i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=390303i rcvbuf_errors=390303i sndbuf_errors=0i rx_bytes_delta=1258961i rx_packets_delta=1005i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[SENTINEL.SH] [2025-03-19T16:17:58Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:17:59.614314372Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=197556i num_reverse_index_keys=197556i
[2025-03-19T16:18:00.619626536Z INFO solana_metrics::metrics] datapoint: program_id_index_stats num_secondary_keys=1i num_inner_keys=197952i num_reverse_index_keys=197952i
[2025-03-19T16:18:00.628198017Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=907i out_datagrams_delta=0i rcvbuf_errors_delta=907i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=391210i rcvbuf_errors=391210i sndbuf_errors=0i rx_bytes_delta=1199220i rx_packets_delta=965i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=194i tx_packets_delta=3i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:18:00.641982726Z INFO solana_accounts_db::accounts_db] rent_collector: RentCollector { epoch: 852, epoch_schedule: EpochSchedule { slots_per_epoch: 432000, leader_schedule_slot_offset: 432000, warmup: false, first_normal_epoch: 0, first_normal_slot: 0 }, slots_per_year: 78892314.984, rent: Rent { lamports_per_byte_year: 3480, exemption_threshold: 2.0, burn_percent: 100 } }
[2025-03-19T16:18:01.144239049Z INFO solana_metrics::metrics] datapoint: accounts_index_startup entries_created=6236626i entries_reused=0i
[2025-03-19T16:18:01.144253409Z INFO solana_metrics::metrics] datapoint: accounts_index_startup estimate_mem_bytes=723504i flush_should_evict_us=0i count_in_mem=15073i count=153282015i bg_waiting_percent=76.54609680175781 bg_throttling_wait_percent=65.53219604492188 slot_list_len=0i ref_count=0i slot_list_cached=0i min_in_bin_mem=0i max_in_bin_mem=15i count_from_bins_mem=15046i median_from_bins_mem=1i min_in_bin_disk=18214i max_in_bin_disk=25537i count_from_bins_disk=153282015i median_from_bins_disk=18706i gets_from_mem=0i get_mem_us=0i gets_missing=0i get_missing_us=0i entries_from_mem=26675i entry_mem_us=3i load_disk_found_count=15073i load_disk_found_us=794i load_disk_missing_count=0i load_disk_missing_us=0i entries_missing=15073i entry_missing_us=32i failed_to_evict=0i updates_in_mem=41748i get_range_us=0i inserts=6236626i deletes=0i active_threads=1i items=0i keys=0i ms_per_age=400i buckets_scanned=220470i flush_scan_us=25i flush_update_us=0i flush_grow_us=0i flush_evict_us=0i disk_index_resizes=3072i disk_index_failed_resizes=830i disk_index_max_size=48298i disk_index_new_file_us=84539i disk_index_resize_us=16079065i disk_index_flush_file_us=0i disk_index_index_file_size=13354652304i index_exceptional_entry=0i disk_index_data_file_size=0i disk_index_data_file_count=0i disk_index_find_index_entry_mut_us=4456678i disk_index_flush_mmap_us=442764i disk_data_resizes=0i disk_data_max_size=0i disk_data_new_file_us=0i disk_data_resize_us=0i disk_data_flush_file_us=0i disk_data_flush_mmap_us=0i flush_entries_updated_on_disk=0i flush_entries_evicted_from_mem=0i
[2025-03-19T16:18:01.539446707Z INFO solana_accounts_db::accounts_db] accounts data len: 268104871589
[2025-03-19T16:18:01.539486457Z INFO solana_accounts_db::accounts_db] insert all zero slots to clean at startup 1308
[2025-03-19T16:18:01.586661382Z INFO solana_metrics::metrics] datapoint: generate_index overall_us=150435413i total_us=149476459i scan_stores_us=1391412i insertion_time_us=1521284i min_bin_size_in_mem=0i max_bin_size_in_mem=15i storage_size_storages_us=22469i index_flush_us=456107i total_rent_paying=0i amount_to_top_off_rent=0i total_items_including_duplicates=153349555i total_items_in_mem=25522i accounts_data_len_dedup_time_us=274922i total_duplicate_slot_keys=93062i total_num_unique_duplicate_keys=25522i num_duplicate_accounts=67540i populate_duplicate_keys_us=75159i total_slots=201498i slots_to_clean=0i copy_data_us=1312581i par_duplicates_lt_hash_us=0i num_zero_lamport_single_refs=71600i visit_zero_lamports_us=91165i all_accounts_are_zero_lamports_slots=1308i
[2025-03-19T16:18:01.588740273Z INFO solana_accounts_db::accounts_index] secondary index: ProgramId
[2025-03-19T16:18:01.588747423Z INFO solana_accounts_db::secondary_index] owner: AddressLookupTab1e1111111111111111111111111, accounts: 197969
[2025-03-19T16:18:01.588783713Z INFO solana_metrics::metrics] datapoint: reconstruct_accountsdb_from_fields() accountsdb-notify-at-start-us=150437586i
[SENTINEL.SH] [2025-03-19T16:18:01Z] [WAIT] Health check failed. Retrying in 3s.
[2025-03-19T16:18:02.224741661Z INFO solana_runtime::bank] Loading Stakes took: took 631ms
[2025-03-19T16:18:02.244352100Z INFO solana_accounts_db::accounts_db] Background account hasher has started
[2025-03-19T16:18:02.245756970Z INFO solana_metrics::metrics] datapoint: bank-new-from-fields accounts_data_len-from-snapshot=268105451833i accounts_data_len-from-generate_index=268104871589i stakes_accounts_load_duration_us=631835i
[2025-03-19T16:18:02.253718989Z INFO solana_runtime::serde_snapshot] rent_collector: RentCollector { epoch: 852, epoch_schedule: EpochSchedule { slots_per_epoch: 8192, leader_schedule_slot_offset: 8192, warmup: false, first_normal_epoch: 0, first_normal_slot: 0 }, slots_per_year: 78892314.984, rent: Rent { lamports_per_byte_year: 3480, exemption_threshold: 2.0, burn_percent: 50 } }
[2025-03-19T16:18:02.253775649Z INFO solana_runtime::snapshot_bank_utils] Rebuilding status cache from /snapshots/snapshots/tmp-snapshot-archive-oYufwS/snapshots/status_cache
[2025-03-19T16:18:02.262679695Z INFO solana_runtime::snapshot_bank_utils] Rebuilt bank for slot: 368429552
[2025-03-19T16:18:02.262775614Z INFO solana_runtime::snapshot_bank_utils] rebuild bank from snapshots took 152.1s
[2025-03-19T16:18:02.262782664Z INFO solana_runtime::bank] Verifying accounts...
[2025-03-19T16:18:02.274174076Z INFO solana_runtime::bank] Verifying accounts, in background? true, verify kind: Merkle
[2025-03-19T16:18:02.274213935Z INFO solana_runtime::bank] Verifying accounts... In background.
[2025-03-19T16:18:02.274216246Z INFO solana_runtime::bank] Cleaning... Skipped.
[2025-03-19T16:18:02.274217406Z INFO solana_runtime::bank] Shrinking... Skipped.
[2025-03-19T16:18:02.274218606Z INFO solana_runtime::bank] Verifying bank...
[2025-03-19T16:18:02.274311004Z INFO solana_runtime::bank] Initial background accounts hash verification has started
[2025-03-19T16:18:02.274547351Z INFO solana_accounts_db::accounts_db] skipped rewrite hashes 368429552 0
[2025-03-19T16:18:02.274646509Z INFO solana_runtime::bank] bank frozen: 368429552 hash: A5WxBQKtZvfmNt8zQcYswbpum1Vekwaj1vmZ3KK4bAjr accounts_delta: GXMLqu2jPkegZ6HY9TeiNzik92xkuzm5CxAo36FDPpcW signature_count: 31 last_blockhash: 31kX6RRGCxYDAdfiW8b1z59d13cEfnR7J3WNB3bCLLeL capitalization: 1280562325418914242, stats: BankHashStats { num_updated_accounts: 79, num_removed_accounts: 0, num_lamports_stored: 42877118734840345, total_data_len: 243550, num_executable_accounts: 0 }
[2025-03-19T16:18:02.274656189Z INFO solana_runtime::bank] Verifying bank... Done.
[2025-03-19T16:18:02.274655959Z INFO solana_metrics::metrics] datapoint: bank-hash_internal_state slot=368429552i total_us=416i accounts_delta_hash_us=414i
[2025-03-19T16:18:02.274661599Z INFO solana_metrics::metrics] datapoint: verify_snapshot_bank clean_us=1i shrink_us=1i verify_accounts_us=11433i verify_bank_us=436i
[2025-03-19T16:18:02.275369230Z INFO solana_metrics::metrics] datapoint: bank_from_snapshot_archives untar_full_snapshot_archive_us=221964509i untar_incremental_snapshot_archive_us=890027i rebuild_bank_us=152053119i verify_bank_us=11875i
[2025-03-19T16:18:02.286023942Z INFO solana_metrics::metrics] datapoint: accounts_db_active hash=1i
[2025-03-19T16:18:02.286160320Z INFO solana_metrics::metrics] datapoint: accounts_db_active hash_scan=1i
[2025-03-19T16:18:02.407666664Z INFO solana_entry::poh] Running 4000000 hashes...
[2025-03-19T16:18:02.648373886Z INFO solana_core::validator] PoH speed check: computed hashes per second 16618981, target hashes per second 10000000
[2025-03-19T16:18:02.648556334Z INFO solana_core::validator] shred version: 2405, hard forks: HardForks { hard_forks: [(128373320, 1), (128669404, 1), (162574136, 1), (276140931, 1), (278340533, 1), (293327999, 1), (308671012, 1), (308678163, 2), (308687771, 1)] }
[2025-03-19T16:18:02.648956358Z INFO solana_core::validator] Blockstore contains data from slot 368150532 to 368418744, the latest hard fork is 308687771
[2025-03-19T16:18:02.648959948Z INFO solana_core::validator] Skipping the blockstore check for shreds with incorrect version
[2025-03-19T16:18:02.648975048Z INFO solana_core::validator] ContactInfo { pubkey: BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar, wallclock: 1742401082648, outset: 1742400707151562, shred_version: 2405, version: 2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave), addrs: [162.19.43.1], sockets: [SocketEntry { key: 0, index: 0, offset: 25000 }, SocketEntry { key: 10, index: 0, offset: 1 }, SocketEntry { key: 11, index: 0, offset: 1 }, SocketEntry { key: 5, index: 0, offset: 1 }, SocketEntry { key: 6, index: 0, offset: 1 }, SocketEntry { key: 9, index: 0, offset: 1 }, SocketEntry { key: 12, index: 0, offset: 1 }, SocketEntry { key: 8, index: 0, offset: 3 }, SocketEntry { key: 7, index: 0, offset: 1 }, SocketEntry { key: 4, index: 0, offset: 2 }, SocketEntry { key: 1, index: 0, offset: 1 }], extensions: [], cache: [162.19.43.1:25000, 162.19.43.1:25013, 0.0.0.0:0, 0.0.0.0:0, 162.19.43.1:25012, 162.19.43.1:25003, 162.19.43.1:25004, 162.19.43.1:25010, 162.19.43.1:25009, 162.19.43.1:25005, 162.19.43.1:25001, 162.19.43.1:25002, 162.19.43.1:25006] }
[2025-03-19T16:18:02.648995968Z INFO solana_core::validator] local gossip address: 0.0.0.0:25000
[2025-03-19T16:18:02.648999617Z INFO solana_core::validator] local broadcast address: 0.0.0.0:25014
[2025-03-19T16:18:02.649002337Z INFO solana_core::validator] local repair address: 0.0.0.0:25008
[2025-03-19T16:18:02.649006387Z INFO solana_core::validator] local retransmit address: 0.0.0.0:25007
[2025-03-19T16:18:02.651150767Z INFO solana_gossip::cluster_info] Loaded contact info for 174 nodes from /ledger/contact-info.bin
[2025-03-19T16:18:02.651381484Z INFO solana_core::validator] Using: block-verification-method: unified-scheduler, block-production-method: central-scheduler, transaction-structure: sdk
[2025-03-19T16:18:02.651435754Z INFO solana_core::validator] Disabled banking trace
[2025-03-19T16:18:02.651464193Z INFO solana_core::snapshot_packager_service] SnapshotPackagerService has started
[2025-03-19T16:18:02.651496473Z INFO solana_runtime::accounts_background_service] AccountsBackgroundService has started
[2025-03-19T16:18:02.651515312Z INFO solana_core::accounts_hash_verifier] AccountsHashVerifier has started
[2025-03-19T16:18:02.651517322Z INFO solana_runtime::bank_forks] Installed new scheduler_pool into bank_forks: SchedulerPool { scheduler_inners: Mutex { data: [], poisoned: false, .. }, trashed_scheduler_inners: Mutex { data: [], poisoned: false, .. }, timeout_listeners: Mutex { data: [], poisoned: false, .. }, handler_count: 4, common_handler_context: CommonHandlerContext { log_messages_bytes_limit: None, transaction_status_sender: None, replay_vote_sender: Some(Sender { .. }), prioritization_fee_cache: PrioritizationFeeCache { cache: RwLock { data: {}, poisoned: false, .. }, service_thread: Some(JoinHandle { .. }), sender: Sender { .. }, metrics: PrioritizationFeeCacheMetrics { successful_transaction_update_count: 0, purged_duplicated_bank_count: 0, total_update_elapsed_us: 0, total_cache_lock_elapsed_us: 0, total_entry_update_elapsed_us: 0, total_block_finalize_elapsed_us: 0 } } }, banking_stage_handler_context: Mutex { data: None, poisoned: false, .. }, weak_self: (Weak), next_scheduler_id: 0, max_usage_queue_count: 262144, _phantom: PhantomData<solana_unified_scheduler_pool::DefaultTaskHandler> }
[2025-03-19T16:18:02.651608891Z INFO solana_metrics::metrics] datapoint: accounts_db_active flush=1i
[2025-03-19T16:18:02.651616201Z INFO solana_metrics::metrics] datapoint: accounts_db-flush_accounts_cache total_new_cleaned_roots=0i num_cleaned_roots_flushed=0i total_new_excess_roots=0i num_excess_roots_flushed=0i excess_slot_count=0i unflushable_unrooted_slot_count=0i flush_roots_elapsed=9i account_bytes_saved=0i num_accounts_saved=0i store_accounts_total_us=0i update_index_us=0i store_accounts_elapsed_us=0i handle_reclaims_elapsed_us=0i
[2025-03-19T16:18:02.651619461Z INFO solana_metrics::metrics] datapoint: accounts_db_active flush=0i
[2025-03-19T16:18:02.651620871Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean=1i
[2025-03-19T16:18:02.663046142Z INFO solana_accounts_db::accounts_db] total_stores: 201498, newest_slot: 368429552, oldest_slot: 355536592
[2025-03-19T16:18:02.663081262Z INFO solana_metrics::metrics] datapoint: accounts_db-stores total_count=201498i total_bytes=289542776728i total_alive_bytes=579085553456i total_alive_ratio=2
[2025-03-19T16:18:02.663087682Z INFO solana_metrics::metrics] datapoint: accounts_db-perf-stats delta_hash_num=1i delta_hash_scan_us=321i delta_hash_accumulate_us=91i skipped_rewrites_num=0i
[2025-03-19T16:18:02.663089692Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean_construct_candidates=1i
[2025-03-19T16:18:02.729075647Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean_construct_candidates=0i
[2025-03-19T16:18:02.729113596Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean_scan_candidates=1i
[2025-03-19T16:18:02.753199232Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean_scan_candidates=0i
[2025-03-19T16:18:02.753243461Z INFO solana_metrics::metrics] datapoint: accounts_db_active clean_old_accounts=1i
[2025-03-19T16:18:03.128734923Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=1191i out_datagrams_delta=0i rcvbuf_errors_delta=1191i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=392401i rcvbuf_errors=392401i sndbuf_errors=0i rx_bytes_delta=1577278i rx_packets_delta=1266i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2025-03-19T16:18:03.542338727Z INFO solana_connection_cache::connection_cache] Creating ConnectionCache connection_cache_tpu_quic, pool size: 1
[2025-03-19T16:18:03.542437296Z INFO solana_connection_cache::connection_cache] Creating ConnectionCache connection_cache_vote_udp, pool size: 1
[2025-03-19T16:18:03.542600693Z INFO solana_rpc::rpc_service] rpc bound to 0.0.0.0:8899
[2025-03-19T16:18:03.542604863Z INFO solana_rpc::rpc_service] rpc configuration: JsonRpcConfig { enable_rpc_transaction_history: false, enable_extended_tx_metadata_storage: false, faucet_addr: None, health_check_slot_distance: 128, skip_preflight_health_check: false, rpc_bigtable_config: None, max_multiple_accounts: Some(100), account_indexes: AccountSecondaryIndexes { keys: Some(AccountSecondaryIndexesIncludeExclude { exclude: false, keys: {AddressLookupTab1e1111111111111111111111111} }), indexes: {SplTokenMint, ProgramId, SplTokenOwner} }, rpc_threads: 16, rpc_blocking_threads: 4, rpc_niceness_adj: 0, full_api: false, rpc_scan_and_fix_roots: false, max_request_body_size: Some(51200), disable_health_check: false }
[2025-03-19T16:18:03.546580788Z INFO solana_net_utils::ip_echo_server] bound to 0.0.0.0:25000
[2025-03-19T16:18:03.546913094Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:53298
[2025-03-19T16:18:03.546947113Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:46174
[2025-03-19T16:18:03.546950633Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:50118
[2025-03-19T16:18:03.546954203Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:48442
[2025-03-19T16:18:03.546956783Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:58400
[2025-03-19T16:18:03.546959593Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:57236
[2025-03-19T16:18:03.546962403Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:34174
[2025-03-19T16:18:03.546965933Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:40234
[2025-03-19T16:18:03.546969813Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:36628
[2025-03-19T16:18:03.546972183Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:37536
[2025-03-19T16:18:03.546974393Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:38532
[2025-03-19T16:18:03.546977473Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:52398
[2025-03-19T16:18:03.546979483Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:36430
[2025-03-19T16:18:03.546981553Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:53174
[2025-03-19T16:18:03.546983583Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:45648
[2025-03-19T16:18:03.546985763Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:56230
[2025-03-19T16:18:03.546988283Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:55120
[2025-03-19T16:18:03.546990483Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:60208
[2025-03-19T16:18:03.546992543Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:58248
[2025-03-19T16:18:03.546994593Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:58622
[2025-03-19T16:18:03.546996593Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:38730
[2025-03-19T16:18:03.546998573Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:40654
[2025-03-19T16:18:03.547000922Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:47550
[2025-03-19T16:18:03.547002973Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:51602
[2025-03-19T16:18:03.547005073Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:58142
[2025-03-19T16:18:03.547007132Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:43286
[2025-03-19T16:18:03.547009462Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:51676
[2025-03-19T16:18:03.547011562Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:33768
[2025-03-19T16:18:03.547013892Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:45770
[2025-03-19T16:18:03.547184260Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:38304
[2025-03-19T16:18:03.547512235Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547687023Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:51928
[2025-03-19T16:18:03.547690733Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:54642
[2025-03-19T16:18:03.547711353Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547737252Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547738902Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547742512Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:35920
[2025-03-19T16:18:03.547759912Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547764212Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547766062Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:41034
[2025-03-19T16:18:03.547767982Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:40542
[2025-03-19T16:18:03.547769442Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:34472
[2025-03-19T16:18:03.547770852Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:35858
[2025-03-19T16:18:03.547772322Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:37654
[2025-03-19T16:18:03.547773712Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:57918
[2025-03-19T16:18:03.547775162Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:49624
[2025-03-19T16:18:03.547776522Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:48054
[2025-03-19T16:18:03.547777962Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:53398
[2025-03-19T16:18:03.547779292Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:60300
[2025-03-19T16:18:03.547779532Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547780712Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:34514
[2025-03-19T16:18:03.547782812Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:57994
[2025-03-19T16:18:03.547784142Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:37432
[2025-03-19T16:18:03.547785502Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:39684
[2025-03-19T16:18:03.547786812Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:35164
[2025-03-19T16:18:03.547788172Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:49580
[2025-03-19T16:18:03.547789712Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:41752
[2025-03-19T16:18:03.547791002Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:40226
[2025-03-19T16:18:03.547792302Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:55754
[2025-03-19T16:18:03.547793662Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:57768
[2025-03-19T16:18:03.547794971Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:55896
[2025-03-19T16:18:03.547796482Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:35974
[2025-03-19T16:18:03.547797762Z INFO solana_net_utils::ip_echo_server] connection from 100.64.5.1:34054
[2025-03-19T16:18:03.547801502Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547817871Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547819711Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547836891Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547837471Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547855381Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547856481Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547873301Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547880000Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547892850Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547902550Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547911000Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547921570Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547928600Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547939800Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547945840Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547962229Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547968349Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547980999Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547985859Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.547998529Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548004169Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548018319Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548020848Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548042008Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548053308Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548067308Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548073628Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548085567Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548093367Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548107667Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548110817Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548125197Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548128337Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548146347Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548146607Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548164616Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548168906Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548184626Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548188456Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548203296Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548206246Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548222266Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548223246Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548240925Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548241205Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548260715Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548263055Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.548283305Z INFO solana_net_utils::ip_echo_server] session failed: Custom { kind: UnexpectedEof, error: "early eof" }
[2025-03-19T16:18:03.551926614Z INFO solana_ledger::blockstore_metric_report_service] BlockstoreMetricReportService has started
[2025-03-19T16:18:03.565555995Z INFO solana_ledger::blockstore_processor] Processing ledger from slot 368429552...
[2025-03-19T16:18:03.566337585Z INFO solana_ledger::blockstore] Marking slot 368429552 and any full children slots as connected
[2025-03-19T16:18:03.566371024Z INFO solana_ledger::blockstore_processor] ledger holds data through slot 368429552
[2025-03-19T16:18:03.567436279Z INFO solana_ledger::blockstore_processor] load_frozen_forks() latest root from blockstore: 368429552, max_root: 368429552
[2025-03-19T16:18:03.569894275Z INFO solana_ledger::blockstore_processor] ledger processing timing: ExecuteTimings { metrics: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], details: ExecuteDetailsTimings { serialize_us: 0, create_vm_us: 0, execute_us: 0, deserialize_us: 0, get_or_create_executor_us: 0, changed_account_count: 0, total_account_count: 0, create_executor_register_syscalls_us: 0, create_executor_load_elf_us: 0, create_executor_verify_code_us: 0, create_executor_jit_compile_us: 0, per_program_timings: {} }, execute_accessories: ExecuteAccessoryTimings { feature_set_clone_us: 0, get_executors_us: 0, process_message_us: 0, process_instructions: ExecuteProcessInstructionTimings { total_us: 0, verify_caller_us: 0, process_executable_chain_us: 0, verify_callee_us: 0 } } }
[2025-03-19T16:18:03.569916814Z INFO solana_ledger::blockstore_processor] ledger processed in 4 ms, 322 µs and 790 ns. root slot is 368429552, 1 bank: 368429552
[2025-03-19T16:18:03.570125501Z ERROR solana_core::validator] Rebuilding a new tower from the latest vote account due to failed tower restore: IO Error: No such file or directory (os error 2)
[2025-03-19T16:18:03.570169011Z INFO solana_core::validator] Tower state: Tower { node_pubkey: BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar, threshold_depth: 8, threshold_size: 0.6666666666666666, vote_state: TowerVoteState { votes: [], root_slot: Some(368429552) }, last_vote: TowerSync(TowerSync { lockouts: [], root: None, hash: 11111111111111111111111111111111, timestamp: None, block_id: 11111111111111111111111111111111 }), last_vote_tx_blockhash: Uninitialized, last_timestamp: BlockTimestamp { slot: 0, timestamp: 0 }, stray_restored_slot: None, last_switch_threshold_check: None }
[2025-03-19T16:18:03.573158640Z INFO solana_streamer::streamer] streamer send Gossip hosts: count:10 [(132.145.143.18, SendStats { bytes: 396, count: 3 }), (5.161.254.100, SendStats { bytes: 264, count: 2 }), (208.115.193.194, SendStats { bytes: 132, count: 1 }), (47.76.150.188, SendStats { bytes: 132, count: 1 }), (204.16.244.203, SendStats { bytes: 132, count: 1 })]
thread 'solSvrfyShred07' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
[2025-03-19T16:18:03.616602617Z INFO solana_ledger::blockstore_cleanup_service] BlockstoreCleanupService has started with max ledger shreds=50000000
[2025-03-19T16:18:03.636781307Z INFO solana_poh::poh_recorder] reset poh from: 31kX6RRGCxYDAdfiW8b1z59d13cEfnR7J3WNB3bCLLeL,23579491495,368429552 to: 31kX6RRGCxYDAdfiW8b1z59d13cEfnR7J3WNB3bCLLeL,368429552
[2025-03-19T16:18:03.636823987Z INFO solana_core::replay_stage] BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar reset PoH to tick 23579491392 (within slot 368429552). I am not in the leader schedule yet
[2025-03-19T16:18:03.638748690Z INFO solana_quic_client::nonblocking::quic_client] Local endpoint is : UdpSocket { addr: 0.0.0.0:8000, fd: 26 }
[2025-03-19T16:18:03.640224399Z INFO solana_core::replay_stage] Frozen bank vote state slot Some(368145381)
is newer than our local vote state slot None,
adopting the bank vote state as our own.
Bank votes: [LandedVote { latency: 1, lockout: Lockout { slot: 368145351, confirmation_count: 31 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145352, confirmation_count: 30 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145353, confirmation_count: 29 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145354, confirmation_count: 28 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145355, confirmation_count: 27 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145356, confirmation_count: 26 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145357, confirmation_count: 25 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145358, confirmation_count: 24 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145359, confirmation_count: 23 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145360, confirmation_count: 22 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145361, confirmation_count: 21 } }, LandedVote { latency: 2, lockout: Lockout { slot: 368145362, confirmation_count: 20 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145363, confirmation_count: 19 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145364, confirmation_count: 18 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145365, confirmation_count: 17 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145366, confirmation_count: 16 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145367, confirmation_count: 15 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145368, confirmation_count: 14 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145369, confirmation_count: 13 } }, LandedVote { latency: 2, lockout: Lockout { slot: 368145370, confirmation_count: 12 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145371, confirmation_count: 11 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145372, confirmation_count: 10 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145373, confirmation_count: 9 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145374, confirmation_count: 8 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145375, confirmation_count: 7 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145376, confirmation_count: 6 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145377, confirmation_count: 5 } }, LandedVote { latency: 2, lockout: Lockout { slot: 368145378, confirmation_count: 4 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145379, confirmation_count: 3 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145380, confirmation_count: 2 } }, LandedVote { latency: 1, lockout: Lockout { slot: 368145381, confirmation_count: 1 } }], root: Some(368145350),
Local votes: [], root: Some(368429552)
[2025-03-19T16:18:03.640257209Z INFO solana_core::replay_stage] Local root is larger than on chain root,
overwrote bank root Some(368429552) and updated votes []
[2025-03-19T16:18:03.640716562Z INFO solana_streamer::nonblocking::quic] Start quic_streamer_tpu_vote quic server on [UdpSocket { addr: 0.0.0.0:25006, fd: 65 }]
[2025-03-19T16:18:03.646877857Z INFO solana_core::replay_stage] FwR3PbjS5iyqzLiLugrBqKSa5EKZ4vK9SKs7eQXtT59f slot_weight: 368429552 0.0% 0
[2025-03-19T16:18:03.646897637Z INFO solana_core::replay_stage] vote bank: None reset bank: 368429552
[2025-03-19T16:18:03.646915577Z INFO solana_poh::poh_recorder] reset poh from: 31kX6RRGCxYDAdfiW8b1z59d13cEfnR7J3WNB3bCLLeL,23579491393,368429552 to: 31kX6RRGCxYDAdfiW8b1z59d13cEfnR7J3WNB3bCLLeL,368429552
[2025-03-19T16:18:03.646927347Z INFO solana_core::replay_stage] BUxL4tfirArVhqZmjyzvQtBgL9DnV4UZBGHJtxXhW8ar reset PoH to tick 23579491392 (within slot 368429552). I am not in the leader schedule yet
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::join::join_context::{{closure}}
6: rayon::iter::plumbing::bridge_producer_consumer::helper
7: rayon_core::join::join_context::{{closure}}
8: rayon::iter::plumbing::bridge_producer_consumer::helper
9: rayon_core::job::StackJob<L,F,R>::run_inline
10: rayon_core::join::join_context::{{closure}}
11: rayon::iter::plumbing::bridge_producer_consumer::helper
12: rayon_core::join::join_context::{{closure}}
13: rayon::iter::plumbing::bridge_producer_consumer::helper
14: rayon_core::join::join_context::{{closure}}
15: rayon::iter::plumbing::bridge_producer_consumer::helper
16: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
17: rayon_core::registry::WorkerThread::wait_until_cold
18: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred00' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::job::StackJob<L,F,R>::run_inline
6: rayon_core::join::join_context::{{closure}}
7: rayon::iter::plumbing::bridge_producer_consumer::helper
8: rayon_core::job::StackJob<L,F,R>::run_inline
9: rayon_core::join::join_context::{{closure}}
10: rayon::iter::plumbing::bridge_producer_consumer::helper
11: rayon_core::job::StackJob<L,F,R>::run_inline
12: rayon_core::join::join_context::{{closure}}
13: rayon::iter::plumbing::bridge_producer_consumer::helper
14: rayon_core::join::join_context::{{closure}}
15: rayon::iter::plumbing::bridge_producer_consumer::helper
16: rayon_core::join::join_context::{{closure}}
17: rayon::iter::plumbing::bridge_producer_consumer::helper
18: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
19: rayon_core::registry::WorkerThread::wait_until_cold
20: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred06' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::join::join_context::{{closure}}
6: rayon::iter::plumbing::bridge_producer_consumer::helper
7: rayon_core::job::StackJob<L,F,R>::run_inline
8: rayon_core::join::join_context::{{closure}}
9: rayon::iter::plumbing::bridge_producer_consumer::helper
10: rayon_core::join::join_context::{{closure}}
11: rayon::iter::plumbing::bridge_producer_consumer::helper
12: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
13: rayon_core::registry::WorkerThread::wait_until_cold
14: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred02' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::join::join_context::{{closure}}
6: rayon::iter::plumbing::bridge_producer_consumer::helper
7: rayon_core::job::StackJob<L,F,R>::run_inline
8: rayon_core::join::join_context::{{closure}}
9: rayon::iter::plumbing::bridge_producer_consumer::helper
10: rayon_core::job::StackJob<L,F,R>::run_inline
11: rayon_core::join::join_context::{{closure}}
12: rayon::iter::plumbing::bridge_producer_consumer::helper
13: rayon_core::join::join_context::{{closure}}
14: rayon::iter::plumbing::bridge_producer_consumer::helper
15: rayon::iter::plumbing::bridge_producer_consumer::helper
16: rayon_core::join::join_context::{{closure}}
17: rayon::iter::plumbing::bridge_producer_consumer::helper
18: rayon_core::job::StackJob<L,F,R>::run_inline
19: rayon_core::join::join_context::{{closure}}
20: rayon::iter::plumbing::bridge_producer_consumer::helper
21: rayon_core::join::join_context::{{closure}}
22: rayon::iter::plumbing::bridge_producer_consumer::helper
23: rayon_core::join::join_context::{{closure}}
24: rayon::iter::plumbing::bridge_producer_consumer::helper
25: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
26: rayon_core::registry::WorkerThread::wait_until_cold
27: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred01' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::join::join_context::{{closure}}
6: rayon::iter::plumbing::bridge_producer_consumer::helper
7: rayon_core::join::join_context::{{closure}}
8: rayon::iter::plumbing::bridge_producer_consumer::helper
9: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
10: rayon_core::registry::WorkerThread::wait_until_cold
11: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred05' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::job::StackJob<L,F,R>::run_inline
6: rayon_core::join::join_context::{{closure}}
7: rayon::iter::plumbing::bridge_producer_consumer::helper
8: rayon_core::join::join_context::{{closure}}
9: rayon::iter::plumbing::bridge_producer_consumer::helper
10: rayon_core::join::join_context::{{closure}}
11: rayon::iter::plumbing::bridge_producer_consumer::helper
12: rayon_core::join::join_context::{{closure}}
13: rayon::iter::plumbing::bridge_producer_consumer::helper
14: rayon::iter::plumbing::bridge_producer_consumer::helper
15: rayon_core::job::StackJob<L,F,R>::run_inline
16: rayon_core::join::join_context::{{closure}}
17: rayon::iter::plumbing::bridge_producer_consumer::helper
18: rayon_core::join::join_context::{{closure}}
19: rayon::iter::plumbing::bridge_producer_consumer::helper
20: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
21: rayon_core::registry::WorkerThread::wait_until_cold
22: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred03' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::join::join_context::{{closure}}
6: rayon::iter::plumbing::bridge_producer_consumer::helper
7: rayon_core::job::StackJob<L,F,R>::run_inline
8: rayon_core::join::join_context::{{closure}}
9: rayon::iter::plumbing::bridge_producer_consumer::helper
10: rayon_core::join::join_context::{{closure}}
11: rayon::iter::plumbing::bridge_producer_consumer::helper
12: rayon_core::job::StackJob<L,F,R>::run_inline
13: rayon_core::join::join_context::{{closure}}
14: rayon::iter::plumbing::bridge_producer_consumer::helper
15: rayon::iter::plumbing::bridge_producer_consumer::helper
16: rayon_core::join::join_context::{{closure}}
17: rayon::iter::plumbing::bridge_producer_consumer::helper
18: rayon_core::join::join_context::{{closure}}
19: rayon::iter::plumbing::bridge_producer_consumer::helper
20: rayon_core::join::join_context::{{closure}}
21: rayon::iter::plumbing::bridge_producer_consumer::helper
22: rayon_core::join::join_context::{{closure}}
23: rayon::iter::plumbing::bridge_producer_consumer::helper
24: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
25: rayon_core::registry::WorkerThread::wait_until_cold
26: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'solSvrfyShred04' panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::option::expect_failed
3: solana_turbine::cluster_nodes::ClusterNodes<solana_turbine::retransmit_stage::RetransmitStage>::get_retransmit_parent
4: rayon::iter::plumbing::Producer::fold_with
5: rayon_core::job::StackJob<L,F,R>::run_inline
6: rayon_core::join::join_context::{{closure}}
7: rayon::iter::plumbing::bridge_producer_consumer::helper
8: rayon_core::job::StackJob<L,F,R>::run_inline
9: rayon_core::join::join_context::{{closure}}
10: rayon::iter::plumbing::bridge_producer_consumer::helper
11: rayon_core::join::join_context::{{closure}}
12: rayon::iter::plumbing::bridge_producer_consumer::helper
13: rayon_core::join::join_context::{{closure}}
14: rayon::iter::plumbing::bridge_producer_consumer::helper
15: rayon::iter::plumbing::bridge_producer_consumer::helper
16: rayon_core::join::join_context::{{closure}}
17: rayon::iter::plumbing::bridge_producer_consumer::helper
18: rayon_core::join::join_context::{{closure}}
19: rayon::iter::plumbing::bridge_producer_consumer::helper
20: rayon_core::join::join_context::{{closure}}
21: rayon::iter::plumbing::bridge_producer_consumer::helper
22: rayon_core::join::join_context::{{closure}}
23: rayon::iter::plumbing::bridge_producer_consumer::helper
24: rayon_core::join::join_context::{{closure}}
25: rayon::iter::plumbing::bridge_producer_consumer::helper
26: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
27: rayon_core::registry::WorkerThread::wait_until_cold
28: rayon_core::registry::ThreadBuilder::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2025-03-19T16:18:03.671822191Z INFO solana_streamer::nonblocking::quic] Start quic_streamer_tpu quic server on [UdpSocket { addr: 0.0.0.0:25009, fd: 54 }]
[2025-03-19T16:18:03.677273636Z INFO solana_quic_client::nonblocking::quic_client] Made connection to 147.28.165.119:8010 id 139691089668112 try_count 0, from connection cache warming?: true
[2025-03-19T16:18:03.691801864Z INFO solana_streamer::nonblocking::quic] Start quic_streamer_tpu_forwards quic server on [UdpSocket { addr: 0.0.0.0:25010, fd: 63 }]
[2025-03-19T16:18:03.705375126Z INFO agave_validator] Validator initialized
[2025-03-19T16:18:03.877405220Z INFO solana_quic_client::nonblocking::quic_client] Local endpoint is : UdpSocket { addr: 0.0.0.0:8001, fd: 39 }
[2025-03-19T16:18:03.913537169Z INFO solana_metrics::metrics] datapoint: memory-stats total=270139744256i swap_total=0i buffers_bytes=4972163072i cached_bytes=132965650432i free_percent=29.121297239938706 used_bytes=63273492480i avail_percent=76.5774959718484 buffers_percent=1.8405892423175214 cached_percent=49.22106178718896 swap_free_percent=0
[2025-03-19T16:18:03.913547339Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=96676i reads_merged=34508i sectors_read=5200536i time_reading_ms=15981i writes_completed=10497i writes_merged=134100i sectors_written=1156620i time_writing_ms=1194i io_in_progress=2i time_io_ms=15224i time_io_weighted_ms=17176i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=6i
[2025-03-19T16:18:03.913550759Z INFO solana_metrics::metrics] datapoint: process_blockstore_from_root total_time_us=4322i frozen_banks=1i slot=368429552i num_slots_processed=0i num_new_roots_found=0i forks=1i
[2025-03-19T16:18:03.913553359Z INFO solana_metrics::metrics] datapoint: shred_fetch index_overrun=0i shred_count=64i num_shreds_merkle_code=0i num_shreds_merkle_code_chained=0i num_shreds_merkle_data=0i num_shreds_merkle_data_chained=0i ping_count=0i ping_err_verify_count=0i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=64i bad_shred_type=0i shred_version_mismatch=0i bad_parent_offset=0i
[2025-03-19T16:18:03.913556389Z INFO solana_metrics::metrics] datapoint: Gossip streamer-send-sample_duration_ms=0i streamer-send-host_count=10i streamer-send-bytes_total=1716i streamer-send-pkt_count_total=13i streamer-send-host_bytes_min=132i streamer-send-host_bytes_max=396i streamer-send-host_bytes_mean=172i streamer-send-host_bytes_90pct=396i streamer-send-host_bytes_50pct=132i streamer-send-host_bytes_10pct=132i
[2025-03-19T16:18:03.913559499Z INFO solana_metrics::metrics] datapoint: tower-observed slot=368145381i root=368145350i
[2025-03-19T16:18:03.913561599Z INFO solana_metrics::metrics] datapoint: compute_bank_stats-best_slot computed_slot=368429552i overall_best_slot=368429552i overall_best_hash="A5WxBQKtZvfmNt8zQcYswbpum1Vekwaj1vmZ3KK4bAjr" elapsed=3i
[2025-03-19T16:18:03.913565759Z INFO solana_metrics::metrics] datapoint: bank_weight slot=368429552i fork_stake=0i fork_weight=0
[2025-03-19T16:18:03.913567249Z INFO solana_metrics::metrics] datapoint: blocks_produced num_blocks_on_fork=0i num_dropped_blocks_on_fork=0i
[2025-03-19T16:18:03.913569179Z INFO solana_metrics::metrics] datapoint: replay-loop-voting-stats generate_vote_us=0i update_commitment_cache_us=0i
[2025-03-19T16:18:03.913570919Z INFO solana_metrics::metrics] datapoint: replay-loop-timing-stats loop_count=1i total_elapsed_us=1742401083647000i collect_frozen_banks_elapsed_us=0i compute_bank_stats_elapsed_us=6675i select_vote_and_reset_forks_elapsed_us=2i start_leader_elapsed_us=2i reset_bank_elapsed_us=33i voting_elapsed_us=0i select_forks_elapsed_us=0i compute_slot_stats_elapsed_us=2i generate_new_bank_forks_elapsed_us=2888i replay_active_banks_elapsed_us=323i process_ancestor_hashes_duplicate_slots_elapsed_us=1i process_duplicate_confirmed_slots_elapsed_us=0i process_unfrozen_gossip_verified_vote_hashes_elapsed_us=1i process_popular_pruned_forks_elapsed_us=0i wait_receive_elapsed_us=534i heaviest_fork_failures_elapsed_us=0i bank_count=0i process_duplicate_slots_elapsed_us=1i repair_correct_slots_elapsed_us=0i retransmit_not_propagated_elapsed_us=0i generate_new_bank_forks_read_lock_us=0i generate_new_bank_forks_get_slots_since_us=1541i generate_new_bank_forks_loop_us=0i generate_new_bank_forks_write_lock_us=0i replay_blockstore_us=0i
[2025-03-19T16:18:03.913574599Z INFO solana_metrics::metrics] datapoint: shred_fetch_repair index_overrun=0i shred_count=2i num_shreds_merkle_code=0i num_shreds_merkle_code_chained=0i num_shreds_merkle_data=0i num_shreds_merkle_data_chained=0i ping_count=1i ping_err_verify_count=0i slot_bad_deserialize=0i index_bad_deserialize=0i index_out_of_bounds=0i slot_out_of_range=1i bad_shred_type=0i shred_version_mismatch=0i bad_parent_offset=0i
[2025-03-19T16:18:03.913577119Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solSvrfyShred07" one=1i message="panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key" location="turbine/src/cluster_nodes.rs:276:33" version="2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave)"
[2025-03-19T16:18:04.051222929Z INFO solana_accounts_db::accounts_db] remove_dead_slots_metadata: 855 dead slots
[2025-03-19T16:18:04.131025013Z INFO solana_quic_client::nonblocking::quic_client] Made connection to 147.28.198.57:8010 id 139691090402320 try_count 0, from connection cache warming?: true
[2025-03-19T16:18:04.154166532Z INFO solana_accounts_db::accounts_db] remove_dead_slots_metadata: 915 dead slots
[2025-03-19T16:18:04.348348609Z INFO solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 368430600, root: 368429552, slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: 6iWkC4DUGuZGUyuxDj1iQEv1f3QTFTLvRfmwJBE8uwTU, bank_status: Unprocessed })
[2025-03-19T16:18:04.448493010Z INFO solana_core::repair::cluster_slot_state_verifier] check_slot_agrees_with_cluster() slot: 368430598, root: 368429552, slot_state_update: DuplicateConfirmed(DuplicateConfirmedState { duplicate_confirmed_hash: 4a8caPSYrvnsw1moJ7WiSDKEfaAzVzmwpp1Tp8taYs37, bank_status: Unprocessed })
[2025-03-19T16:18:04.566423195Z INFO solana_streamer::streamer] streamer send Repair hosts: count:0 []
[2025-03-19T16:18:04.685430404Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solSvrfyShred00" one=1i message="panicked at turbine/src/cluster_nodes.rs:276:33:
no entry found for key" location="turbine/src/cluster_nodes.rs:276:33" version="2.2.3 (src:0b37b8fd; feat:3294202862, client:Agave)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment