stall-01.log

[4921] 2021/10/21 16:59:00.201527 [INF] Starting nats-server
[4921] 2021/10/21 16:59:00.245599 [INF]   Version:  2.6.2
[4921] 2021/10/21 16:59:00.245620 [INF]   Git:      [f7c3ac5]
[4921] 2021/10/21 16:59:00.245627 [DBG]   Go build: go1.16.9
[4921] 2021/10/21 16:59:00.245633 [INF]   Name:     nats-cluster-2
[4921] 2021/10/21 16:59:00.245647 [INF]   Node:     VpUo9efM
[4921] 2021/10/21 16:59:00.245658 [INF]   ID:       NC5BJX54OY2DLE5M2PW6NYAVGLZN34YUICMW3JNG5RZB74NNZU66L33J
[4921] 2021/10/21 16:59:00.245707 [INF] Using configuration file: nats2.conf
[4921] 2021/10/21 16:59:00.245767 [DBG] Created system account: "$SYS"
[4921] 2021/10/21 16:59:00.247275 [INF] Starting JetStream
[4921] 2021/10/21 16:59:00.247608 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[4921] 2021/10/21 16:59:00.247621 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[4921] 2021/10/21 16:59:00.247627 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[4921] 2021/10/21 16:59:00.247632 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[4921] 2021/10/21 16:59:00.247638 [INF]
[4921] 2021/10/21 16:59:00.247643 [INF]          https://docs.nats.io/jetstream
[4921] 2021/10/21 16:59:00.247648 [INF]
[4921] 2021/10/21 16:59:00.247654 [INF] ---------------- JETSTREAM ----------------
[4921] 2021/10/21 16:59:00.247670 [INF]   Max Memory:      1.00 GB
[4921] 2021/10/21 16:59:00.247679 [INF]   Max Storage:     5.00 GB
[4921] 2021/10/21 16:59:00.247684 [INF]   Store Directory: "data2/jetstream"
[4921] 2021/10/21 16:59:00.247690 [INF] -------------------------------------------
[4921] 2021/10/21 16:59:00.247839 [DBG]   Exports:
[4921] 2021/10/21 16:59:00.247849 [DBG]      $JS.API.>
[4921] 2021/10/21 16:59:00.247945 [DBG] Enabled JetStream for account "$G"
[4921] 2021/10/21 16:59:00.247957 [DBG]   Max Memory:      -1 B
[4921] 2021/10/21 16:59:00.247964 [DBG]   Max Storage:     -1 B
[4921] 2021/10/21 16:59:00.248063 [DBG] Recovering JetStream state for account "$G"
[4921] 2021/10/21 16:59:00.283041 [INF]   Restored 81 messages for stream "corejobs"
[4921] 2021/10/21 16:59:00.283115 [INF]   Recovering 6 consumers for stream - "corejobs"
[4921] 2021/10/21 16:59:00.317197 [DBG] JetStream state for account "$G" recovered
[4921] 2021/10/21 16:59:00.317207 [INF] Starting JetStream cluster
[4921] 2021/10/21 16:59:00.317209 [DBG] JetStream cluster checking for stable cluster name and peers
[4921] 2021/10/21 16:59:00.317211 [INF] Creating JetStream metadata controller
[4921] 2021/10/21 16:59:00.317529 [INF] JetStream cluster recovering state
[4921] 2021/10/21 16:59:00.317834 [DBG] RAFT [VpUo9efM - _meta_] Started
[4921] 2021/10/21 16:59:00.317916 [DBG] Starting metadata monitor
[4921] 2021/10/21 16:59:00.317961 [INF] Listening for client connections on 0.0.0.0:4224
[4921] 2021/10/21 16:59:00.317966 [DBG] Get non local IPs for "0.0.0.0"
[4921] 2021/10/21 16:59:00.318126 [DBG]   ip=192.168.86.197
[4921] 2021/10/21 16:59:00.318131 [DBG]   ip=2600:1700:9778:cd7f:56f0:767b:e7ea:4fb4
[4921] 2021/10/21 16:59:00.318177 [DBG]   ip=192.168.49.1
[4921] 2021/10/21 16:59:00.318206 [DBG]   ip=172.17.0.1
[4921] 2021/10/21 16:59:00.318226 [DBG]   ip=172.19.0.1
[4921] 2021/10/21 16:59:00.318234 [INF] Server is ready
[4921] 2021/10/21 16:59:00.318256 [INF] Cluster name is nats-cluster
[4921] 2021/10/21 16:59:00.318269 [INF] Listening for route connections on 0.0.0.0:6224
[4921] 2021/10/21 16:59:00.318370 [DBG] JetStream cluster creating raft group:&{Name:S-R3F-bSTo304i Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.320600 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Update peers from leader to map[EEUOgTxU:0xc0001bfe40 FFfB0Dc4:0xc0001bfe30 VpUo9efM:0xc0001bfe20]
[4921] 2021/10/21 16:59:00.320960 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Update peers from leader to map[EEUOgTxU:0xc0001bfe40 FFfB0Dc4:0xc0001bfe30 VpUo9efM:0xc0001bfe20]
[4921] 2021/10/21 16:59:00.321482 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Update peers from leader to map[EEUOgTxU:0xc0001bfe40 FFfB0Dc4:0xc0001bfe30 VpUo9efM:0xc0001bfe20]
[4921] 2021/10/21 16:59:00.323255 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Started
[4921] 2021/10/21 16:59:00.324673 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-t4uNCeJ4 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.328351 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc000986d90 FFfB0Dc4:0xc000986d80 VpUo9efM:0xc000986d70]
[4921] 2021/10/21 16:59:00.328397 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc000986d90 FFfB0Dc4:0xc000986d80 VpUo9efM:0xc000986d70]
[4921] 2021/10/21 16:59:00.328419 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Started
[4921] 2021/10/21 16:59:00.328426 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.328440 [DBG] Starting consumer monitor for '$G > corejobs > accessors-BranchDeletion-worker
[4921] 2021/10/21 16:59:00.328494 [DBG] Starting stream monitor for '$G > corejobs'
[4921] 2021/10/21 16:59:00.328555 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-hGUj1SUd Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.329094 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Started
[4921] 2021/10/21 16:59:00.329104 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.329115 [DBG] Starting consumer monitor for '$G > corejobs > chat-SendMessageNotification-worker
[4921] 2021/10/21 16:59:00.329152 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-LbOAptkd Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.330881 [DBG] Trying to connect to route on localhost:6222 (127.0.0.1:6222)
[4921] 2021/10/21 16:59:00.331089 [DBG] 127.0.0.1:6222 - rid:25 - Route connect msg sent
[4921] 2021/10/21 16:59:00.331144 [INF] 127.0.0.1:6222 - rid:25 - Route connection created
[4921] 2021/10/21 16:59:00.331208 [DBG] 127.0.0.1:6222 - rid:25 - Registering remote route "NAASKK6U7M5S45MJNXJJ4LDKJKJACVAQFWPM3B54XJONCIJ7O73XBVGO"
[4921] 2021/10/21 16:59:00.331242 [DBG] 127.0.0.1:6222 - rid:25 - Sent local subscriptions to route
[4921] 2021/10/21 16:59:00.332822 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Started
[4921] 2021/10/21 16:59:00.332838 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.332855 [DBG] Starting consumer monitor for '$G > corejobs > news-ResendNotification-worker
[4921] 2021/10/21 16:59:00.332897 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-TZm4O8z6 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.333543 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Update peers from leader to map[EEUOgTxU:0xc0006781d0 FFfB0Dc4:0xc0006781c0 VpUo9efM:0xc0006781b0]
[4921] 2021/10/21 16:59:00.333571 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Started
[4921] 2021/10/21 16:59:00.333580 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.333593 [DBG] Starting consumer monitor for '$G > corejobs > notifications-ProcessBucket-worker
[4921] 2021/10/21 16:59:00.333652 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-HaQEnM15 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.337052 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc000694e70 FFfB0Dc4:0xc000694e90 VpUo9efM:0xc000694e80]
[4921] 2021/10/21 16:59:00.337297 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc000694e70 FFfB0Dc4:0xc000694e90 VpUo9efM:0xc000694e80]
[4921] 2021/10/21 16:59:00.337781 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc000694e70 FFfB0Dc4:0xc000694e90 VpUo9efM:0xc000694e80]
[4921] 2021/10/21 16:59:00.337824 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Started
[4921] 2021/10/21 16:59:00.337852 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.337871 [DBG] Starting consumer monitor for '$G > corejobs > notifications-SendPush-worker
[4921] 2021/10/21 16:59:00.337952 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-qU4dGbjF Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:00.338378 [DBG] Trying to connect to route on localhost:6223 (127.0.0.1:6223)
[4921] 2021/10/21 16:59:00.338519 [DBG] 127.0.0.1:6223 - rid:28 - Route connect msg sent
[4921] 2021/10/21 16:59:00.338537 [INF] 127.0.0.1:6223 - rid:28 - Route connection created
[4921] 2021/10/21 16:59:00.338674 [DBG] 127.0.0.1:6223 - rid:28 - Registering remote route "NA3FJFGNNGTPXT3HNKAASQ64YKKA64FW4QGEUXMQYXNA3FLO54VGGQLJ"
[4921] 2021/10/21 16:59:00.338717 [DBG] 127.0.0.1:6223 - rid:28 - Sent local subscriptions to route
[4921] 2021/10/21 16:59:00.342533 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc000132d00 FFfB0Dc4:0xc000132d10 VpUo9efM:0xc000132cf0]
[4921] 2021/10/21 16:59:00.342876 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc000132d00 FFfB0Dc4:0xc000132d10 VpUo9efM:0xc000132cf0]
[4921] 2021/10/21 16:59:00.343529 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc000132d00 FFfB0Dc4:0xc000132d10 VpUo9efM:0xc000132cf0]
[4921] 2021/10/21 16:59:00.344321 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Started
[4921] 2021/10/21 16:59:00.344330 [DBG] JetStream cluster, consumer was already running
[4921] 2021/10/21 16:59:00.344336 [DBG] Recovered JetStream cluster metadata
[4921] 2021/10/21 16:59:00.344348 [DBG] Starting consumer monitor for '$G > corejobs > test-LoadTest-worker
[4921] 2021/10/21 16:59:00.353221 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Installing snapshot of 209974 bytes
[4921] 2021/10/21 16:59:00.368088 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.368112 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] AppendEntry did not match 103 36184 with 97 36181
[4921] 2021/10/21 16:59:00.394019 [DBG] RAFT [VpUo9efM - _meta_] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.394089 [DBG] RAFT [VpUo9efM - _meta_] AppendEntry did not match 831 1450 with 826 1449
[4921] 2021/10/21 16:59:00.404683 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.404736 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] AppendEntry did not match 94 35299 with 89 35296
[4921] 2021/10/21 16:59:00.412486 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.412552 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] AppendEntry did not match 136 37193 with 131 37192
[4921] 2021/10/21 16:59:00.413977 [DBG] JetStream cluster failed to delete msg 96351 from stream "corejobs" for account "": stream store EOF
[4921] 2021/10/21 16:59:00.471089 [INF] 127.0.0.1:34502 - rid:30 - Route connection created
[4921] 2021/10/21 16:59:00.471774 [DBG] 127.0.0.1:34502 - rid:30 - Detected duplicate remote route "NAASKK6U7M5S45MJNXJJ4LDKJKJACVAQFWPM3B54XJONCIJ7O73XBVGO"
[4921] 2021/10/21 16:59:00.471834 [INF] 127.0.0.1:34502 - rid:30 - Router connection closed: Duplicate Route
[4921] 2021/10/21 16:59:00.487848 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] AppendEntry updating leader to "EEUOgTxU"
[4921] 2021/10/21 16:59:00.487933 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] AppendEntry did not match 103 92209 with 99 92206
[4921] 2021/10/21 16:59:00.635060 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] AppendEntry updating leader to "EEUOgTxU"
[4921] 2021/10/21 16:59:00.635136 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] AppendEntry did not match 114 49839 with 109 49836
[4921] 2021/10/21 16:59:00.637154 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Update peers from leader to map[EEUOgTxU:0xc000678310 FFfB0Dc4:0xc000678300 VpUo9efM:0xc0006781b0]
[4921] 2021/10/21 16:59:00.658222 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.658263 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] AppendEntry did not match 123 35641 with 118 35638
[4921] 2021/10/21 16:59:00.798402 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:00.798460 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] AppendEntry did not match 101 36226 with 96 36223
[4921] 2021/10/21 16:59:00.868869 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:00.869072 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Update peers from leader to map[EEUOgTxU:0xc000658590 FFfB0Dc4:0xc000658580 VpUo9efM:0xc000658570]
[4921] 2021/10/21 16:59:00.894070 [DBG] RAFT [VpUo9efM - _meta_] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:00.894266 [DBG] RAFT [VpUo9efM - _meta_] Update peers from leader to map[EEUOgTxU:0xc0000bab10 FFfB0Dc4:0xc0000bab20 VpUo9efM:0xc0000bab00]
[4921] 2021/10/21 16:59:00.905562 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:00.905753 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc000986f10 FFfB0Dc4:0xc000986f00 VpUo9efM:0xc000986d70]
[4921] 2021/10/21 16:59:00.911535 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:00.911717 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Update peers from leader to map[EEUOgTxU:0xc0009879b0 FFfB0Dc4:0xc0009879a0 VpUo9efM:0xc0001bfe20]
[4921] 2021/10/21 16:59:00.987860 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:00.988043 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Update peers from leader to map[EEUOgTxU:0xc0006583b0 FFfB0Dc4:0xc0006583c0 VpUo9efM:0xc0006583a0]
[4921] 2021/10/21 16:59:01.034001 [INF] 127.0.0.1:34504 - rid:31 - Route connection created
[4921] 2021/10/21 16:59:01.034571 [DBG] 127.0.0.1:34504 - rid:31 - Detected duplicate remote route "NA3FJFGNNGTPXT3HNKAASQ64YKKA64FW4QGEUXMQYXNA3FLO54VGGQLJ"
[4921] 2021/10/21 16:59:01.034622 [INF] 127.0.0.1:34504 - rid:31 - Router connection closed: Duplicate Route
[4921] 2021/10/21 16:59:01.134874 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:01.158353 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:01.158620 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc000133a60 FFfB0Dc4:0xc000133a70 VpUo9efM:0xc000694e80]
[4921] 2021/10/21 16:59:01.298067 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:01.298162 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc0001bf630 FFfB0Dc4:0xc0001bf640 VpUo9efM:0xc000132cf0]
[4921] 2021/10/21 16:59:01.318748 [DBG] Checking JetStream cluster state
[4921] 2021/10/21 16:59:01.367532 [DBG] 127.0.0.1:6222 - rid:25 - Router Ping Timer
[4921] 2021/10/21 16:59:01.379110 [DBG] 127.0.0.1:6223 - rid:28 - Router Ping Timer

----------------------------- AFTER PUBLISH ------------------------------------

[4921] 2021/10/21 16:59:48.206610 [WRN] Got stream sequence mismatch for '$G > corejobs'
[4921] 2021/10/21 16:59:48.206667 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.206932 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Deleted
[4921] 2021/10/21 16:59:48.207390 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207400 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207439 [DBG] Exiting consumer monitor for '$G > corejobs > test-LoadTest-worker'
[4921] 2021/10/21 16:59:48.207441 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207461 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Shutdown
[4921] 2021/10/21 16:59:48.207493 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207501 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207527 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207531 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Shutdown
[4921] 2021/10/21 16:59:48.207547 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207554 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207587 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207594 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Shutdown
[4921] 2021/10/21 16:59:48.207614 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207620 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207628 [DBG] Exiting consumer monitor for '$G > corejobs > accessors-BranchDeletion-worker'
[4921] 2021/10/21 16:59:48.207646 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207653 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Shutdown
[4921] 2021/10/21 16:59:48.207672 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207679 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207700 [DBG] Exiting consumer monitor for '$G > corejobs > chat-SendMessageNotification-worker'
[4921] 2021/10/21 16:59:48.207707 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207713 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Shutdown
[4921] 2021/10/21 16:59:48.207730 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207737 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207757 [DBG] Exiting consumer monitor for '$G > corejobs > news-ResendNotification-worker'
[4921] 2021/10/21 16:59:48.207769 [DBG] Exiting consumer monitor for '$G > corejobs > notifications-ProcessBucket-worker'
[4921] 2021/10/21 16:59:48.207811 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 16:59:48.207820 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Shutdown
[4921] 2021/10/21 16:59:48.207848 [DBG] Exiting consumer monitor for '$G > corejobs > notifications-SendPush-worker'
[4921] 2021/10/21 16:59:48.207858 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207866 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207926 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 16:59:48.207934 [WRN] Resetting stream '$G > corejobs'
[4921] 2021/10/21 16:59:48.207942 [DBG] Exiting stream monitor for '$G > corejobs'
[4921] 2021/10/21 16:59:48.207976 [DBG] JetStream cluster creating raft group:&{Name:S-R3F-bSTo304i Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.208630 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Started
[4921] 2021/10/21 16:59:48.217250 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-HaQEnM15 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.220477 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc0009821f0 FFfB0Dc4:0xc000982210 VpUo9efM:0xc000982200]
[4921] 2021/10/21 16:59:48.220701 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc0009821f0 FFfB0Dc4:0xc000982210 VpUo9efM:0xc000982200]
[4921] 2021/10/21 16:59:48.221210 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc0009821f0 FFfB0Dc4:0xc000982210 VpUo9efM:0xc000982200]
[4921] 2021/10/21 16:59:48.221239 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Started
[4921] 2021/10/21 16:59:48.221308 [DBG] Starting stream monitor for '$G > corejobs'
[4921] 2021/10/21 16:59:48.227296 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-qU4dGbjF Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.230345 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc001790090 FFfB0Dc4:0xc0017900a0 VpUo9efM:0xc001790080]
[4921] 2021/10/21 16:59:48.230566 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc001790090 FFfB0Dc4:0xc0017900a0 VpUo9efM:0xc001790080]
[4921] 2021/10/21 16:59:48.231037 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc001790090 FFfB0Dc4:0xc0017900a0 VpUo9efM:0xc001790080]
[4921] 2021/10/21 16:59:48.231981 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Started
[4921] 2021/10/21 16:59:48.232056 [DBG] Starting consumer monitor for '$G > corejobs > notifications-SendPush-worker
[4921] 2021/10/21 16:59:48.237596 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-t4uNCeJ4 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.241443 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc001cfad80 FFfB0Dc4:0xc001cfad70 VpUo9efM:0xc001cfad60]
[4921] 2021/10/21 16:59:48.241467 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc001cfad80 FFfB0Dc4:0xc001cfad70 VpUo9efM:0xc001cfad60]
[4921] 2021/10/21 16:59:48.241493 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Started
[4921] 2021/10/21 16:59:48.241569 [DBG] Starting consumer monitor for '$G > corejobs > test-LoadTest-worker
[4921] 2021/10/21 16:59:48.246971 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-hGUj1SUd Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.247339 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Started
[4921] 2021/10/21 16:59:48.247471 [DBG] Starting consumer monitor for '$G > corejobs > accessors-BranchDeletion-worker
[4921] 2021/10/21 16:59:48.252750 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-LbOAptkd Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.255543 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Started
[4921] 2021/10/21 16:59:48.255633 [DBG] Starting consumer monitor for '$G > corejobs > chat-SendMessageNotification-worker
[4921] 2021/10/21 16:59:48.260855 [DBG] JetStream cluster creating raft group:&{Name:C-R3F-TZm4O8z6 Peers:[EEUOgTxU VpUo9efM FFfB0Dc4] Storage:File Preferred: node:<nil>}
[4921] 2021/10/21 16:59:48.261406 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Update peers from leader to map[EEUOgTxU:0xc0001be340 FFfB0Dc4:0xc0001be330 VpUo9efM:0xc0001be310]
[4921] 2021/10/21 16:59:48.261417 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Update peers from leader to map[EEUOgTxU:0xc0001be340 FFfB0Dc4:0xc0001be330 VpUo9efM:0xc0001be310]
[4921] 2021/10/21 16:59:48.261436 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Started
[4921] 2021/10/21 16:59:48.261470 [DBG] Starting consumer monitor for '$G > corejobs > news-ResendNotification-worker
[4921] 2021/10/21 16:59:48.266700 [DBG] Starting consumer monitor for '$G > corejobs > notifications-ProcessBucket-worker
[4921] 2021/10/21 16:59:48.298061 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:48.298157 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Update peers from leader to map[EEUOgTxU:0xc003b54300 FFfB0Dc4:0xc003b54310 VpUo9efM:0xc001790080]
[4921] 2021/10/21 16:59:48.369189 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:48.369310 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Update peers from leader to map[EEUOgTxU:0xc0001be0d0 FFfB0Dc4:0xc0001be0c0 VpUo9efM:0xc0001be0b0]
[4921] 2021/10/21 16:59:48.404929 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:48.405070 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Update peers from leader to map[EEUOgTxU:0xc001cfafa0 FFfB0Dc4:0xc001cfaf90 VpUo9efM:0xc001cfad60]
[4921] 2021/10/21 16:59:48.487459 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] AppendEntry updating leader to "EEUOgTxU"
[4921] 2021/10/21 16:59:48.487607 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Update peers from leader to map[EEUOgTxU:0xc001cfb270 FFfB0Dc4:0xc001cfb260 VpUo9efM:0xc001cfb250]
[4921] 2021/10/21 16:59:48.634640 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] AppendEntry updating leader to "EEUOgTxU"
[4921] 2021/10/21 16:59:48.658091 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:48.658378 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Update peers from leader to map[EEUOgTxU:0xc000ba0e00 FFfB0Dc4:0xc000ba0df0 VpUo9efM:0xc000982200]
[4921] 2021/10/21 16:59:48.911504 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] AppendEntry updating leader to "FFfB0Dc4"
[4921] 2021/10/21 16:59:48.911570 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] AppendEntry did not match 136 37194 with 0 0
[4921] 2021/10/21 16:59:48.913041 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Update peers from leader to map[EEUOgTxU:0xc0001af8d0 FFfB0Dc4:0xc0001af8e0 VpUo9efM:0xc0001af8c0]
[4921] 2021/10/21 16:59:48.913536 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Truncating and repairing WAL
[4921] 2021/10/21 16:59:48.913571 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:48.937912 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Pausing apply channel
[4921] 2021/10/21 16:59:49.412007 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] AppendEntry did not match 136 37194 with 136 37193
[4921] 2021/10/21 16:59:49.911550 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Canceling catchup subscription since we are now up to date
[4921] 2021/10/21 16:59:49.911615 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:50.411553 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:50.911953 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:51.411301 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:51.911509 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:52.411415 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:52.911743 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:53.411923 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:53.911679 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:53.939215 [WRN] Catchup for stream '$G > corejobs' stalled
[4921] 2021/10/21 16:59:54.411944 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:54.912339 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:55.411698 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:55.912024 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:56.411929 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:56.911483 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:57.411834 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:57.911959 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:58.412247 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:58.911330 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:58.940256 [WRN] Catchup for stream '$G > corejobs' stalled
[4921] 2021/10/21 16:59:59.411726 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 16:59:59.911743 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 17:00:00.411657 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 17:00:00.911934 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 17:00:01.412440 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 17:00:01.911902 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Paused, not applying 37194
[4921] 2021/10/21 17:00:02.062531 [DBG] Trapped "interrupt" signal
[4921] 2021/10/21 17:00:02.062570 [DBG] Shutting down all raft nodes
[4921] 2021/10/21 17:00:02.062664 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.062677 [DBG] RAFT [VpUo9efM - C-R3F-HaQEnM15] Shutdown
[4921] 2021/10/21 17:00:02.062697 [DBG] Exiting consumer monitor for '$G > corejobs > notifications-SendPush-worker'
[4921] 2021/10/21 17:00:02.062727 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.062781 [DBG] RAFT [VpUo9efM - C-R3F-qU4dGbjF] Shutdown
[4921] 2021/10/21 17:00:02.062735 [DBG] Exiting consumer monitor for '$G > corejobs > test-LoadTest-worker'
[4921] 2021/10/21 17:00:02.062859 [DBG] Exiting consumer monitor for '$G > corejobs > accessors-BranchDeletion-worker'
[4921] 2021/10/21 17:00:02.062863 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.062888 [DBG] RAFT [VpUo9efM - C-R3F-t4uNCeJ4] Shutdown
[4921] 2021/10/21 17:00:02.062959 [DBG] Exiting consumer monitor for '$G > corejobs > chat-SendMessageNotification-worker'
[4921] 2021/10/21 17:00:02.062972 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.063008 [DBG] RAFT [VpUo9efM - C-R3F-hGUj1SUd] Shutdown
[4921] 2021/10/21 17:00:02.063117 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.063140 [DBG] RAFT [VpUo9efM - C-R3F-LbOAptkd] Shutdown
[4921] 2021/10/21 17:00:02.063144 [DBG] Exiting consumer monitor for '$G > corejobs > news-ResendNotification-worker'
[4921] 2021/10/21 17:00:02.063203 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.063222 [DBG] RAFT [VpUo9efM - C-R3F-TZm4O8z6] Shutdown
[4921] 2021/10/21 17:00:02.063262 [DBG] Exiting metadata monitor
[4921] 2021/10/21 17:00:02.063265 [DBG] Exiting consumer monitor for '$G > corejobs > notifications-ProcessBucket-worker'
[4921] 2021/10/21 17:00:02.063285 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.063325 [DBG] RAFT [VpUo9efM - _meta_] Shutdown
[4921] 2021/10/21 17:00:02.063389 [DBG] SYSTEM - System connection closed: Internal Client
[4921] 2021/10/21 17:00:02.063409 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Shutdown
[4921] 2021/10/21 17:00:02.063432 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Resuming apply channel
[4921] 2021/10/21 17:00:02.063468 [DBG] RAFT [VpUo9efM - S-R3F-bSTo304i] Resuming 2 replays
[4921] 2021/10/21 17:00:02.063487 [DBG] Exiting stream monitor for '$G > corejobs'
[4921] 2021/10/21 17:00:02.063715 [INF] Initiating Shutdown...
[4921] 2021/10/21 17:00:02.063738 [INF] Initiating JetStream Shutdown...
[4921] 2021/10/21 17:00:02.063823 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063848 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063882 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063896 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063907 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063915 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063925 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063934 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063944 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063952 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063963 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063972 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063985 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.063994 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.064033 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.064177 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[4921] 2021/10/21 17:00:02.064195 [INF] JetStream Shutdown
[4921] 2021/10/21 17:00:02.064291 [INF] 127.0.0.1:6222 - rid:25 - Router connection closed: Server Shutdown
[4921] 2021/10/21 17:00:02.064300 [DBG] Client accept loop exiting..
[4921] 2021/10/21 17:00:02.064324 [DBG] SYSTEM - System connection closed: Client Closed
[4921] 2021/10/21 17:00:02.064323 [INF] 127.0.0.1:6223 - rid:28 - Router connection closed: Server Shutdown
[4921] 2021/10/21 17:00:02.064333 [DBG] Route accept loop exiting..
[4921] 2021/10/21 17:00:02.064558 [DBG] 127.0.0.1:6222 - rid:25 - Removing 76 subscriptions for account "$SYS"
[4921] 2021/10/21 17:00:02.064639 [INF] Server Exiting..
[4921] 2021/10/21 17:00:02.064655 [DBG] 127.0.0.1:6223 - rid:28 - Removing 90 subscriptions for account "$SYS"