Skip to content

Instantly share code, notes, and snippets.

@gjaldon
Last active December 8, 2021 01:32
Show Gist options
  • Save gjaldon/4128b1eac9e8fb51d5c5ce899db56d9b to your computer and use it in GitHub Desktop.
Save gjaldon/4128b1eac9e8fb51d5c5ce899db56d9b to your computer and use it in GitHub Desktop.
2021-12-08 00:39:15 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 00:39:20 INFO AbstractOperator:455 - Reconciliation #220(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:39:21 WARN AbstractOperator:377 - Reconciliation #218(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:40:07 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:40:08 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 00:40:56 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:40:56 INFO KafkaRoller:294 - Reconciliation #214(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 00:40:57 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:40:57 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 00:41:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:41:11 INFO AbstractOperator:217 - Reconciliation #223(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:41:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #223(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:41:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #223(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:41:11 INFO AbstractOperator:455 - Reconciliation #223(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:41:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:41:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:41:14 INFO AbstractOperator:217 - Reconciliation #224(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:41:14 INFO AbstractOperator:217 - Reconciliation #225(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:41:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #225(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:41:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #225(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:41:14 INFO AbstractOperator:455 - Reconciliation #225(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:41:20 INFO AbstractOperator:455 - Reconciliation #224(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:41:21 WARN AbstractOperator:377 - Reconciliation #222(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:41:49 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:41:50 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/1 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2021-12-08 00:41:50 INFO KafkaRoller:294 - Reconciliation #214(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-1 is currently not rollable, retrying after at least 500ms
2021-12-08 00:41:50 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:41:51 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/1 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2021-12-08 00:41:51 INFO KafkaRoller:294 - Reconciliation #214(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-1 is currently not rollable, retrying after at least 1000ms
2021-12-08 00:41:52 INFO KafkaRoller:500 - Reconciliation #214(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:41:53 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 00:43:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:43:11 INFO AbstractOperator:217 - Reconciliation #227(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:43:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #227(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:43:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #227(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:43:11 INFO AbstractOperator:455 - Reconciliation #227(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:43:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:43:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:43:14 INFO AbstractOperator:217 - Reconciliation #228(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:43:14 INFO AbstractOperator:217 - Reconciliation #229(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:43:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #229(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:43:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #229(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:43:15 INFO AbstractOperator:455 - Reconciliation #229(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:43:20 INFO AbstractOperator:455 - Reconciliation #228(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:43:21 WARN AbstractOperator:377 - Reconciliation #226(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:45:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:45:11 INFO AbstractOperator:217 - Reconciliation #231(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:45:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #231(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:45:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #231(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:45:11 INFO AbstractOperator:455 - Reconciliation #231(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:45:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:45:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:45:14 INFO AbstractOperator:217 - Reconciliation #232(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:45:14 INFO AbstractOperator:217 - Reconciliation #233(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:45:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #233(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:45:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #233(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:45:14 INFO AbstractOperator:455 - Reconciliation #233(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:45:21 WARN AbstractOperator:377 - Reconciliation #230(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:45:22 INFO AbstractOperator:455 - Reconciliation #232(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:46:47 INFO AbstractOperator:455 - Reconciliation #214(timer) Kafka(staging/analytics): reconciled
2021-12-08 00:47:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:47:11 INFO AbstractOperator:217 - Reconciliation #234(timer) Kafka(staging/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:47:11 INFO AbstractOperator:217 - Reconciliation #235(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:47:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:47:12 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #235(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:47:12 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #235(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:47:12 INFO AbstractOperator:455 - Reconciliation #235(timer) KafkaRebalance(staging/my-rebalance): reconciled
Dec 08, 2021 12:47:15 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main]=Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2903 ms, time limit is 2000 ms
2021-12-08 00:47:15 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:47:15 INFO AbstractOperator:217 - Reconciliation #236(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:47:15 INFO AbstractOperator:217 - Reconciliation #237(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:47:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #237(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:47:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #237(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:47:15 INFO AbstractOperator:455 - Reconciliation #237(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:47:15 INFO ZookeeperLeaderFinder:92 - Trusting certificate ca.crt from Secret analytics-cluster-ca-cert
2021-12-08 00:47:15 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.p12 in Secret analytics-cluster-ca-cert
2021-12-08 00:47:15 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.password in Secret analytics-cluster-ca-cert
2021-12-08 00:47:16 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-0 is not a leader
2021-12-08 00:47:17 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-1 is not a leader
2021-12-08 00:47:18 INFO ZookeeperLeaderFinder:222 - Pod analytics-zookeeper-2 is leader
2021-12-08 00:47:18 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-0
2021-12-08 00:47:22 INFO AbstractOperator:455 - Reconciliation #236(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:47:58 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 00:48:53 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-2
2021-12-08 00:49:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:49:11 INFO AbstractOperator:217 - Reconciliation #239(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:49:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #239(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:49:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #239(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:49:11 INFO AbstractOperator:455 - Reconciliation #239(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:49:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:49:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:49:14 INFO AbstractOperator:217 - Reconciliation #240(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:49:14 INFO AbstractOperator:217 - Reconciliation #241(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:49:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #241(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:49:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #241(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:49:15 INFO AbstractOperator:455 - Reconciliation #241(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:49:20 INFO AbstractOperator:455 - Reconciliation #240(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:49:21 WARN AbstractOperator:377 - Reconciliation #238(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:49:35 INFO KafkaRoller:500 - Reconciliation #234(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:49:36 INFO KafkaRoller:294 - Reconciliation #234(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 00:49:36 INFO KafkaRoller:500 - Reconciliation #234(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:49:37 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 00:50:28 INFO KafkaRoller:500 - Reconciliation #234(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:50:29 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 00:51:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:51:11 INFO AbstractOperator:217 - Reconciliation #243(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:51:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #243(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:51:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #243(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:51:11 INFO AbstractOperator:455 - Reconciliation #243(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:51:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:51:14 INFO KafkaRoller:500 - Reconciliation #234(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:51:14 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 00:51:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:51:14 INFO AbstractOperator:217 - Reconciliation #244(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:51:14 INFO AbstractOperator:217 - Reconciliation #245(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:51:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #245(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:51:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #245(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:51:15 INFO AbstractOperator:455 - Reconciliation #245(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:51:20 INFO AbstractOperator:455 - Reconciliation #244(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:51:21 WARN AbstractOperator:377 - Reconciliation #242(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:53:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:53:11 INFO AbstractOperator:217 - Reconciliation #247(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:53:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #247(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:53:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #247(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:53:11 INFO AbstractOperator:455 - Reconciliation #247(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:53:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:53:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:53:14 INFO AbstractOperator:217 - Reconciliation #248(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:53:14 INFO AbstractOperator:217 - Reconciliation #249(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:53:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #249(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:53:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #249(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:53:15 INFO AbstractOperator:455 - Reconciliation #249(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:53:20 INFO AbstractOperator:455 - Reconciliation #248(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:53:21 WARN AbstractOperator:377 - Reconciliation #246(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:55:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:55:11 INFO AbstractOperator:217 - Reconciliation #251(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:55:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #251(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:55:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #251(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:55:11 INFO AbstractOperator:455 - Reconciliation #251(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:55:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:55:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:55:14 INFO AbstractOperator:217 - Reconciliation #252(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:55:14 INFO AbstractOperator:217 - Reconciliation #253(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:55:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #253(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:55:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #253(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:55:15 INFO AbstractOperator:455 - Reconciliation #253(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:55:20 INFO AbstractOperator:455 - Reconciliation #252(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:55:21 WARN AbstractOperator:377 - Reconciliation #250(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:56:37 INFO AbstractOperator:455 - Reconciliation #234(timer) Kafka(staging/analytics): reconciled
2021-12-08 00:57:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:57:11 INFO AbstractOperator:217 - Reconciliation #254(timer) Kafka(staging/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:57:11 INFO AbstractOperator:217 - Reconciliation #255(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:57:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #255(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:57:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #255(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:57:11 INFO AbstractOperator:455 - Reconciliation #255(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:57:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:57:13 INFO ZookeeperLeaderFinder:92 - Trusting certificate ca.crt from Secret analytics-cluster-ca-cert
2021-12-08 00:57:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.p12 in Secret analytics-cluster-ca-cert
2021-12-08 00:57:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.password in Secret analytics-cluster-ca-cert
2021-12-08 00:57:14 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-0 is not a leader
2021-12-08 00:57:15 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:57:15 INFO AbstractOperator:217 - Reconciliation #256(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:57:15 INFO AbstractOperator:217 - Reconciliation #257(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:57:15 INFO ZookeeperLeaderFinder:222 - Pod analytics-zookeeper-1 is leader
2021-12-08 00:57:15 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-0
2021-12-08 00:57:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #257(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:57:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #257(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:57:15 INFO AbstractOperator:455 - Reconciliation #257(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:57:21 INFO AbstractOperator:455 - Reconciliation #256(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:58:00 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-2
2021-12-08 00:58:34 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 00:59:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 00:59:11 INFO AbstractOperator:217 - Reconciliation #259(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:59:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #259(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:59:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #259(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:59:11 INFO AbstractOperator:455 - Reconciliation #259(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 00:59:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 00:59:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 00:59:14 INFO AbstractOperator:217 - Reconciliation #260(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 00:59:14 INFO AbstractOperator:217 - Reconciliation #261(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 00:59:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #261(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 00:59:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #261(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 00:59:14 INFO AbstractOperator:455 - Reconciliation #261(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 00:59:21 WARN AbstractOperator:377 - Reconciliation #258(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 00:59:21 INFO AbstractOperator:455 - Reconciliation #260(timer) Kafka(prod/analytics): reconciled
2021-12-08 00:59:22 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 00:59:23 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 01:00:01 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:02 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 01:00:02 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:03 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:03 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 250ms
2021-12-08 01:00:03 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:04 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 500ms
2021-12-08 01:00:04 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:04 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:04 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 500ms
2021-12-08 01:00:04 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:05 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 1000ms
2021-12-08 01:00:05 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:06 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:06 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 1000ms
2021-12-08 01:00:06 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:06 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 2000ms
2021-12-08 01:00:07 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:07 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:07 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 2000ms
2021-12-08 01:00:09 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:09 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2021-12-08 01:00:09 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-1 is currently not rollable, retrying after at least 4000ms
2021-12-08 01:00:09 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:10 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:10 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 4000ms
2021-12-08 01:00:13 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:14 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2021-12-08 01:00:14 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-1 is currently not rollable, retrying after at least 8000ms
2021-12-08 01:00:14 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:16 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:00:16 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 8000ms
2021-12-08 01:00:22 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:23 INFO KafkaRoller:294 - Reconciliation #254(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 16000ms
2021-12-08 01:00:24 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:00:25 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 01:01:08 INFO KafkaRoller:500 - Reconciliation #254(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:01:09 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 01:01:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:01:11 INFO AbstractOperator:217 - Reconciliation #263(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:01:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #263(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:01:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #263(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:01:11 INFO AbstractOperator:455 - Reconciliation #263(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:01:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:01:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:01:14 INFO AbstractOperator:217 - Reconciliation #264(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:01:14 INFO AbstractOperator:217 - Reconciliation #265(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:01:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #265(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:01:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #265(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:01:14 INFO AbstractOperator:455 - Reconciliation #265(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:01:20 INFO AbstractOperator:455 - Reconciliation #264(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:01:21 WARN AbstractOperator:377 - Reconciliation #262(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:03:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:03:11 INFO AbstractOperator:217 - Reconciliation #267(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:03:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #267(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:03:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #267(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:03:11 INFO AbstractOperator:455 - Reconciliation #267(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:03:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:03:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:03:14 INFO AbstractOperator:217 - Reconciliation #268(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:03:14 INFO AbstractOperator:217 - Reconciliation #269(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:03:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #269(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:03:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #269(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:03:14 INFO AbstractOperator:455 - Reconciliation #269(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:03:20 INFO AbstractOperator:455 - Reconciliation #268(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:03:21 WARN AbstractOperator:377 - Reconciliation #266(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:05:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:05:11 INFO AbstractOperator:217 - Reconciliation #271(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:05:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #271(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:05:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #271(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:05:11 INFO AbstractOperator:455 - Reconciliation #271(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:05:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:05:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:05:14 INFO AbstractOperator:217 - Reconciliation #272(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:05:14 INFO AbstractOperator:217 - Reconciliation #273(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:05:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #273(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:05:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #273(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:05:15 INFO AbstractOperator:455 - Reconciliation #273(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:05:20 INFO AbstractOperator:455 - Reconciliation #272(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:05:21 WARN AbstractOperator:377 - Reconciliation #270(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:06:17 INFO AbstractOperator:455 - Reconciliation #254(timer) Kafka(staging/analytics): reconciled
2021-12-08 01:07:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:07:11 INFO AbstractOperator:217 - Reconciliation #274(timer) Kafka(staging/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:07:11 INFO AbstractOperator:217 - Reconciliation #275(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:07:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #275(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:07:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #275(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:07:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:07:11 INFO AbstractOperator:455 - Reconciliation #275(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:07:13 INFO ZookeeperLeaderFinder:92 - Trusting certificate ca.crt from Secret analytics-cluster-ca-cert
2021-12-08 01:07:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.p12 in Secret analytics-cluster-ca-cert
2021-12-08 01:07:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.password in Secret analytics-cluster-ca-cert
2021-12-08 01:07:14 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-0 is not a leader
2021-12-08 01:07:15 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:07:15 INFO AbstractOperator:217 - Reconciliation #276(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:07:15 INFO AbstractOperator:217 - Reconciliation #277(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:07:15 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-1 is not a leader
2021-12-08 01:07:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #277(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:07:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #277(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:07:15 INFO AbstractOperator:455 - Reconciliation #277(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:07:15 INFO ZookeeperLeaderFinder:222 - Pod analytics-zookeeper-2 is leader
2021-12-08 01:07:15 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-0
2021-12-08 01:07:21 INFO AbstractOperator:455 - Reconciliation #276(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:07:53 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 01:09:03 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-2
2021-12-08 01:09:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:09:11 INFO AbstractOperator:217 - Reconciliation #279(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:09:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #279(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:09:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #279(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:09:11 INFO AbstractOperator:455 - Reconciliation #279(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:09:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:09:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:09:14 INFO AbstractOperator:217 - Reconciliation #280(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:09:14 INFO AbstractOperator:217 - Reconciliation #281(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:09:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #281(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:09:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #281(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:09:14 INFO AbstractOperator:455 - Reconciliation #281(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:09:20 INFO AbstractOperator:455 - Reconciliation #280(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:09:21 WARN AbstractOperator:377 - Reconciliation #278(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:09:44 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:09:45 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 01:09:45 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:09:46 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 01:10:22 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:23 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:23 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 250ms
2021-12-08 01:10:23 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:24 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 0 is restarted.
2021-12-08 01:10:24 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-0 is currently not rollable, retrying after at least 500ms
2021-12-08 01:10:24 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:24 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:24 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 500ms
2021-12-08 01:10:24 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:25 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 0 is restarted.
2021-12-08 01:10:25 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-0 is currently not rollable, retrying after at least 1000ms
2021-12-08 01:10:25 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:26 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:26 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 1000ms
2021-12-08 01:10:26 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:27 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 2000ms
2021-12-08 01:10:27 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:27 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:27 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 2000ms
2021-12-08 01:10:29 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:29 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 4000ms
2021-12-08 01:10:29 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:30 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:30 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 4000ms
2021-12-08 01:10:33 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:34 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 8000ms
2021-12-08 01:10:34 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:35 INFO KafkaAvailability:121 - strimzi.cruisecontrol.modeltrainingsamples/0 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted.
2021-12-08 01:10:35 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 2 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-2 is currently not rollable, retrying after at least 8000ms
2021-12-08 01:10:42 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:42 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 16000ms
2021-12-08 01:10:43 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:10:44 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 01:11:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:11:11 INFO AbstractOperator:217 - Reconciliation #283(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:11:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #283(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:11:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #283(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:11:11 INFO AbstractOperator:455 - Reconciliation #283(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:11:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:11:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:11:14 INFO AbstractOperator:217 - Reconciliation #284(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:11:14 INFO AbstractOperator:217 - Reconciliation #285(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:11:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #285(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:11:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #285(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:11:15 INFO AbstractOperator:455 - Reconciliation #285(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:11:20 INFO AbstractOperator:455 - Reconciliation #284(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:11:21 WARN AbstractOperator:377 - Reconciliation #282(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:11:21 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:11:22 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 0 is restarted.
2021-12-08 01:11:22 INFO KafkaRoller:294 - Reconciliation #274(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-0 is currently not rollable, retrying after at least 32000ms
2021-12-08 01:11:54 INFO KafkaRoller:500 - Reconciliation #274(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:11:55 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 01:13:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:13:11 INFO AbstractOperator:217 - Reconciliation #287(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:13:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #287(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:13:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #287(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:13:11 INFO AbstractOperator:455 - Reconciliation #287(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:13:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:13:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:13:14 INFO AbstractOperator:217 - Reconciliation #288(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:13:14 INFO AbstractOperator:217 - Reconciliation #289(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:13:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #289(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:13:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #289(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:13:15 INFO AbstractOperator:455 - Reconciliation #289(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:13:20 INFO AbstractOperator:455 - Reconciliation #288(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:13:21 WARN AbstractOperator:377 - Reconciliation #286(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:15:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:15:11 INFO AbstractOperator:217 - Reconciliation #291(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:15:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #291(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:15:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #291(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:15:11 INFO AbstractOperator:455 - Reconciliation #291(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:15:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:15:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:15:14 INFO AbstractOperator:217 - Reconciliation #292(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:15:14 INFO AbstractOperator:217 - Reconciliation #293(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:15:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #293(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:15:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #293(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:15:15 INFO AbstractOperator:455 - Reconciliation #293(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:15:20 INFO AbstractOperator:455 - Reconciliation #292(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:15:21 WARN AbstractOperator:377 - Reconciliation #290(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:16:46 INFO AbstractOperator:455 - Reconciliation #274(timer) Kafka(staging/analytics): reconciled
2021-12-08 01:17:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:17:11 INFO AbstractOperator:217 - Reconciliation #294(timer) Kafka(staging/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:17:11 INFO AbstractOperator:217 - Reconciliation #295(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:17:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #295(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:17:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #295(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:17:11 INFO AbstractOperator:455 - Reconciliation #295(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:17:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:17:13 INFO ZookeeperLeaderFinder:92 - Trusting certificate ca.crt from Secret analytics-cluster-ca-cert
2021-12-08 01:17:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.p12 in Secret analytics-cluster-ca-cert
2021-12-08 01:17:13 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.password in Secret analytics-cluster-ca-cert
2021-12-08 01:17:14 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-0 is not a leader
2021-12-08 01:17:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:17:14 INFO AbstractOperator:217 - Reconciliation #296(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:17:14 INFO AbstractOperator:217 - Reconciliation #297(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:17:14 INFO ZookeeperLeaderFinder:222 - Pod analytics-zookeeper-1 is leader
2021-12-08 01:17:14 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-0
2021-12-08 01:17:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #297(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:17:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #297(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:17:15 INFO AbstractOperator:455 - Reconciliation #297(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:17:20 INFO AbstractOperator:455 - Reconciliation #296(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:17:48 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-2
2021-12-08 01:18:27 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 01:19:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:19:11 INFO AbstractOperator:217 - Reconciliation #299(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:19:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #299(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:19:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #299(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:19:11 INFO AbstractOperator:455 - Reconciliation #299(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:19:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:19:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:19:14 INFO AbstractOperator:217 - Reconciliation #300(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:19:14 INFO AbstractOperator:217 - Reconciliation #301(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:19:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #301(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:19:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #301(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:19:15 INFO AbstractOperator:455 - Reconciliation #301(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:19:17 INFO KafkaRoller:500 - Reconciliation #294(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:19:18 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 01:19:21 WARN AbstractOperator:377 - Reconciliation #298(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:19:21 INFO AbstractOperator:455 - Reconciliation #300(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:20:17 INFO KafkaRoller:500 - Reconciliation #294(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:20:17 INFO KafkaRoller:294 - Reconciliation #294(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 01:20:18 INFO KafkaRoller:500 - Reconciliation #294(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:20:18 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 01:21:07 INFO KafkaRoller:500 - Reconciliation #294(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:21:07 INFO KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/1 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2021-12-08 01:21:07 INFO KafkaRoller:294 - Reconciliation #294(timer) Kafka(staging/analytics): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod analytics-kafka-1 is currently not rollable, retrying after at least 500ms
2021-12-08 01:21:08 INFO KafkaRoller:500 - Reconciliation #294(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:21:09 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 01:21:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:21:11 INFO AbstractOperator:217 - Reconciliation #303(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:21:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #303(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:21:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #303(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:21:11 INFO AbstractOperator:455 - Reconciliation #303(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:21:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:21:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:21:14 INFO AbstractOperator:217 - Reconciliation #304(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:21:14 INFO AbstractOperator:217 - Reconciliation #305(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:21:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #305(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:21:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #305(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:21:14 INFO AbstractOperator:455 - Reconciliation #305(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:21:20 INFO AbstractOperator:455 - Reconciliation #304(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:21:21 WARN AbstractOperator:377 - Reconciliation #302(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:23:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:23:11 INFO AbstractOperator:217 - Reconciliation #307(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:23:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #307(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:23:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #307(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:23:11 INFO AbstractOperator:455 - Reconciliation #307(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:23:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:23:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:23:14 INFO AbstractOperator:217 - Reconciliation #308(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:23:14 INFO AbstractOperator:217 - Reconciliation #309(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:23:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #309(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:23:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #309(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:23:15 INFO AbstractOperator:455 - Reconciliation #309(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:23:20 INFO AbstractOperator:455 - Reconciliation #308(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:23:21 WARN AbstractOperator:377 - Reconciliation #306(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:25:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:25:11 INFO AbstractOperator:217 - Reconciliation #311(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:25:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #311(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:25:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #311(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:25:11 INFO AbstractOperator:455 - Reconciliation #311(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:25:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:25:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:25:14 INFO AbstractOperator:217 - Reconciliation #312(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:25:14 INFO AbstractOperator:217 - Reconciliation #313(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:25:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #313(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:25:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #313(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:25:14 INFO AbstractOperator:455 - Reconciliation #313(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:25:19 INFO AbstractOperator:455 - Reconciliation #312(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:25:21 WARN AbstractOperator:377 - Reconciliation #310(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:26:17 INFO AbstractOperator:455 - Reconciliation #294(timer) Kafka(staging/analytics): reconciled
2021-12-08 01:27:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:27:11 INFO AbstractOperator:217 - Reconciliation #314(timer) Kafka(staging/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:27:11 INFO AbstractOperator:217 - Reconciliation #315(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:27:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #315(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:27:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #315(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:27:11 INFO AbstractOperator:455 - Reconciliation #315(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:27:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:27:14 INFO ZookeeperLeaderFinder:92 - Trusting certificate ca.crt from Secret analytics-cluster-ca-cert
2021-12-08 01:27:14 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.p12 in Secret analytics-cluster-ca-cert
2021-12-08 01:27:14 WARN ZookeeperLeaderFinder:101 - Ignoring non-certificate ca.password in Secret analytics-cluster-ca-cert
2021-12-08 01:27:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:27:14 INFO AbstractOperator:217 - Reconciliation #316(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:27:14 INFO AbstractOperator:217 - Reconciliation #317(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:27:14 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-0 is not a leader
2021-12-08 01:27:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #317(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:27:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #317(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:27:15 INFO AbstractOperator:455 - Reconciliation #317(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:27:15 INFO ZookeeperLeaderFinder:225 - Pod analytics-zookeeper-1 is not a leader
2021-12-08 01:27:16 INFO ZookeeperLeaderFinder:222 - Pod analytics-zookeeper-2 is leader
2021-12-08 01:27:16 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-0
2021-12-08 01:27:21 INFO AbstractOperator:455 - Reconciliation #316(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:27:43 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-1
2021-12-08 01:28:35 INFO PodOperator:65 - Rolling update of staging/analytics-zookeeper: Rolling pod analytics-zookeeper-2
2021-12-08 01:29:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:29:11 INFO AbstractOperator:217 - Reconciliation #319(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:29:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #319(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:29:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #319(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:29:11 INFO AbstractOperator:455 - Reconciliation #319(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:29:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:29:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:29:14 INFO AbstractOperator:217 - Reconciliation #320(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:29:14 INFO AbstractOperator:217 - Reconciliation #321(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:29:15 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #321(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:29:15 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #321(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:29:15 INFO AbstractOperator:455 - Reconciliation #321(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:29:15 INFO KafkaRoller:500 - Reconciliation #314(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:29:16 INFO KafkaRoller:294 - Reconciliation #314(timer) Kafka(staging/analytics): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod analytics-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2021-12-08 01:29:16 INFO KafkaRoller:500 - Reconciliation #314(timer) Kafka(staging/analytics): Pod 1 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:29:18 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-1
2021-12-08 01:29:21 WARN AbstractOperator:377 - Reconciliation #318(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
2021-12-08 01:29:21 INFO AbstractOperator:455 - Reconciliation #320(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:30:02 INFO KafkaRoller:500 - Reconciliation #314(timer) Kafka(staging/analytics): Pod 2 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:30:03 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-2
2021-12-08 01:31:02 INFO KafkaRoller:500 - Reconciliation #314(timer) Kafka(staging/analytics): Pod 0 needs to be restarted. Reason: [server certificates changed]
2021-12-08 01:31:03 INFO PodOperator:65 - Rolling update of staging/analytics-kafka: Rolling pod analytics-kafka-0
2021-12-08 01:31:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace staging...
2021-12-08 01:31:11 INFO AbstractOperator:217 - Reconciliation #323(timer) KafkaRebalance(staging/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:31:11 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #323(timer) KafkaRebalance(staging/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:31:11 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #323(timer) KafkaRebalance(staging/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:31:11 INFO AbstractOperator:455 - Reconciliation #323(timer) KafkaRebalance(staging/my-rebalance): reconciled
2021-12-08 01:31:11 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2021-12-08 01:31:14 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace prod...
2021-12-08 01:31:14 INFO AbstractOperator:217 - Reconciliation #324(timer) Kafka(prod/analytics): Kafka analytics will be checked for creation or modification
2021-12-08 01:31:14 INFO AbstractOperator:217 - Reconciliation #325(timer) KafkaRebalance(prod/my-rebalance): KafkaRebalance my-rebalance will be checked for creation or modification
2021-12-08 01:31:14 INFO KafkaRebalanceAssemblyOperator:319 - Reconciliation #325(timer) KafkaRebalance(prod/my-rebalance): Rebalance action from state [NotReady]
2021-12-08 01:31:14 INFO KafkaRebalanceAssemblyOperator:333 - Reconciliation #325(timer) KafkaRebalance(prod/my-rebalance): State updated to [NotReady] with annotation strimzi.io/rebalance=null
2021-12-08 01:31:14 INFO AbstractOperator:455 - Reconciliation #325(timer) KafkaRebalance(prod/my-rebalance): reconciled
2021-12-08 01:31:20 INFO AbstractOperator:455 - Reconciliation #324(timer) Kafka(prod/analytics): reconciled
2021-12-08 01:31:21 WARN AbstractOperator:377 - Reconciliation #322(timer) Kafka(staging/analytics): Failed to acquire lock lock::staging::Kafka::analytics within 10000ms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment