Skip to content

Instantly share code, notes, and snippets.

@galexrt
Created June 2, 2018 11:18
Show Gist options
  • Save galexrt/d32cab3c4029617c9e8180f66d286129 to your computer and use it in GitHub Desktop.
Save galexrt/d32cab3c4029617c9e8180f66d286129 to your computer and use it in GitHub Desktop.
The new active mgr gets stuck in `active, starting`.
+ rook-ceph-mgr0-7fd45d88f5-w59tt › rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.480868 I | rookcmd: starting Rook v0.7.0-163.g5d1d3b5a with arguments '/usr/local/bin/rook mgr --config-dir=/var/lib/rook'
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.480962 I | rookcmd: flag values: --admin-secret=*****, --ceph-config-override=/etc/rook/config/override.conf, --cluster-name=rook, --config-dir=/var/lib/rook, --fsid=, --help=false, --log-level=INFO, --mgr-keyring=AQDOXhJba4L5LBAADDhc0csZbyOME5RfzvALvg==, --mgr-name=rook-ceph-mgr0, --mon-endpoints=rook-ceph-mon0=10.96.149.29:6790,rook-ceph-mon1=10.108.88.252:6790,rook-ceph-mon2=10.101.36.178:6790, --mon-secret=*****, --private-ipv4=172.17.0.8, --public-ipv4=172.17.0.8
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.480972 I | cephmon: parsing mon endpoints: rook-ceph-mon0=10.96.149.29:6790,rook-ceph-mon1=10.108.88.252:6790,rook-ceph-mon2=10.101.36.178:6790
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.480977 I | cephmgr: Starting MGR rook-ceph-mgr0 with keyring AQDOXhJba4L5LBAADDhc0csZbyOME5RfzvALvg==
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.480983 I | cephmgr: Conf files: dir=/var/lib/rook/mgrrook-ceph-mgr0 keyring=/var/lib/rook/mgrrook-ceph-mgr0/keyring
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481446 I | cephmon: writing config file /var/lib/rook/mgrrook-ceph-mgr0/rook.config
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481509 I | cephmon: copying config to /etc/ceph/ceph.conf
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481627 I | cephmgr: starting ceph-mgr
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481665 I | cephmgr: Config file /var/lib/rook/mgrrook-ceph-mgr0/rook.config:
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 [global]
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 run dir = /var/lib/rook/mgrrook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon initial members = rook-ceph-mon0 rook-ceph-mon1 rook-ceph-mon2
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon host = 10.96.149.29:6790,10.108.88.252:6790,10.101.36.178:6790
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 log file = /dev/stdout
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon cluster log file = /dev/stdout
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 public addr = 172.17.0.8
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 cluster addr = 172.17.0.8
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon keyvaluedb = rocksdb
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon_allow_pool_delete = true
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mon_max_pg_per_osd = 1000
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug default = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug rados = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug mon = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug osd = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug bluestore = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug filestore = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug journal = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 debug leveldb = 0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 filestore_omap_backend = rocksdb
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pg bits = 11
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pgp bits = 11
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pool default size = 1
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pool default min size = 1
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pool default pg num = 100
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd pool default pgp num = 100
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 osd objectstore =
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 rbd_default_features = 3
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 fatal signal handlers = false
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 [mgr.rook-ceph-mgr0]
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 keyring = /var/lib/rook/mgrrook-ceph-mgr0/keyring
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 mgr data = /var/lib/rook/mgrrook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481678 I | cephmgr: Config file /var/lib/rook/mgrrook-ceph-mgr0/keyring:
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 [mgr.rook-ceph-mgr0]
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 key = AQDOXhJba4L5LBAADDhc0csZbyOME5RfzvALvg==
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 caps mon = "allow *"
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.481683 I | exec: Running command: ceph-mgr --foreground --cluster=rook --conf=/var/lib/rook/mgrrook-ceph-mgr0/rook.config --keyring=/var/lib/rook/mgrrook-ceph-mgr0/keyring -i rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.496548 I | ceph-mgr: 2018-06-02 11:15:00.496428 7fb9e1e55680 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 11
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.496729 I | ceph-mgr: 2018-06-02 11:15:00.496670 7fb9e1e55680 0 pidfile_write: ignore empty --pid-file
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.524043 I | ceph-mgr: 2018-06-02 11:15:00.523938 7fb9e1e55680 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.533123 I | ceph-mgr: 2018-06-02 11:15:00.532947 7fb9d91bb700 1 mgr init Loading python module 'balancer'
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.548790 I | ceph-mgr: 2018-06-02 11:15:00.548687 7fb9d91bb700 1 mgr init Loading python module 'prometheus'
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.614406 I | ceph-mgr: 2018-06-02 11:15:00.614313 7fb9d91bb700 1 mgr init Loading python module 'restful'
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:00.754065 I | ceph-mgr: 2018-06-02 11:15:00.753856 7fb9d91bb700 1 mgr init Loading python module 'status'
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.278070 I | ceph-mgr: 2018-06-02 11:15:01.278023 7fb9d91bb700 1 mgr handle_mgr_map Activating!
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.279261 I | ceph-mgr: 2018-06-02 11:15:01.279222 7fb9d91bb700 1 mgr handle_mgr_map I am now activating
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.295739 I | ceph-mgr: 2018-06-02 11:15:01.295656 7fb9ce4c9700 1 mgr load Constructed class from module: balancer
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.296188 I | ceph-mgr: 2018-06-02 11:15:01.296140 7fb9ce4c9700 1 mgr load Constructed class from module: prometheus
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.296865 I | ceph-mgr: 2018-06-02 11:15:01.296845 7fb9ce4c9700 1 mgr load Constructed class from module: restful
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.298558 I | ceph-mgr: 2018-06-02 11:15:01.297017 7fb9ce4c9700 1 mgr load Constructed class from module: status
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.300343 I | ceph-mgr: 2018-06-02 11:15:01.300264 7fb9ce4c9700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:01.302058 I | ceph-mgr: 2018-06-02 11:15:01.301960 7fb9cbcc4700 1 mgr[restful] server not running: no certificate configured
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:02.526747 I | ceph-mgr: 2018-06-02 11:15:02.524810 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:04.526535 I | ceph-mgr: 2018-06-02 11:15:04.525944 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:06.526708 I | ceph-mgr: 2018-06-02 11:15:06.526598 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:07.082073 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:07] "GET /metrics HTTP/1.1" 200 23070 "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:08.529024 I | ceph-mgr: 2018-06-02 11:15:08.528426 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:10.529815 I | ceph-mgr: 2018-06-02 11:15:10.529260 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:12.529956 I | ceph-mgr: 2018-06-02 11:15:12.529849 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:14.530425 I | ceph-mgr: 2018-06-02 11:15:14.530340 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:16.531267 I | ceph-mgr: 2018-06-02 11:15:16.530863 7fb9d61b5700 1 mgr send_beacon active
+ rook-ceph-mgr1-5574d59cb9-227jp › rook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325349 I | rookcmd: starting Rook v0.7.0-163.g5d1d3b5a with arguments '/usr/local/bin/rook ceph mgr --config-dir=/var/lib/rook'
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325493 I | rookcmd: flag values: --admin-secret=*****, --ceph-config-override=/etc/rook/config/override.conf, --cluster-name=rook, --config-dir=/var/lib/rook, --fsid=, --help=false, --log-level=INFO, --mgr-keyring=AQDnYxJbaa1rIxAAqcEVcGmiHOcj1qUeEa9b1Q==, --mgr-name=rook-ceph-mgr1, --mon-endpoints=rook-ceph-mon0=10.96.149.29:6790,rook-ceph-mon1=10.108.88.252:6790,rook-ceph-mon2=10.101.36.178:6790, --mon-secret=*****, --private-ipv4=172.17.0.13, --public-ipv4=172.17.0.13
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325513 I | cephmon: parsing mon endpoints: rook-ceph-mon0=10.96.149.29:6790,rook-ceph-mon1=10.108.88.252:6790,rook-ceph-mon2=10.101.36.178:6790
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325523 I | cephmgr: Starting MGR rook-ceph-mgr1 with keyring AQDnYxJbaa1rIxAAqcEVcGmiHOcj1qUeEa9b1Q==
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325533 I | cephmgr: Conf files: dir=/var/lib/rook/mgrrook-ceph-mgr1 keyring=/var/lib/rook/mgrrook-ceph-mgr1/keyring
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.325896 I | cephmon: writing config file /var/lib/rook/mgrrook-ceph-mgr1/rook.config
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.326197 I | cephmon: copying config to /etc/ceph/ceph.conf
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.326547 I | cephmgr: starting ceph-mgr
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.326741 I | cephmgr: Config file /var/lib/rook/mgrrook-ceph-mgr1/rook.config:
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 [global]
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 run dir = /var/lib/rook/mgrrook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon initial members = rook-ceph-mon0 rook-ceph-mon1 rook-ceph-mon2
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon host = 10.96.149.29:6790,10.108.88.252:6790,10.101.36.178:6790
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 log file = /dev/stdout
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon cluster log file = /dev/stdout
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 public addr = 172.17.0.13
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 cluster addr = 172.17.0.13
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon keyvaluedb = rocksdb
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon_allow_pool_delete = true
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mon_max_pg_per_osd = 1000
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug default = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug rados = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug mon = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug osd = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug bluestore = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug filestore = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug journal = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 debug leveldb = 0
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 filestore_omap_backend = rocksdb
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pg bits = 11
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pgp bits = 11
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pool default size = 1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pool default min size = 1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pool default pg num = 100
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd pool default pgp num = 100
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 osd objectstore =
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 rbd_default_features = 3
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 fatal signal handlers = false
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 [mgr.rook-ceph-mgr1]
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 keyring = /var/lib/rook/mgrrook-ceph-mgr1/keyring
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 mgr data = /var/lib/rook/mgrrook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.326875 I | cephmgr: Config file /var/lib/rook/mgrrook-ceph-mgr1/keyring:
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 [mgr.rook-ceph-mgr1]
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 key = AQDnYxJbaa1rIxAAqcEVcGmiHOcj1qUeEa9b1Q==
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 caps mon = "allow *"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.326893 I | exec: Running command: ceph-mgr --foreground --cluster=rook --conf=/var/lib/rook/mgrrook-ceph-mgr1/rook.config --keyring=/var/lib/rook/mgrrook-ceph-mgr1/keyring -i rook-ceph-mgr1
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.340340 I | ceph-mgr: 2018-06-02 11:15:16.339873 7fe4a9fa3680 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 13
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.340404 I | ceph-mgr: 2018-06-02 11:15:16.339980 7fe4a9fa3680 0 pidfile_write: ignore empty --pid-file
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.362977 I | ceph-mgr: 2018-06-02 11:15:16.362940 7fe4a9fa3680 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.372115 I | ceph-mgr: 2018-06-02 11:15:16.371922 7fe4a1309700 1 mgr init Loading python module 'balancer'
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.387937 I | ceph-mgr: 2018-06-02 11:15:16.387829 7fe4a1309700 1 mgr init Loading python module 'prometheus'
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.448060 I | ceph-mgr: 2018-06-02 11:15:16.447930 7fe4a1309700 1 mgr init Loading python module 'restful'
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.639958 I | ceph-mgr: 2018-06-02 11:15:16.639836 7fe4a1309700 1 mgr init Loading python module 'status'
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:16.661788 I | ceph-mgr: 2018-06-02 11:15:16.661737 7fe4a1309700 1 mgr load Constructed class from module: prometheus
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:17.533092 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:17] "GET /metrics HTTP/1.1" 200 23070 "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:18.206612 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:18] "GET /metrics HTTP/1.1" 200 23070 "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:18.364430 I | ceph-mgr: 2018-06-02 11:15:18.364319 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:18.531697 I | ceph-mgr: 2018-06-02 11:15:18.531289 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:18.813359 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:18] "GET /metrics HTTP/1.1" 200 23070 "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:19.406916 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:19] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:20.066592 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:20] "GET /metrics HTTP/1.1" 200 23070 "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:20.365180 I | ceph-mgr: 2018-06-02 11:15:20.365050 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:20.531890 I | ceph-mgr: 2018-06-02 11:15:20.531794 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:20.720294 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:20] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:22.365969 I | ceph-mgr: 2018-06-02 11:15:22.365864 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:22.532743 I | ceph-mgr: 2018-06-02 11:15:22.532613 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:24.367511 I | ceph-mgr: 2018-06-02 11:15:24.367358 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:24.533227 I | ceph-mgr: 2018-06-02 11:15:24.533087 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:26.367972 I | ceph-mgr: 2018-06-02 11:15:26.367890 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-w59tt rook-ceph-mgr0 2018-06-02 11:15:26.533939 I | ceph-mgr: 2018-06-02 11:15:26.533824 7fb9d61b5700 1 mgr send_beacon active
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:28.368852 I | ceph-mgr: 2018-06-02 11:15:28.368437 7fe49e303700 1 mgr send_beacon standby
- rook-ceph-mgr0-7fd45d88f5-w59tt
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:30.368989 I | ceph-mgr: 2018-06-02 11:15:30.368890 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:32.056844 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:32] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:32.369428 I | ceph-mgr: 2018-06-02 11:15:32.369330 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:32.903003 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:32] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:34.369940 I | ceph-mgr: 2018-06-02 11:15:34.369816 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:36.371220 I | ceph-mgr: 2018-06-02 11:15:36.371085 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:36.525368 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:36] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:36.931668 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:36] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:37.375156 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:37] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:37.702641 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:37] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:37.976296 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:37] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:38.124680 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:38] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:38.371935 I | ceph-mgr: 2018-06-02 11:15:38.371819 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:39.489960 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:15:39] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:40.372516 I | ceph-mgr: 2018-06-02 11:15:40.372355 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:42.373858 I | ceph-mgr: 2018-06-02 11:15:42.373738 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:44.374610 I | ceph-mgr: 2018-06-02 11:15:44.374501 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:46.375459 I | ceph-mgr: 2018-06-02 11:15:46.375349 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:48.376141 I | ceph-mgr: 2018-06-02 11:15:48.376030 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:50.377157 I | ceph-mgr: 2018-06-02 11:15:50.376959 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:52.378071 I | ceph-mgr: 2018-06-02 11:15:52.377945 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:54.379627 I | ceph-mgr: 2018-06-02 11:15:54.379511 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:56.380418 I | ceph-mgr: 2018-06-02 11:15:56.380303 7fe49e303700 1 mgr send_beacon standby
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:58.006565 I | ceph-mgr: 2018-06-02 11:15:58.006035 7fe4a1309700 1 mgr handle_mgr_map Activating!
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:58.007031 I | ceph-mgr: 2018-06-02 11:15:58.006482 7fe4a1309700 1 mgr handle_mgr_map I am now activating
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:15:58.381145 I | ceph-mgr: 2018-06-02 11:15:58.380930 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:00.381880 I | ceph-mgr: 2018-06-02 11:16:00.381771 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:02.382969 I | ceph-mgr: 2018-06-02 11:16:02.382447 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:02.865833 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:16:02] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:04.383105 I | ceph-mgr: 2018-06-02 11:16:04.382985 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.384172 I | ceph-mgr: 2018-06-02 11:16:06.384044 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822031 I | ceph-mgr: 2018-06-02 11:16:06.821545 7fe495e16700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'prometheus' while running on mgr.rook-ceph-mgr1: IOError("Port 9283 not bound on '::'",)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822070 I | ceph-mgr: 2018-06-02 11:16:06.821588 7fe495e16700 -1 prometheus.serve:
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822074 I | ceph-mgr: 2018-06-02 11:16:06.821599 7fe495e16700 -1 Traceback (most recent call last):
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822076 I | ceph-mgr: File "/usr/lib64/ceph/mgr/prometheus/module.py", line 673, in serve
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822079 I | ceph-mgr: cherrypy.engine.start()
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822081 I | ceph-mgr: File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 250, in start
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822083 I | ceph-mgr: raise e_info
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822086 I | ceph-mgr: ChannelFailures: IOError("Port 9283 not bound on '::'",)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:06.822088 I | ceph-mgr:
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:08.385288 I | ceph-mgr: 2018-06-02 11:16:08.385118 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:10.386115 I | ceph-mgr: 2018-06-02 11:16:10.385944 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:12.386914 I | ceph-mgr: 2018-06-02 11:16:12.386792 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:14.388203 I | ceph-mgr: 2018-06-02 11:16:14.387367 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:16.387925 I | ceph-mgr: 2018-06-02 11:16:16.387767 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:18.388597 I | ceph-mgr: 2018-06-02 11:16:18.388484 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:20.389208 I | ceph-mgr: 2018-06-02 11:16:20.389042 7fe49e303700 1 mgr send_beacon active (starting)
+ rook-ceph-mgr0-7fd45d88f5-fqtwj › rook-ceph-mgr0
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.302314 I | ceph-mgr: 2018-06-02 11:16:21.302139 7f453d45c680 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 12
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.302521 I | ceph-mgr: 2018-06-02 11:16:21.302305 7f453d45c680 0 pidfile_write: ignore empty --pid-file
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.325428 I | ceph-mgr: 2018-06-02 11:16:21.325021 7f453d45c680 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.333057 I | ceph-mgr: 2018-06-02 11:16:21.332949 7f45347c2700 1 mgr init Loading python module 'balancer'
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.348803 I | ceph-mgr: 2018-06-02 11:16:21.348711 7f45347c2700 1 mgr init Loading python module 'prometheus'
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.408753 I | ceph-mgr: 2018-06-02 11:16:21.408708 7f45347c2700 1 mgr init Loading python module 'restful'
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.550057 I | ceph-mgr: 2018-06-02 11:16:21.549891 7f45347c2700 1 mgr init Loading python module 'status'
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:21.572470 I | ceph-mgr: 2018-06-02 11:16:21.572293 7f45347c2700 1 mgr load Constructed class from module: prometheus
rook-ceph-mgr1-5574d59cb9-227jp rook-ceph-mgr1 2018-06-02 11:16:22.393989 I | ceph-mgr: 2018-06-02 11:16:22.391904 7fe49e303700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:23.325567 I | ceph-mgr: 2018-06-02 11:16:23.325447 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:25.326612 I | ceph-mgr: 2018-06-02 11:16:25.326133 7f45317bc700 1 mgr send_beacon standby
- rook-ceph-mgr1-5574d59cb9-227jp
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:27.326836 I | ceph-mgr: 2018-06-02 11:16:27.326677 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:28.083919 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:16:28] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:29.328423 I | ceph-mgr: 2018-06-02 11:16:29.328297 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:31.328899 I | ceph-mgr: 2018-06-02 11:16:31.328788 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:33.329445 I | ceph-mgr: 2018-06-02 11:16:33.329347 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:33.413127 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:16:33] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:35.330049 I | ceph-mgr: 2018-06-02 11:16:35.329919 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:37.330745 I | ceph-mgr: 2018-06-02 11:16:37.330580 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:39.331641 I | ceph-mgr: 2018-06-02 11:16:39.331503 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:41.332337 I | ceph-mgr: 2018-06-02 11:16:41.332182 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:43.333413 I | ceph-mgr: 2018-06-02 11:16:43.333290 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:45.334238 I | ceph-mgr: 2018-06-02 11:16:45.333854 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:47.334406 I | ceph-mgr: 2018-06-02 11:16:47.334274 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:49.335778 I | ceph-mgr: 2018-06-02 11:16:49.335696 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:51.336544 I | ceph-mgr: 2018-06-02 11:16:51.336385 7f45317bc700 1 mgr send_beacon standby
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:53.013014 I | ceph-mgr: 2018-06-02 11:16:53.012899 7f45347c2700 1 mgr handle_mgr_map Activating!
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:53.013871 I | ceph-mgr: 2018-06-02 11:16:53.013435 7f45347c2700 1 mgr handle_mgr_map I am now activating
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:53.337672 I | ceph-mgr: 2018-06-02 11:16:53.337514 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:55.339199 I | ceph-mgr: 2018-06-02 11:16:55.338986 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:57.340566 I | ceph-mgr: 2018-06-02 11:16:57.340465 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:16:59.341818 I | ceph-mgr: 2018-06-02 11:16:59.341626 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:01.343267 I | ceph-mgr: 2018-06-02 11:17:01.343165 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:02.139491 I | ceph-mgr: ::ffff:172.17.0.11 - - [02/Jun/2018:11:17:02] "GET /metrics HTTP/1.1" 200 - "" "curl/7.29.0"
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:03.344497 I | ceph-mgr: 2018-06-02 11:17:03.344353 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:05.345633 I | ceph-mgr: 2018-06-02 11:17:05.345512 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:07.346930 I | ceph-mgr: 2018-06-02 11:17:07.346790 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:09.347598 I | ceph-mgr: 2018-06-02 11:17:09.347516 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.348236 I | ceph-mgr: 2018-06-02 11:17:11.348116 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.727391 I | ceph-mgr: 2018-06-02 11:17:11.727297 7f45292cf700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'prometheus' while running on mgr.rook-ceph-mgr0: IOError("Port 9283 not bound on '::'",)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.727459 I | ceph-mgr: 2018-06-02 11:17:11.727325 7f45292cf700 -1 prometheus.serve:
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728244 I | ceph-mgr: 2018-06-02 11:17:11.727326 7f45292cf700 -1 Traceback (most recent call last):
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728331 I | ceph-mgr: File "/usr/lib64/ceph/mgr/prometheus/module.py", line 673, in serve
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728627 I | ceph-mgr: cherrypy.engine.start()
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728726 I | ceph-mgr: File "/usr/lib/python2.7/site-packages/cherrypy/process/wspbus.py", line 250, in start
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728746 I | ceph-mgr: raise e_info
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728749 I | ceph-mgr: ChannelFailures: IOError("Port 9283 not bound on '::'",)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:11.728752 I | ceph-mgr:
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:13.349163 I | ceph-mgr: 2018-06-02 11:17:13.349039 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:15.350288 I | ceph-mgr: 2018-06-02 11:17:15.350159 7f45317bc700 1 mgr send_beacon active (starting)
rook-ceph-mgr0-7fd45d88f5-fqtwj rook-ceph-mgr0 2018-06-02 11:17:17.351217 I | ceph-mgr: 2018-06-02 11:17:17.350913 7f45317bc700 1 mgr send_beacon active (starting)
- rook-ceph-mgr0-7fd45d88f5-fqtwj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment