GlusterFS notes 01. heketi logs

gistfile1.txt

$ kubectl describe pod heketi-679447869-25f6n
...
Events:
  Type     Reason       Age   From                Message
  ----     ------       ----  ----                -------
  Normal   Scheduled    7m    default-scheduler   Successfully assigned default/heketi-679447869-25f6n to srv-vm-02
  Warning  FailedMount  7m    kubelet, srv-vm-02  MountVolume.SetUp failed for volume "db" : mount failed: mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/c5c266b6-da0f-11e8-bd66-000c29769173/volumes/kubernetes.io~glusterfs/db --scope -- mount -t glusterfs -o log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/db/heketi-679447869-25f6n-glusterfs.log,log-level=ERROR,backup-volfile-servers=192.168.11.14:192.168.11.15:192.168.11.16,auto_unmount 192.168.11.14:heketidbstorage /var/lib/kubelet/pods/c5c266b6-da0f-11e8-bd66-000c29769173/volumes/kubernetes.io~glusterfs/db
Output: Running scope as unit: run-r7ad457c5864e4b4fa8a91c6af02a507a.scope
[2018-10-27 17:43:13.968958] E [glusterfsd.c:795:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
Mount failed. Please check the log file for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue:
The message "E [MSGID: 108006] [afr-common.c:5413:__afr_handle_child_down_event] 0-heketidbstorage-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up." repeated 2 times between [2018-10-27 17:43:13.991771] and [2018-10-27 17:43:13.993168]
The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-heketidbstorage-dht: dict is null" repeated 2 times between [2018-10-27 17:43:13.994386] and [2018-10-27 17:43:13.997768]