jbw976

jared@Jareds-MacBook-Pro ~/dev/k8s-vagrant-multi-node ((HEAD detached at upstream/multi_os))
>  make up -j 2
vagrant up
NODE=1 vagrant up
Bringing machine 'node1' up with 'virtualbox' provider...
Bringing machine 'master' up with 'virtualbox' provider...
==> master: Box 'generic/fedora29' could not be found. Attempting to find and install...
    master: Box Provider: virtualbox
==> node1: Box 'generic/fedora29' could not be found. Attempting to find and install...
    master: Box Version: >= 0

jbw976

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000

jbw976

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000

jbw976

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000

jbw976

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000

jbw976

> kubectl -n rook-minio get pod -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    creationTimestamp: 2018-11-04T03:48:38Z
    generateName: my-store-
    labels:
      app: minio

jbw976

Slack link (which may be archived due to 10k message limit): https://rook-io.slack.com/archives/C764K425D/p1533770804000083

we have kubelet logs: https://jenkins.rook.io/blue/organizations/jenkins/rook%2Frook/detail/PR-2010/3/artifacts

and it will even be useful to debug the statefulset issue in this build

travisn [4:33 PM]
in the kubelet log:
```22:58:24.839204   14247 desired_state_of_world_populator.go:311] Failed to add volume "rookpvc" (specName: "pvc-875ab62f-9b5e-11e8-b0eb-0af5d80321b6") for pod "875d980e-9b5e-11e8-b0eb-0af5d80321b6" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pvc-875ab62f-9b5e-11e8-b0eb-0af5d80321b6" err=no volume plugin matched```
hmm, why isn’t the flex volume plugin found?

jbw976

Remove k8s-nvme-01.acme.org node from Rook orchestration

delete OSD replica set, this will also stop/kill the OSD pod
 kubectl -n rook delete replicaset rook-ceph-osd-k8s-nvme-01.acme.org

remove entry for k8s-nvme-01.acme.org node from the orchestration status map
 kubectl -n rook edit cm rook-ceph-osd-orchestration-status

delete all node's OSD config maps: rook-ceph-osd-XX-fs-backup (93,94,95,96,97,98,99,100,101).  example:
 kubectl -n rook delete cm rook-ceph-osd-93-fs-backup

jbw976

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-01-08T14:11:51Z
  labels:
    app: cluster-base-storage
    chart: storage-0.1.0
    heritage: Tiller

jbw976

#!/bin/bash -e

image_name_pattern=$1
if [[ -z ${image_name_pattern} ]]; then
    echo "image_name_pattern required"
fi

vms="core-01 core-02"

vernum=$2