EFK Logging Deployments for Kubernetes 1.3+, for use with kops

README.md

The kubernetes.io/cluster-service annotation are commented out because I was having issues with kubernetes deleting my deployments.

The StorageClass resource is commented out because I'm still on 1.3.7 (upgrading to 1.4 as we speak) and am using emptyDirs for now.

Note that in kibana, your logs are tagged with logstash-* even though we're using fluentd; it's due to the default fluentd config in the gcr image. Maybe someone will get around to fixing that eventually?

elasticsearch.yml

# apiVersion: storage.k8s.io/v1beta1
# kind: StorageClass
# metadata:
#   name: default
# provisioner: kubernetes.io/aws-ebs
# parameters:
#   type: gp2

# @TODO(shrugs) move to using a StorageClass to dynamically create this EBS volume

# # This is a claim for a PersistentVolume based on the StorageClass above
# apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
#   name: elasticsearch-logging-data-claim
#   namespace: kube-system
# spec:
#   accessModes:
#     - ReadWriteOnce
#   resources:
#     requests:
#       storage: 100Gi
# ---

#
# Heavily based on:
# https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    component: elasticsearch-logging
    # kubernetes.io/cluster-service: "true"
    # ^ this deployment is a cluster "addon"
spec:
  replicas: 2
  minReadySeconds: 30
  selector:
    matchLabels:
      component: elasticsearch-logging
  template:
    metadata:
      name: elasticsearch-logging
      labels:
        component: elasticsearch-logging
        # kubernetes.io/cluster-service: "true"
        # ^ this pod is a cluster "addon"
    spec:
      containers:
        - name: elasticsearch-logging
          image: gcr.io/google_containers/elasticsearch:1.9
          imagePullPolicy: Always
          ports:
            - containerPort: 9200
              name: db
              protocol: TCP
            - containerPort: 9300
              name: transport
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
      volumes:
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    component: elasticsearch-logging
    # kubernetes.io/cluster-service: "true"
    # ^ this service is a cluster "addon"
spec:
  selector:
    component: elasticsearch-logging
  ports:
    - protocol: TCP
      port: 9200
      targetPort: db

fluentd.yml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-logging
  namespace: kube-system
  labels:
    component: fluentd-logging
spec:
  template:
    metadata:
      name: fluentd-logging
      namespace: kube-system
      labels:
        component: fluentd-logging
    spec:
      containers:
        - name: fluentd-logging
          image: gcr.io/google_containers/fluentd-elasticsearch:1.19
          imagePullPolicy: Always
          env:
            - name: FLUENTD_ARGS
              value: -q
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
          resources:
            limits:
              memory: 200Mi
              cpu: 100m
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

kibana.yml

# Heavily based on:
# https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/kibana-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    component: kibana-logging
    # kubernetes.io/cluster-service: "true"
    # ^ this deployment is a cluster "addon"
spec:
  replicas: 1
  selector:
    matchLabels:
      component: kibana-logging
  template:
    metadata:
      name: kibana-logging
      namespace: kube-system
      labels:
        component: kibana-logging
        # kubernetes.io/cluster-service: "true"
        # ^ this pod is a cluster "addon"
    spec:
      containers:
        - name: kibana-logging
          image: gcr.io/google_containers/kibana:1.3
          imagePullPolicy: Always
          ports:
            - containerPort: 5601
              name: kibana
              protocol: TCP
          resources:
            limits:
              cpu: 100m
            requests:
              cpu: 100m
          env:
            - name: "ELASTICSEARCH_URL"
              value: "http://elasticsearch-logging:9200"
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    component: kibana-logging
    # kubernetes.io/cluster-service: "true"
    # ^ this service is a cluster "addon"
spec:
  selector:
    component: kibana-logging
  ports:
    - port: 80
      targetPort: kibana

Note that in kibana, your logs are tagged with logstash-* even though we're using fluentd; it's due to the default fluentd config in the gcr image. Maybe someone will get around to fixing that eventually?

• I am also seeing this issue. What do in this case? Its not working for both "logstash-" and "fluentd-".