____ ___ ____ ____ _ _ ___ __ __ _____ ____
| _ \_ _/ ___| / ___| | / \ |_ _| \/ | ____| _ \
| | | | |\___ \| | | | / _ \ | || |\/| | _| | |_) |
| |_| | | ___) | |___| |___ / ___ \ | || | | | |___| _ <
|____/___|____/ \____|_____/_/ \_\___|_| |_|_____|_| \_\
This is meant as one real-world example of setting up the SignalFx SmartAgent and is not intended to be a replacement for the official documentation. Please make sure that you have read the Kubernetes setup instructions available at https://github.com/signalfx/signalfx-agent/blob/master/docs/kubernetes-setup.md. I am not responsible if you don't read the documentation, you screw something up, or something spontaneously combusts.
- SignalFx SmartAgent Kubernetes Example
- Troubleshooting
- Supplemental Commands
- Kubernetes: v1.10.2
- OS: CentOS Linux release 7.4.1708
- Calico v3.1.1
╭─jivanov@centos-kubernetes-01 ~
╰─$ nano serviceaccount.yaml
╭─jivanov@centos-kubernetes-01 ~
╰─$ cat serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: signalfx-agent
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: signalfx-agent
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: signalfx-agent
namespace: default
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl create -f serviceaccount.yaml
serviceaccount "signalfx-agent" created
NOTE: WITH THE SERVICE ACCOUNT PERMISSIONS ABOVE, THIS STEP IS NOT NECESSARY.
╭─jivanov@centos-kubernetes-01 ~
╰─$ wget https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/clusterrole.yaml
--2018-05-04 10:56:13-- https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/clusterrole.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.44.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.44.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1115 (1.1K) [text/plain]
Saving to: ‘clusterrole.yaml’
100%[===============================================================================================================================================>] 1,115 --.-K/s in 0s
2018-05-04 10:56:13 (102 MB/s) - ‘clusterrole.yaml’ saved [1115/1115]
NOTE: WITH THE SERVICE ACCOUNT PERMISSIONS ABOVE, THIS STEP IS NOT NECESSARY.
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl apply -f clusterrole.yaml
clusterrole.rbac.authorization.k8s.io "signalfx-agent" created
NOTE: WITH THE SERVICE ACCOUNT PERMISSIONS ABOVE, THIS STEP IS NOT NECESSARY.
╭─jivanov@centos-kubernetes-01 ~
╰─$ wget https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/clusterrolebinding.yaml
--2018-05-04 10:44:35-- https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/clusterrolebinding.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.44.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.44.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 370 [text/plain]
Saving to: ‘clusterrolebinding.yaml’
100%[=======================================================================================================================>] 370 --.-K/s in 0s
2018-05-04 10:44:36 (34.9 MB/s) - ‘clusterrolebinding.yaml’ saved [370/370]
NOTE: WITH THE SERVICE ACCOUNT PERMISSIONS ABOVE, THIS STEP IS NOT NECESSARY.
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl apply -f clusterrolebinding.yaml
clusterrole.rbac.authorization.k8s.io "signalfx-agent" configured
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl create secret generic signalfx-agent --from-literal=access-token=<CENSORED>
secret "signalfx-agent" created
╭─jivanov@centos-kubernetes-01 ~
╰─$ wget https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/configmap.yaml
--2018-05-01 16:07:22-- https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/configmap.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.44.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.44.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4829 (4.7K) [text/plain]
Saving to: ‘configmap.yaml’
100%[==================================================================================================================>] 4,829 --.-K/s in 0s
2018-05-01 16:07:22 (40.2 MB/s) - ‘configmap.yaml’ saved [4829/4829]
╭─jivanov@centos-kubernetes-01 ~
╰─$ wget https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/daemonset.yaml
--2018-05-01 16:08:03-- https://raw.githubusercontent.com/signalfx/signalfx-agent/master/deployments/k8s/daemonset.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.184.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.184.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2172 (2.1K) [text/plain]
Saving to: ‘daemonset.yaml’
100%[==================================================================================================================>] 2,172 --.-K/s in 0s
2018-05-01 16:08:03 (27.5 MB/s) - ‘daemonset.yaml’ saved [2172/2172]
As described in https://github.com/signalfx/signalfx-agent/blob/master/docs/kubernetes-setup.md.
╭─jivanov@centos-kubernetes-01 ~
╰─$ nano -w configmap.yaml
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl create -f configmap.yaml -f daemonset.yaml
configmap "signalfx-agent" created
daemonset.extensions "signalfx-agent" created
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl describe daemonset.apps/signalfx-agent
Name: signalfx-agent
Selector: app=signalfx-agent
Node-Selector: <none>
Labels: app=signalfx-agent
version=3.1.2
Annotations: <none>
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status: 1 Running / 2 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=signalfx-agent
version=3.1.2
Service Account: signalfx-agent
Containers:
signalfx-agent:
Image: quay.io/signalfx/signalfx-agent:3.1.2
Port: <none>
Host Port: <none>
Command:
/bin/signalfx-agent
Environment:
SFX_ACCESS_TOKEN: <set to the key 'access-token' in secret 'signalfx-agent'> Optional: false
MY_POD_NAME: (v1:metadata.name)
MY_NODE_NAME: (v1:spec.nodeName)
MY_NAMESPACE: (v1:metadata.namespace)
Mounts:
/etc/machine-id from machine-id (ro)
/etc/signalfx from config (rw)
/hostfs from hostfs (ro)
/var/run/docker.sock from docker (ro)
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: signalfx-agent
Optional: false
hostfs:
Type: HostPath (bare host directory volume)
Path: /
HostPathType:
docker:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
machine-id:
Type: HostPath (bare host directory volume)
Path: /etc/machine-id
HostPathType:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 41s daemonset-controller Created pod: signalfx-agent-vvrhg
Normal SuccessfulCreate 41s daemonset-controller Created pod: signalfx-agent-n4p8j
Normal SuccessfulCreate 41s daemonset-controller Created pod: signalfx-agent-9tbd9
Setting up the pods may take some time depending on the cluster size and configuration but you should start seeing data within a few minutes.
Run:
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl describe nodes
If you see nodes that are in NotReady state or unable to connect to nodes or not able to see all nodes, then make sure the kubelet service is running (and enabled to start on bootup):
╭─jivanov@centos-kubernetes-03 ~
╰─$ sudo systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
╭─jivanov@centos-kubernetes-03 ~
╰─$ sudo systemctl start kubelet.service
Problem:
May 1 17:05:51 centos-kubernetes-01 kubelet: I0501 17:05:51.023810 1413 logs.go:49] http: TLS handshake error from 192.168.1.115:33800: remote error: tls: bad certificate
Or:
May 1 17:05:51 centos-kubernetes-01 journal: time="2018-05-01T22:05:51Z" level=error msg="Couldn't get cAdvisor container stats" error="failed to get all container stats from Kubelet URL "https://centos-kubernetes-01.localdomain:10250/stats/container/": Post https://centos-kubernetes-01.localdomain:10250/stats/container/: x509: certificate signed by unknown authority"
Solution:
Edit the configmap.yaml file and add skipVerify: true as documented ( https://github.com/signalfx/signalfx-agent/blob/master/docs/monitors/kubelet-stats.md ) so this:
- type: kubelet-stats
kubeletAPI:
authType: serviceAccount
Becomes this:
- type: kubelet-stats
kubeletAPI:
authType: serviceAccount
skipVerify: true
Follow the "Delete and re-create the SignalFx SmartAgent daemonset" instructions under "SUPPLEMENTAL COMMANDS".
Problem:
May 1 17:30:10 centos-kubernetes-02 journal: time="2018-05-01T22:30:10Z" level=error msg="Couldn't get machine info: Kubelet request failed - "403 Forbidden", response: "Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=spec)""
Solution:
Sacrifice a virgin goat.
Problem:
May 1 17:30:11 centos-kubernetes-02 journal: E0501 22:30:11.412502 1 reflector.go:205] github.com/signalfx/signalfx-agent/internal/monitors/kubernetes/cluster/clusterstate.go:72: Failed to list *v1beta1.ReplicaSet: replicasets.extensions is forbidden: User "system:serviceaccount:default:signalfx-agent" cannot list replicasets.extensions at the cluster scope
Solution:
The service account does not have the required permissions. Please refer to the above for a suggested service account configuration.
This is useful if you have had to modify the configmap.yaml file.
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl delete ds signalfx-agent
daemonset.extensions "signalfx-agent" deleted
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl replace -f configmap.yaml -f daemonset.yaml
configmap "signalfx-agent" replaced
daemonset.extensions "signalfx-agent" replaced
(NOTE: You may have to start with create which may fail then follow that with the replace)
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl delete serviceaccount/signalfx-agent
serviceaccount "signalfx-agent" deleted
╭─jivanov@centos-kubernetes-01 ~
╰─$ kubectl create -f serviceaccount.yaml
serviceaccount "signalfx-agent" created
clusterrolebinding.rbac.authorization.k8s.io "signalfx-agent" created
This can be used with the Kubernetes Dashboard (Web UI (Dashboard)) as well as cURL commands or other tools for interacting with the Kubernetes API. This assumes that Kubernetes has been configured properly and that additional information, like Use an HTTP Proxy to Access the Kubernetes API, has been followed.
-
Create a SSH tunnel that binds a local port to a remote port for securely communicating with the cluster:
╭─jamie@jamie-signalfx ~ ╰─$ ssh -p <SSH PORT IF NOT 22> -C -L 127.0.0.1:8001:127.0.0.1:8001 <USER>@<KUBERNETES NODE>The tunneling format is
<local interface>:<local port>:<remote interface>:<remote port>; you can modify this as necessary but be sure to update the following commands if you change anything. Please keep in mind if you create a tunnel on a public network interface as that could cause security concerns. -
Run
kubectl proxyto start the proxy. This can be run from a screen or tmux session if you want to run it in the background.╭─jivanov@centos-kubernetes-01 ~ ╰─$ kubectl proxy Starting to serve on 127.0.0.1:8001 -
Your Kubernetes API is now accessible to your local machine's loopback interface on port 8001 (or whatever port you decided to use). If you setup the dashboard, then you can access http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ otherwise the full Kubernetes is at your disposal over a secure tunnel.
- Commands have moved to Kubernetes.Commands.Troubleshooting.md.