- pods are the base unit of operation for Kubernetes. Pods are the most visible, viable, and ephemeral units that comprise one or more tightly coupled containers. Kubernetes does not operate at the level of containers. There can be multiple pods in a single server node and data sharing easily happens in between pods. Kubernetes automatically provision and allocate pods for various services. Each pod has its own IP address and shares the localhost and volumes.
- Labels are typically the metadata that is attached to objects, including pods.
- Services offer a low-overhead way to route all kinds of service requests to a set of pods to accomplish the requests. Read:
- https://docs.google.com/document/d/1E03-g0h3MgFlohXqPNYCjl7Pv_CA_l0zhD6K3MZ8O0M/edit?usp=sharing
- https://www.freecodecamp.org/news/learn-kubernetes-in-under-3-hours-a-detailed-guide-to-orchestrating-containers-114ff420e882/
- https://neuvector.com/network-security/advanced-kubernetes-networking/
- https://layer5.io/landscape/
- https://www.tfir.io/2019/05/17/whats-the-right-ingress-controller-for-my-kubernetes-environment/
The currently running massive and monolithic applications get modernized and migrated to cloud environments to reap the distinct benefits of the cloud paradigm
This is all about designing, developing, debugging, delivering, and deploying applications directly on cloud environments by intrinsically leveraging the non-functional capabilities of cloud environments
Microservices are the architecture paradigm wherein a monolithic (silo) application is decomposed into small tiny micro applications which are packaged and deployed independently. Read:
- https://martinfowler.com/microservices/
- https://www.zeolearn.com/magazine/microservices-architecture
- https://microservices.io/patterns/microservices.html
- https://docs.microsoft.com/en-us/azure/architecture/microservices/design/data-considerations
- http://www.mammatustech.com/high-speed-microservices
- https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md
- High cohesion among services: A microservice should have one single focus and the sole responsibility for that action. It should not change as a result of other related services. Services should be easily rewritable so that we can achieve scalability, reliability, and flexibility. It should handle a single business function and domain-specific functionality.
- Autonomous service: A service should independently handle its work without the help of any other services. It should not be tightly integrated with any other service; it should remain loosely coupled in nature. By autonomous, we mean that a microservice should not change because of the external components with which it interacts. Autonomous services honor contracts and interfaces. They should be stateless, independently changeable, independently deployable, backwards compatible, and they should support concurrent development.
- Business domain-centric service: Each individual service should perform or represent a single business function. This could be a calculation of sales, tax, income tax, or any other function related to a specific area. Each service should bound or define its scope. Business-centric code can help to provide more cohesion and make services more responsive to handle any changes in the domain or business logic requirements.
- Resilience: Resilience is a standard these days when providing a service to a customer. Failure to provide resilience may result in another endpoint not providing a response to your microservices. Designing your service in microformat helps to overcome failure. Our service should register itself during startup and de-register itself upon failure. This should be part of a dynamic discovery service, such as the auto-creation of a queue or the auto-removal of the queue in a message queue. There could be a number of problems or exceptions that a network-based service could encounter. It should be able to handle delays and the unavailability of another service.
- Observable service or functionality: Observability is another important design principle while working on distributed microservices. When a complex interconnected service breaks, it can take hours or days to isolate issues. We should design our services in such a way that we can check the health of any service by either showing its status on a health page or by sending it to a central logging service such as Splunk, Logstash, syslogd, Logentries, Datadog, or Sumo Logic. Observability is required to support reliable, scalable, and cost-effective services and metrics to scale up, metrics to scale down, and metrics to alert the team. This kind of monitoring and logging needs to be located at a central place. In a containerized environment, auto deployment should be able to auto-detect when a deployment fails so that it can be rolled back quickly to an older running version. Observability can be related to CPU usage, memory usage, network input/output metrics, disk metrics, the number of connections to a service, and so on. All these metrics are easily available and measurable through tools such as Check_MK, Nagios, New Relic, AppDynamics, StatsD, and Graphana.
- Observability not only helps in terms of providing a technical solution but also so that we can identify business decision-making, like the sales of a specific service or the returns for a specific product. Automation: Microservices also create challenges for an operation team with regard to deployment, verifying functionality, and performing various types of testing. There is now a wide range of automation tools available on market that can easily be integrated to achieve automated deployment, verification, testing, failure, and rollbacks. Some of the famous tools are Jenkins; Teamcity; Bamboo; Git workflow plugins; GitLab CI/CD; UI test tools, such as Selenium, PhantomJS, Nightwatch, BrowserStack; and many more. One important point here is that while Docker changed the container market when it was developed, it was hard to implement in a production environment, where a complete stack is required to maintain it as a production-level service. There wasn't much clarity with regard to monitoring or deployment. After that, Google released Borg in the form of Kubernetes and changed the container market again by providing easy deployment and rollback options with easy service and routing functionalities that were perfect for production-grade deployments.
NOTE:
- Microservices prefer letting each service manage its own database The key point it that the services should have no knowledge of each other's underlying database. https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md#service-essentials-2
- Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP.
- Services can be developed and deployed independently of one another.
- In the microservices world, the network is your biggest point of failure. In production you will fail if too much connections between. https://medium.com/@oprearocks/blasphemy-multiple-microservices-shared-database-f525025a8a81
- 'micro' part doesn't mean replace every class with a network service, but componentize a monolithic application into sensibly sized components, each one dealing with an aspect of your program.
- use In-memory service data
git clone https://github.com/bagder/curl.git
# requires https://askubuntu.com/a/826891
sudo apt-get build-dep curl
cd curl
./buildconf
./configure --with-ssl --disable-shared
make -j8
# sudo make uninstall
sudo make install
curl -V
# Features: ... SSL ... TLS-SRP ...
Follow https://www.virtualbox.org/wiki/Downloads
Follow https://phoenixnap.com/kb/how-to-install-docker-on-ubuntu-18-04
Or under proxy, see https://gist.github.com/blockspacer/893b31e61c88f6899ffd0813111b3e41
Follow https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version --client
OR
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s http://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
Follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
curl -SSL https://dl.k8s.io/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubeadm > ./kubeadm
chmod a+rx kubeadm
sudo mv kubeadm /usr/bin/kubeadm
kubeadm version
If under proxy - See https://wiki.christophchamp.com/index.php?title=Kubernetes/the-hard-way
Follow https://kubernetes.io/docs/setup/learning-environment/minikube/#installation (For Windows: https://meteatamel.wordpress.com/2018/02/14/minikube-on-windows/)
curl -Lo minikube http://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
&& chmod +x minikube
sudo install minikube /usr/local/bin
minikube status
minikube stop
# OR minikube delete
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
--extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
--extra-config=apiserver.authorization-mode=RBAC \
--insecure-registry='localhost' \
--insecure-registry='127.0.0.1' \
--insecure-registry "192.168.39.0/24"
# OPTIONAL: open dashboard
minikube addons enable dashboard && kubectl get pods --all-namespaces | grep dashboard && sleep 15 && minikube dashboard
kubectl config use-context minikube
NOTE: minikube start --alsologtostderr --memory=16384 --cpus=4 --disk-size 25GB with 16384 MB of memory and 4 CPUs. 16384 MB is sufficent to run Istio and bookinfo.
# (Optional, recommended) If you want minikube to provide a load balancer for use by Istio, you can use the minikube tunnel feature. Run this command in a different terminal, because the minikube tunnel feature will block your terminal to output diagnostic information about the network:
minikube tunnel
# Sometimes minikube does not clean up the tunnel network properly. To force a proper cleanup:
minikube tunnel --cleanupNOTE: Use may want to run minikube delete before minikube start
NOTE: you can set proxy by --docker-env:
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
--extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
--extra-config=apiserver.authorization-mode=RBAC \
--insecure-registry='localhost' \
--insecure-registry='127.0.0.1' \
--insecure-registry "192.168.39.0/24" \
--docker-env http_proxy=http://172.17.0.1:3128 \
--docker-env https_proxy=http://172.17.0.1:3128 \
--docker-env no_proxy=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru \
--docker-env HTTP_PROXY=http://172.17.0.1:3128 \
--docker-env HTTPS_PROXY=http://172.17.0.1:3128 \
--docker-env NO_PROXY=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru
# add corp certs to ~/.minikube/files/etc/ssl/certs
cp -r /etc/ssl/certs/* ~/.minikube/files/etc/ssl/certs
cp -r /usr/share/ca-certificates/* ~/.minikube/files/etc/ssl/certs
# cert files must have 750 permission
sudo chmod 750 ~/.minikube/files/etc/ssl/certs/*.crt
Now you can ssh into minikube and change daemon.json according to your proxy
minikube ssh
sudo mkdir -p /etc/docker
# NOTE: can't set "insecure-registries" due to `--insecure-registry` minikube arg
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"dns": ["127.0.0.53", "10.8.13.11", "10.8.13.12", "8.8.4.4", "8.8.8.8", "10.8.13.11", "10.8.13.12"],
"registry-mirrors":["https://docker.mirrors.ustc.edu.cn"],
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
}
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
# test pull under minikube
# docker pull grafana/grafana:6.4.3
exit
# see http://rastko.tech/kubernetes/2019/01/01/minikube-on-mac.html
minikube addons enable ingress
# Use `--insecure-registry='192.168.39.0/24'`
# see https://minikube.sigs.k8s.io/docs/tasks/docker_registry/
minikube addons enable registryFollow on linux https://istio.io/docs/setup/platform-setup/minikube/
# monitor memory usage in minikube
minikube ssh -- sudo topFollow on linux https://istio.io/docs (For Windows: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/istio-install.md#windows)
curl -L https://git.io/getLatestIstio | sh -
cd istio*
echo 'export PATH=$(pwd)/bin:$PATH' >> ~/.bashrc
# OR
# sudo cp ./bin/istioctl /usr/local/bin/istioctl
# sudo chmod +x /usr/local/bin/istioctl
istioctl version
Istio releases http://gcsweb.istio.io/gcs/istio-release/releases/1.4.3/
Follow https://istio.io/docs/setup/install/istioctl/
istioctl manifest apply --skip-confirmation
# OR
# istioctl manifest apply --set values.global.mtls.enabled=true,values.security.selfSigned=false --set values.global.controlPlaneSecurityEnabled=true
# see https://istio.io/docs/setup/getting-started/
kubectl get svc -n istio-system
# ensure corresponding Kubernetes pods are deployed and have a STATUS of Running
kubectl get pods -n istio-systemMake sure that all istio-system pods are running or completed before continuing. This can take several minutes when starting the pods for the first time. Be patient.
enable automatic sidecar injection:
kubectl label namespace default istio-injection=enabled
if you want to exclude a specific pod from getting istio sidecar injected, add this to
Deploymentkind
metadata:
annotations:
sidecar.istio.io/inject: "false"(see https://github.com/IBM/istio101/blob/master/workshop/exercise-2/README.md or https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing)
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo.yaml
kubectl get svc -n istio-system
kubectl get pods -n istio-system
NOTE: How to build docker images which will be seen by Kubernetes directly without having to push them anywhere
just run
# NOTE: Later, when we no longer wish to use the Minikube host, we can undo this change by running: eval $(minikube docker-env -u)
eval $(minikube docker-env)and now you can build docker images which will be seen by Kubernetes directly without having to push them anywhere.
istioctl dashboard controlz # Open ControlZ web UI
istioctl dashboard envoy # Open Envoy admin web UI
istioctl dashboard grafana # Open Grafana web UI
istioctl dashboard jaeger # Open Jaeger web UI
istioctl dashboard kiali # Open Kiali web UI
istioctl dashboard prometheus # Open Prometheus web UI
istioctl dashboard zipkin # Open Zipkin web UI
See https://istio.io/docs/reference/commands/istioctl/#istioctl-dashboard
# in Istio >= 1.4.x
istioctl dashboard jaegerOR if Istio < 1.4.x:
kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686URL to Open Jaeger: http://localhost:16686
# in Istio >= 1.4.x
istioctl dashboard grafanaOR if Istio < 1.4.x:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000 &URL to open Grafana: http://localhost:3000/dashboard/db/istio-mesh-dashboard
# in Istio >= 1.4.x
istioctl dashboard prometheusOR if Istio < 1.4.x:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &URL to open Prometheus: http://localhost:9090
Kiali is Istio’s dashboard and this is one of the coolest features in 1.4.x: To open the Kiali dashboard you no longer need to execute complicated port-forwarding commands, simply type
# in Istio >= 1.4.x
istioctl dashboard kialiOR if Istio < 1.4.x:
Run the following command to install Kiali:
bash <(curl -L http://git.io/getLatestKialiKubernetes)Note: For some reason the script didn't work for me. I had to replace one line:
get_downloader
github_api_url="https://api.github.com/repos/kiali/kiali/releases/latest"
kiali_version_we_want="v0.15.0"To launch Kiali you need the IP address and NodePort:
minikube ip
kubectl get svc -n istio-system kiali --output 'jsonpath={.spec.ports[*].nodePort}'URL to open Kiali: https://[minikube-ip]:[kiali-nodeport]/kiali
istioctl verify-installInstall nodejs
# sudo apt remove node npm
# sudo rm /usr/local/bin/node # must be removed
# sudo rm /usr/local/bin/npm # must be removed
# sudo rm -rf /usr/local/lib/node_modules # must be removed
OS_ARCH=x64 # $(uname -m)
NODE_V=v10.18.1
wget https://nodejs.org/dist/$NODE_V/node-$NODE_V-linux-$OS_ARCH.tar.gz
tar -xvf node-$NODE_V-linux-$OS_ARCH.tar.gz
cd node-$NODE_V-linux-$OS_ARCH
sudo cp -R * /usr/local/
sudo chown -R $USER /usr/local/lib/node_modules
cd -
# npm install npm -g # optional
node -v
npm -vUnder proxy: you may want to configure ~/.npmrc like so https://stackoverflow.com/a/36929934
# NOTE: you may want to use NODE_TLS_REJECT_UNAUTHORIZED=0 under proxy during `npm install`
NODE_TLS_REJECT_UNAUTHORIZED=0 \
HTTP_PROXY=http://127.0.0.1:8088 \
HTTPS_PROXY=http://127.0.0.1:8088 \
npm install \
--unsafe-perm bindingInstall protobuf from sources https://developers.google.com/protocol-buffers/docs/downloads and (if exists) remove old protobuf version apt-get remove libprotobuf-dev
NOTE: it is better to clone https://github.com/grpc/grpc/ repo and build protobuf from grpc/third_party/protobuf
Tested with GRPC_RELEASE_TAG=v1.26.x
python -V # Python 2.7 or newer
sudo apt-get install autoconf automake libtool curl make g++ unzip
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./configure --prefix=/usr
make
make check
sudo make install
sudo ldconfig # refresh shared library cache.
protoc --versionInstall Protocol Buffers for Go https://github.com/golang/protobuf#installation
Install grpc (requres protobuf) https://github.com/grpc/grpc/blob/master/BUILDING.md
Install Helm https://helm.sh/docs/intro/install/
mkdir /tmp/helm
cd /tmp/helm
helm_version=v3.0.2-linux-amd64
wget https://get.helm.sh/helm-$helm_version.tar.gz
tar zxvf helm-$helm_version.tar.gz
sudo mv linux-amd64/helm /bin/helm
cd -
helm version
helm repo add stable https://kubernetes-charts.storage.googleapis.comsee https://github.com/ruzickap/k8s-istio-demo#install-helm
see https://github.com/ruzickap/k8s-istio-demo#install-rook
see https://github.com/ruzickap/k8s-istio-demo#install-elasticsearch-kibana-fluentbit
function _out() {
echo "$(date +'%F %H:%M:%S') $@"
}
function checkPrerequisites() {
MISSING_TOOLS=""
git --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} git"
curl --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} curl"
which sed &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} sed"
docker -v &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} docker"
unzip -version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} unzip"
kubectl version --client=true &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} kubectl"
minikube version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} minikube"
if [[ -n "$MISSING_TOOLS" ]]; then
_out "Some tools (${MISSING_TOOLS# }) could not be found, please install them first"
exit 1
else
_out You have all necessary prerequisites installed
fi
if ! kubectl describe namespace default | grep istio-injection=enabled > /dev/null ; then
_out "Istio automatic sidecar injection needs to be enabled. See documentation/SetupLocalEnvironment.md"
fi
}
checkPrerequisiteshttps://blogs.vmware.com/opensource/2019/04/16/implementing-grpc-web-istio-envoy/ https://github.com/lucperkins/colossus/blob/1a487d9768d80094939a8171645227b64fd76b67/userinfo/userinfo-server.cc https://grpc.io/blog/state-of-grpc-web/ https://venilnoronha.io/seamless-cloud-native-apps-with-grpc-web-and-istio & https://github.com/venilnoronha/grpc-web-istio-demo https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing & https://github.com/saturnism/istio-by-example-java https://github.com/grpc/grpc/tree/master/examples/cpp/load_balancing https://istiobyexample.dev/grpc/
https://github.com/IBM/istio101/blob/master/workshop/exercise-3/README.md https://meteatamel.wordpress.com/2018/04/24/istio-101-with-minikube/
https://rinormaloku.com/authorization-in-istio/
https://github.com/castlemilk/kubernetes-cicd/tree/8505444338567cfaad5ab61496ad14da4f578657
https://gameontext.gitbooks.io/gameon-gitbook/content/microservices/
https://github.com/gameontext/gameon/blob/master/Vagrantfile#L62
https://vietnam-devs.github.io/coolstore-microservices/#coolstore-website
- https://github.com/srinandan/istio-workshop/tree/7f8ebcebcc71bb0948ff8803444a70af1ef133f6/misc/websocket
- https://github.com/vietnam-devs/coolstore-microservices/blob/f41b4957d2f4fb4201624f21a54f33ad4ea695b0/deploys/charts/coolstore-istio/templates/api-vs.yaml
- https://github.com/gameontext/gameon/blob/e8b424e531f531a6b14226326ea236363c379a81/kubernetes/istio/istio-gatewayVirtualService.yaml
- https://github.com/MatthieuSegret/yummy-phoenix-graphql/blob/f0b258293697b0b120ef8e8a3b3905043c998617/kubernetes/yummy/templates/networking/virtual-service-ws.yaml#L17
- https://github.com/istio/istio/tree/master/samples/websockets
- https://github.com/mukundha/istio-apigee-samples/blob/918fb72742ac20ac8a656362621b4ca15f39ba31/istio-manifests/tls-gateway.yaml#L33
- https://github.com/DeerNation/deployment/blob/c2ef67e1004d39f48fd1b8ad7d4cdcfc5bcde230/kubernetes/virtual-services.yaml
- https://medium.com/12-developer-labors/angular-chat-using-kubernetes-with-websockets-bb4d87bfe99a
- https://habr.com/ru/post/351012/
- https://hub.docker.com/r/ageapps/docker-chat/
- https://medium.com/@faiyaz26/deploying-a-real-time-notification-system-on-kubernetes-part-1-e64af5c93a2b
NOTE: Deprecated. Websocket upgrades are done automatically starting from Istio 1.0.
read docs
https://github.com/solo-io/solo-docs/blob/9f9ba476fab0803fa7a7b2d74b26e3a1f7c3a325/supergloo/docs/v1/github.com/solo-io/supergloo/api/external/istio/networking/v1alpha3/virtual_service.proto.sk.md
https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/