-
-
Save blockspacer/0237dcdea981f78f8c3ea80b8b1d037d to your computer and use it in GitHub Desktop.
| # About Docker | |
| + https://github.com/katopz/best-practices#docker | |
| # About Kubernetes | |
| + pods are the base unit of operation for Kubernetes. Pods are the most visible, viable, and ephemeral units | |
| that comprise one or more tightly coupled containers. Kubernetes does not operate | |
| at the level of containers. There can be multiple pods in a single server node and | |
| data sharing easily happens in between pods. Kubernetes automatically | |
| provision and allocate pods for various services. Each pod has its own IP address | |
| and shares the localhost and volumes. | |
| + Labels are typically the metadata that is attached to objects, including pods. | |
| + Services offer a low-overhead way to route all kinds of service requests to a set of pods to accomplish the requests. | |
| Read: | |
| + https://docs.google.com/document/d/1E03-g0h3MgFlohXqPNYCjl7Pv_CA_l0zhD6K3MZ8O0M/edit?usp=sharing | |
| + https://www.freecodecamp.org/news/learn-kubernetes-in-under-3-hours-a-detailed-guide-to-orchestrating-containers-114ff420e882/ | |
| + https://neuvector.com/network-security/advanced-kubernetes-networking/ | |
| + https://layer5.io/landscape/ | |
| + https://www.tfir.io/2019/05/17/whats-the-right-ingress-controller-for-my-kubernetes-environment/ | |
| # About Service Mesh | |
| + https://www.slideshare.net/dbryant_uk/microxchg-2018-what-is-a-service-mesh-do-i-need-one-when-developing-cloud-native-microservices | |
| # About Cloud-enabled apps | |
| The currently running massive and monolithic applications get | |
| modernized and migrated to cloud environments to reap the distinct benefits of | |
| the cloud paradigm | |
| # About Cloud-native apps | |
| This is all about designing, developing, debugging, delivering, | |
| and deploying applications directly on cloud environments by intrinsically | |
| leveraging the non-functional capabilities of cloud environments | |
| # About microservices architecture (MSA) | |
| Microservices are the architecture paradigm wherein a monolithic (silo) application is decomposed into small tiny micro applications which are packaged and deployed independently. | |
| Read: | |
| + https://martinfowler.com/microservices/ | |
| + https://www.zeolearn.com/magazine/microservices-architecture | |
| + https://microservices.io/patterns/microservices.html | |
| + https://docs.microsoft.com/en-us/azure/architecture/microservices/design/data-considerations | |
| + http://www.mammatustech.com/high-speed-microservices | |
| + https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md | |
| # Microservice design principles | |
| + High cohesion among services: A microservice should have one single focus and | |
| the sole responsibility for that action. It should not change as a result of other | |
| related services. Services should be easily rewritable so that we can achieve | |
| scalability, reliability, and flexibility. It should handle a single business function | |
| and domain-specific functionality. | |
| + Autonomous service: A service should independently handle its work without | |
| the help of any other services. It should not be tightly integrated with any other | |
| service; it should remain loosely coupled in nature. By autonomous, we mean | |
| that a microservice should not change because of the external components with | |
| which it interacts. Autonomous services honor contracts and interfaces. They | |
| should be stateless, independently changeable, independently deployable, | |
| backwards compatible, and they should support concurrent development. | |
| + Business domain-centric service: Each individual service should perform or | |
| represent a single business function. This could be a calculation of sales, tax, | |
| income tax, or any other function related to a specific area. Each service should | |
| bound or define its scope. Business-centric code can help to provide more | |
| cohesion and make services more responsive to handle any changes in the | |
| domain or business logic requirements. | |
| + Resilience: Resilience is a standard these days when providing a service to a | |
| customer. Failure to provide resilience may result in another endpoint not | |
| providing a response to your microservices. Designing your service in microformat helps to overcome failure. Our service should register itself during | |
| startup and de-register itself upon failure. This should be part of a dynamic | |
| discovery service, such as the auto-creation of a queue or the auto-removal of the | |
| queue in a message queue. There could be a number of problems or exceptions | |
| that a network-based service could encounter. It should be able to handle delays | |
| and the unavailability of another service. | |
| + Observable service or functionality: Observability is another important design | |
| principle while working on distributed microservices. When a complex | |
| interconnected service breaks, it can take hours or days to isolate issues. We | |
| should design our services in such a way that we can check the health of any | |
| service by either showing its status on a health page or by sending it to a central | |
| logging service such as Splunk, Logstash, syslogd, Logentries, Datadog, or Sumo | |
| Logic. Observability is required to support reliable, scalable, and cost-effective | |
| services and metrics to scale up, metrics to scale down, and metrics to alert the | |
| team. This kind of monitoring and logging needs to be located at a central place. | |
| In a containerized environment, auto deployment should be able to auto-detect | |
| when a deployment fails so that it can be rolled back quickly to an older running | |
| version. Observability can be related to CPU usage, memory usage, network | |
| input/output metrics, disk metrics, the number of connections to a service, and so | |
| on. All these metrics are easily available and measurable through tools such as | |
| Check_MK, Nagios, New Relic, AppDynamics, StatsD, and Graphana. | |
| + Observability not only helps in terms of providing a technical solution but also so | |
| that we can identify business decision-making, like the sales of a specific service | |
| or the returns for a specific product. | |
| Automation: Microservices also create challenges for an operation team with | |
| regard to deployment, verifying functionality, and performing various types of | |
| testing. There is now a wide range of automation tools available on market that | |
| can easily be integrated to achieve automated deployment, verification, testing, | |
| failure, and rollbacks. Some of the famous tools are Jenkins; Teamcity; Bamboo; | |
| Git workflow plugins; GitLab CI/CD; UI test tools, such as Selenium, PhantomJS, | |
| Nightwatch, BrowserStack; and many more. One important point here is that | |
| while Docker changed the container market when it was developed, it was hard | |
| to implement in a production environment, where a complete stack is required to | |
| maintain it as a production-level service. There wasn't much clarity with regard | |
| to monitoring or deployment. After that, Google released Borg in the form of | |
| Kubernetes and changed the container market again by providing easy | |
| deployment and rollback options with easy service and routing functionalities | |
| that were perfect for production-grade deployments. | |
| NOTE: | |
| + **Microservices prefer letting each service manage its own database** The key point it that the services should have no knowledge of each other's underlying database. https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md#service-essentials-2 | |
| + Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP. | |
| + Services can be developed and deployed independently of one another. | |
| + In the microservices world, the network is your biggest point of failure. In production you will fail if too much connections between. https://medium.com/@oprearocks/blasphemy-multiple-microservices-shared-database-f525025a8a81 | |
| + 'micro' part doesn't mean replace every class with a network service, but componentize a monolithic application into sensibly sized components, each one dealing with an aspect of your program. | |
| + use In-memory service data | |
| # Install curl with https support | |
| ``` | |
| git clone https://github.com/bagder/curl.git | |
| # requires https://askubuntu.com/a/826891 | |
| sudo apt-get build-dep curl | |
| cd curl | |
| ./buildconf | |
| ./configure --with-ssl --disable-shared | |
| make -j8 | |
| # sudo make uninstall | |
| sudo make install | |
| curl -V | |
| # Features: ... SSL ... TLS-SRP ... | |
| ``` | |
| # Install VirtualBox | |
| Follow https://www.virtualbox.org/wiki/Downloads | |
| # Install docker | |
| Follow https://phoenixnap.com/kb/how-to-install-docker-on-ubuntu-18-04 | |
| Or under proxy, see https://gist.github.com/blockspacer/893b31e61c88f6899ffd0813111b3e41 | |
| # Install kubectl | |
| Follow https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux | |
| ``` | |
| curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl | |
| chmod +x ./kubectl | |
| sudo mv ./kubectl /usr/local/bin/kubectl | |
| kubectl version --client | |
| ``` | |
| OR | |
| ``` | |
| sudo apt-get update && sudo apt-get install -y apt-transport-https | |
| curl -s http://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - | |
| echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list | |
| sudo apt-get update | |
| sudo apt-get install -y kubectl | |
| ``` | |
| # Install kubeadm | |
| Follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ | |
| ``` | |
| curl -SSL https://dl.k8s.io/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubeadm > ./kubeadm | |
| chmod a+rx kubeadm | |
| sudo mv kubeadm /usr/bin/kubeadm | |
| kubeadm version | |
| ``` | |
| ## Under proxy - setup Kubernetes The Hard Way | |
| If under proxy - See https://wiki.christophchamp.com/index.php?title=Kubernetes/the-hard-way | |
| # Install Minikube | |
| Follow https://kubernetes.io/docs/setup/learning-environment/minikube/#installation | |
| (For Windows: https://meteatamel.wordpress.com/2018/02/14/minikube-on-windows/) | |
| ``` | |
| curl -Lo minikube http://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \ | |
| && chmod +x minikube | |
| sudo install minikube /usr/local/bin | |
| minikube status | |
| minikube stop | |
| # OR minikube delete | |
| minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \ | |
| --extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \ | |
| --extra-config=apiserver.authorization-mode=RBAC \ | |
| --insecure-registry='localhost' \ | |
| --insecure-registry='127.0.0.1' \ | |
| --insecure-registry "192.168.39.0/24" | |
| # OPTIONAL: open dashboard | |
| minikube addons enable dashboard && kubectl get pods --all-namespaces | grep dashboard && sleep 15 && minikube dashboard | |
| ``` | |
| ``` | |
| kubectl config use-context minikube | |
| ``` | |
| **NOTE: `minikube start --alsologtostderr --memory=16384 --cpus=4 --disk-size 25GB` with 16384 MB of memory and 4 CPUs. 16384 MB is sufficent to run Istio and bookinfo.** | |
| ```bash | |
| # (Optional, recommended) If you want minikube to provide a load balancer for use by Istio, you can use the minikube tunnel feature. Run this command in a different terminal, because the minikube tunnel feature will block your terminal to output diagnostic information about the network: | |
| minikube tunnel | |
| # Sometimes minikube does not clean up the tunnel network properly. To force a proper cleanup: | |
| minikube tunnel --cleanup | |
| ``` | |
| **NOTE: Use may want to run `minikube delete` before `minikube start`** | |
| **NOTE: you can set proxy by `--docker-env`:** | |
| ``` | |
| minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \ | |
| --extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \ | |
| --extra-config=apiserver.authorization-mode=RBAC \ | |
| --insecure-registry='localhost' \ | |
| --insecure-registry='127.0.0.1' \ | |
| --insecure-registry "192.168.39.0/24" \ | |
| --docker-env http_proxy=http://172.17.0.1:3128 \ | |
| --docker-env https_proxy=http://172.17.0.1:3128 \ | |
| --docker-env no_proxy=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru \ | |
| --docker-env HTTP_PROXY=http://172.17.0.1:3128 \ | |
| --docker-env HTTPS_PROXY=http://172.17.0.1:3128 \ | |
| --docker-env NO_PROXY=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru | |
| # add corp certs to ~/.minikube/files/etc/ssl/certs | |
| cp -r /etc/ssl/certs/* ~/.minikube/files/etc/ssl/certs | |
| cp -r /usr/share/ca-certificates/* ~/.minikube/files/etc/ssl/certs | |
| # cert files must have 750 permission | |
| sudo chmod 750 ~/.minikube/files/etc/ssl/certs/*.crt | |
| ``` | |
| Now you can ssh into minikube and change `daemon.json` according to your proxy | |
| ``` | |
| minikube ssh | |
| sudo mkdir -p /etc/docker | |
| # NOTE: can't set "insecure-registries" due to `--insecure-registry` minikube arg | |
| sudo tee /etc/docker/daemon.json <<-'EOF' | |
| { | |
| "dns": ["127.0.0.53", "10.8.13.11", "10.8.13.12", "8.8.4.4", "8.8.8.8", "10.8.13.11", "10.8.13.12"], | |
| "registry-mirrors":["https://docker.mirrors.ustc.edu.cn"], | |
| "log-driver": "json-file", | |
| "log-opts": { | |
| "max-size": "50m", | |
| "max-file": "3" | |
| } | |
| } | |
| EOF | |
| sudo systemctl daemon-reload | |
| sudo systemctl restart docker | |
| # test pull under minikube | |
| # docker pull grafana/grafana:6.4.3 | |
| exit | |
| ``` | |
| ## Enable minikube addons | |
| ```bash | |
| # see http://rastko.tech/kubernetes/2019/01/01/minikube-on-mac.html | |
| minikube addons enable ingress | |
| # Use `--insecure-registry='192.168.39.0/24'` | |
| # see https://minikube.sigs.k8s.io/docs/tasks/docker_registry/ | |
| minikube addons enable registry | |
| ``` | |
| ## Prepere minikube for istio | |
| Follow on linux https://istio.io/docs/setup/platform-setup/minikube/ | |
| ```bash | |
| # monitor memory usage in minikube | |
| minikube ssh -- sudo top | |
| ``` | |
| # Install istioctl | |
| Follow on linux https://istio.io/docs | |
| (For Windows: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/istio-install.md#windows) | |
| ``` | |
| curl -L https://git.io/getLatestIstio | sh - | |
| cd istio* | |
| echo 'export PATH=$(pwd)/bin:$PATH' >> ~/.bashrc | |
| # OR | |
| # sudo cp ./bin/istioctl /usr/local/bin/istioctl | |
| # sudo chmod +x /usr/local/bin/istioctl | |
| istioctl version | |
| ``` | |
| Istio releases http://gcsweb.istio.io/gcs/istio-release/releases/1.4.3/ | |
| ## Apply istio profile | |
| Follow https://istio.io/docs/setup/install/istioctl/ | |
| ```bash | |
| istioctl manifest apply --skip-confirmation | |
| # OR | |
| # istioctl manifest apply --set values.global.mtls.enabled=true,values.security.selfSigned=false --set values.global.controlPlaneSecurityEnabled=true | |
| # see https://istio.io/docs/setup/getting-started/ | |
| kubectl get svc -n istio-system | |
| # ensure corresponding Kubernetes pods are deployed and have a STATUS of Running | |
| kubectl get pods -n istio-system | |
| ``` | |
| Make sure that all `istio-system` pods are running or completed before continuing. This can take several minutes when starting the pods for the first time. Be patient. | |
| enable automatic sidecar injection: | |
| ``` | |
| kubectl label namespace default istio-injection=enabled | |
| ``` | |
| > if you want to exclude a specific pod from getting istio sidecar injected, add this to `Deployment` kind | |
| ```yaml | |
| metadata: | |
| annotations: | |
| sidecar.istio.io/inject: "false" | |
| ``` | |
| # Add istioctl to PATH | |
| (see https://github.com/IBM/istio101/blob/master/workshop/exercise-2/README.md or https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing) | |
| ``` | |
| for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done | |
| kubectl apply -f install/kubernetes/istio-demo.yaml | |
| kubectl get svc -n istio-system | |
| kubectl get pods -n istio-system | |
| ``` | |
| ## NOTE: How to build docker images which will be seen by Kubernetes directly without having to push them anywhere | |
| just run | |
| ```bash | |
| # NOTE: Later, when we no longer wish to use the Minikube host, we can undo this change by running: eval $(minikube docker-env -u) | |
| eval $(minikube docker-env) | |
| ``` | |
| and now you can build docker images which will be seen by Kubernetes directly without having to push them anywhere. | |
| ## Open istio dashboards | |
| ```bash | |
| istioctl dashboard controlz # Open ControlZ web UI | |
| istioctl dashboard envoy # Open Envoy admin web UI | |
| istioctl dashboard grafana # Open Grafana web UI | |
| istioctl dashboard jaeger # Open Jaeger web UI | |
| istioctl dashboard kiali # Open Kiali web UI | |
| istioctl dashboard prometheus # Open Prometheus web UI | |
| istioctl dashboard zipkin # Open Zipkin web UI | |
| ``` | |
| See https://istio.io/docs/reference/commands/istioctl/#istioctl-dashboard | |
| ## NOTE: How to run Jaeger Dashboard | |
| ```bash | |
| # in Istio >= 1.4.x | |
| istioctl dashboard jaeger | |
| ``` | |
| OR if Istio < 1.4.x: | |
| ```bash | |
| kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686 | |
| ``` | |
| URL to Open Jaeger: http://localhost:16686 | |
| ## NOTE: How to run Grafana Dashboard | |
| ```bash | |
| # in Istio >= 1.4.x | |
| istioctl dashboard grafana | |
| ``` | |
| OR if Istio < 1.4.x: | |
| ```bash | |
| kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000 & | |
| ``` | |
| URL to open Grafana: http://localhost:3000/dashboard/db/istio-mesh-dashboard | |
| ## NOTE: How to run Prometheus Dashboard | |
| ```bash | |
| # in Istio >= 1.4.x | |
| istioctl dashboard prometheus | |
| ``` | |
| OR if Istio < 1.4.x: | |
| ```bash | |
| kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 & | |
| ``` | |
| URL to open Prometheus: http://localhost:9090 | |
| ## NOTE: How to run Kiali | |
| Kiali is Istio’s dashboard and this is one of the coolest features in 1.4.x: To open the Kiali dashboard you no longer need to execute complicated port-forwarding commands, simply type | |
| ```bash | |
| # in Istio >= 1.4.x | |
| istioctl dashboard kiali | |
| ``` | |
| OR if Istio < 1.4.x: | |
| Run the following command to install Kiali: | |
| ```bash | |
| bash <(curl -L http://git.io/getLatestKialiKubernetes) | |
| ``` | |
| Note: For some reason the script didn't work for me. I had to replace one line: | |
| ```bash | |
| get_downloader | |
| github_api_url="https://api.github.com/repos/kiali/kiali/releases/latest" | |
| kiali_version_we_want="v0.15.0" | |
| ``` | |
| To launch Kiali you need the IP address and NodePort: | |
| ```bash | |
| minikube ip | |
| kubectl get svc -n istio-system kiali --output 'jsonpath={.spec.ports[*].nodePort}' | |
| ``` | |
| URL to open Kiali: https://[minikube-ip]:[kiali-nodeport]/kiali | |
| ## Run `istioctl verify-install` | |
| ```bash | |
| istioctl verify-install | |
| ``` | |
| # Dev tools | |
| Install nodejs | |
| ```bash | |
| # sudo apt remove node npm | |
| # sudo rm /usr/local/bin/node # must be removed | |
| # sudo rm /usr/local/bin/npm # must be removed | |
| # sudo rm -rf /usr/local/lib/node_modules # must be removed | |
| OS_ARCH=x64 # $(uname -m) | |
| NODE_V=v10.18.1 | |
| wget https://nodejs.org/dist/$NODE_V/node-$NODE_V-linux-$OS_ARCH.tar.gz | |
| tar -xvf node-$NODE_V-linux-$OS_ARCH.tar.gz | |
| cd node-$NODE_V-linux-$OS_ARCH | |
| sudo cp -R * /usr/local/ | |
| sudo chown -R $USER /usr/local/lib/node_modules | |
| cd - | |
| # npm install npm -g # optional | |
| node -v | |
| npm -v | |
| ``` | |
| Under proxy: you may want to configure `~/.npmrc` like so https://stackoverflow.com/a/36929934 | |
| ```bash | |
| # NOTE: you may want to use NODE_TLS_REJECT_UNAUTHORIZED=0 under proxy during `npm install` | |
| NODE_TLS_REJECT_UNAUTHORIZED=0 \ | |
| HTTP_PROXY=http://127.0.0.1:8088 \ | |
| HTTPS_PROXY=http://127.0.0.1:8088 \ | |
| npm install \ | |
| --unsafe-perm binding | |
| ``` | |
| Install protobuf from sources https://developers.google.com/protocol-buffers/docs/downloads and (if exists) remove old protobuf version `apt-get remove libprotobuf-dev` | |
| NOTE: it is better to clone https://github.com/grpc/grpc/ repo and build protobuf from `grpc/third_party/protobuf` | |
| Tested with `GRPC_RELEASE_TAG=v1.26.x` | |
| ```bash | |
| python -V # Python 2.7 or newer | |
| sudo apt-get install autoconf automake libtool curl make g++ unzip | |
| git clone https://github.com/protocolbuffers/protobuf.git | |
| cd protobuf | |
| git submodule update --init --recursive | |
| ./autogen.sh | |
| ./configure --prefix=/usr | |
| make | |
| make check | |
| sudo make install | |
| sudo ldconfig # refresh shared library cache. | |
| protoc --version | |
| ``` | |
| Install Protocol Buffers for Go https://github.com/golang/protobuf#installation | |
| Install grpc (requres protobuf) https://github.com/grpc/grpc/blob/master/BUILDING.md | |
| Install Helm https://helm.sh/docs/intro/install/ | |
| ```bash | |
| mkdir /tmp/helm | |
| cd /tmp/helm | |
| helm_version=v3.0.2-linux-amd64 | |
| wget https://get.helm.sh/helm-$helm_version.tar.gz | |
| tar zxvf helm-$helm_version.tar.gz | |
| sudo mv linux-amd64/helm /bin/helm | |
| cd - | |
| helm version | |
| helm repo add stable https://kubernetes-charts.storage.googleapis.com | |
| ``` | |
| ## Install Tiller (the Helm server-side component) into the Kubernetes cluster | |
| see https://github.com/ruzickap/k8s-istio-demo#install-helm | |
| ## Install Rook Operator (Ceph storage for k8s): | |
| see https://github.com/ruzickap/k8s-istio-demo#install-rook | |
| ## Install ElasticSearch, Kibana, FluentBit | |
| see https://github.com/ruzickap/k8s-istio-demo#install-elasticsearch-kibana-fluentbit | |
| ## Check prerequisites | |
| ```bash | |
| function _out() { | |
| echo "$(date +'%F %H:%M:%S') $@" | |
| } | |
| function checkPrerequisites() { | |
| MISSING_TOOLS="" | |
| git --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} git" | |
| curl --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} curl" | |
| which sed &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} sed" | |
| docker -v &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} docker" | |
| unzip -version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} unzip" | |
| kubectl version --client=true &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} kubectl" | |
| minikube version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} minikube" | |
| if [[ -n "$MISSING_TOOLS" ]]; then | |
| _out "Some tools (${MISSING_TOOLS# }) could not be found, please install them first" | |
| exit 1 | |
| else | |
| _out You have all necessary prerequisites installed | |
| fi | |
| if ! kubectl describe namespace default | grep istio-injection=enabled > /dev/null ; then | |
| _out "Istio automatic sidecar injection needs to be enabled. See documentation/SetupLocalEnvironment.md" | |
| fi | |
| } | |
| checkPrerequisites | |
| ``` | |
| ## Create app | |
| https://blogs.vmware.com/opensource/2019/04/16/implementing-grpc-web-istio-envoy/ | |
| https://github.com/lucperkins/colossus/blob/1a487d9768d80094939a8171645227b64fd76b67/userinfo/userinfo-server.cc | |
| https://grpc.io/blog/state-of-grpc-web/ | |
| https://venilnoronha.io/seamless-cloud-native-apps-with-grpc-web-and-istio & https://github.com/venilnoronha/grpc-web-istio-demo | |
| https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing & https://github.com/saturnism/istio-by-example-java | |
| https://github.com/grpc/grpc/tree/master/examples/cpp/load_balancing | |
| https://istiobyexample.dev/grpc/ | |
| # Deploy the app to minikube | |
| https://github.com/IBM/istio101/blob/master/workshop/exercise-3/README.md | |
| https://meteatamel.wordpress.com/2018/04/24/istio-101-with-minikube/ | |
| + https://dzone.com/articles/service-mesh-with-istio-on-kubernetes-in-5-steps | |
| # Deploy the app to GKE | |
| https://medium.com/@at_ishikawa/getting-started-to-develop-a-web-application-on-gke-with-istio-for-grpc-web-1d84e084fb01 | |
| # TODO: RBAC | |
| https://rinormaloku.com/authorization-in-istio/ | |
| # TODO: CI/CD | |
| https://github.com/castlemilk/kubernetes-cicd/tree/8505444338567cfaad5ab61496ad14da4f578657 | |
| # TODO: read gameontext | |
| https://gameontext.gitbooks.io/gameon-gitbook/content/microservices/ | |
| # TODO: vagrant | |
| https://github.com/gameontext/gameon/blob/master/Vagrantfile#L62 | |
| # TODO: read CoolStore | |
| https://vietnam-devs.github.io/coolstore-microservices/#coolstore-website | |
| # TODO: websockets | |
| + https://github.com/srinandan/istio-workshop/tree/7f8ebcebcc71bb0948ff8803444a70af1ef133f6/misc/websocket | |
| + https://github.com/vietnam-devs/coolstore-microservices/blob/f41b4957d2f4fb4201624f21a54f33ad4ea695b0/deploys/charts/coolstore-istio/templates/api-vs.yaml | |
| + https://github.com/gameontext/gameon/blob/e8b424e531f531a6b14226326ea236363c379a81/kubernetes/istio/istio-gatewayVirtualService.yaml | |
| + https://github.com/MatthieuSegret/yummy-phoenix-graphql/blob/f0b258293697b0b120ef8e8a3b3905043c998617/kubernetes/yummy/templates/networking/virtual-service-ws.yaml#L17 | |
| + https://github.com/istio/istio/tree/master/samples/websockets | |
| + https://github.com/mukundha/istio-apigee-samples/blob/918fb72742ac20ac8a656362621b4ca15f39ba31/istio-manifests/tls-gateway.yaml#L33 | |
| + https://github.com/DeerNation/deployment/blob/c2ef67e1004d39f48fd1b8ad7d4cdcfc5bcde230/kubernetes/virtual-services.yaml | |
| + https://medium.com/12-developer-labors/angular-chat-using-kubernetes-with-websockets-bb4d87bfe99a | |
| + https://habr.com/ru/post/351012/ | |
| + https://hub.docker.com/r/ageapps/docker-chat/ | |
| + https://medium.com/@faiyaz26/deploying-a-real-time-notification-system-on-kubernetes-part-1-e64af5c93a2b | |
| ## TODO: Running on Google Kubernetes Engine (GKE) | |
| See https://github.com/GoogleCloudPlatform/microservices-demo#option-2-running-on-google-kubernetes-engine-gke |
minikibe agones (make sure you use --kubernetes-version v1.12.10 and agones/release-1.2.0)
see:
- https://habr.com/ru/post/469381/
- https://github.com/evillgenius75/k8s_appdev_adv/blob/master/5_K8S_Extensability/5_1_CRD.md
- https://github.com/guitarrapc/k8s-lab/blob/master/agones/README.md
- https://www.ovh.com/blog/deploying-game-servers-with-agones-on-ovh-managed-kubernetes/
wget --no-check-certificate https://raw.githubusercontent.com/NotGlop/docker-drag/master/docker_pull.py
sudo -E pip2 install requests --index-url=https://pypi.python.org/simple/ --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org
python docker_pull.py gcr.io/agones-images/udp-server:0.15
sudo -E docker load < agones-images_udp-server.tar
python docker_pull.py k8s.gcr.io/kube-apiserver:v1.16.0
sudo -E docker load < agones-images_udp-server.tar
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
--extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
--extra-config=apiserver.authorization-mode=RBAC \
--insecure-registry='localhost' \
--insecure-registry='127.0.0.1' \ \
--insecure-registry "192.168.39.0/24" \
--image-repository=gcr.io/google-containers –kubelet-insecure-tls --insecure-registry=gcr.io
To install Agones, a service account needs permission to create some special RBAC resource types.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:default
kubectl create namespace agones-system
kubectl apply -f https://raw.githubusercontent.com/googleforgames/agones/release-1.2.0/install/yaml/install.yaml
kubectl create -f https://raw.githubusercontent.com/googleforgames/agones/release-1.2.0/examples/simple-udp/gameserver.yaml
To confirm Agones is up and running, run the following command:
kubectl describe --namespace agones-system pods
kubectl get gameservers
minikube ip
kubectl describe gameserver
kubectl get gs
kubectl get gameservers #список всех GameServer
minikube ip
kubectl describe gameserver
kubectl get gs
Must print "ACK: hello", replace port based on kubectl get gs
echo "hello" | nc -u $(minikube ip) 7331
--- Istio using WSO2 identity server. https://medium.com/@balaajanthan/istio-enduser-authentication-with-wso2-identity-server-ba32a1941639 ---
The default WSO2 Identity server POD uses an image which requires a trial license from WSO2 because it has bugfix patches which are under a commercial license.
grpc login
https://github.com/Buzzvil/stella/blob/1f79b22291484dec3291ff94ff152bf643a9d648/authsvc/internal/app/authsrv/server.go
https://github.com/Buzzvil/stella/blob/39cbe6ec529b98424bea2ea38815f0b9a19901fd/deployments/chart/stella/charts/rentalsvc/templates/virtualservice.yaml#L43
https://github.com/xmlking/yeti/search?q=jwt&unscoped_q=jwt
https://github.com/xmlking/yeti/search?q=grpc&unscoped_q=grpc
https://github.com/ZZH-wl/grpc-login
http://lifesaviourfoundation.in/d2pxhn/istio-envoy-filter-example.html
Istio allows for JWT-based end-user authentication. We need to create a ‘Policy’ object that contains the identity service’s (IDCS) issuer name, and its jwks URL . We can use the script below to setup the policy (save it as ‘istio-authn-policy.yaml’ )
https://www.ateam-oracle.com/istio-%3A-end-user-authentication
Bookinfo Using the Authservice for Token Acquisition
https://github.com/istio-ecosystem/authservice
An OIDC provider configured to support Authorization Code grant type. The urls and credentials for this provider will be needed to configure Authservice.
https://github.com/panva/node-oidc-provider
Envoy Filter
Exactly the same behavior as above can be reached by using Envoy Filter to setup JWT Authentication. This option allows much more configuration flexibility(as listed in the Envoy documentation here), such as fully offline JWKS URI. A sample Envoy Filter configuration is provided below.
https://www.ateam-oracle.com/istio-%3A-end-user-authentication
Search query: "keycloak" "grant_type=authorization_code" "redirect_uri"
https://ria.ua.pt/bitstream/10773/23555/1/Disserta%C3%A7%C3%A3o.pdf
The Authorization Code Grant consists of the following calls (these examples came from the OAuth2 spec): https://community.apigee.com/articles/37103/saml2-vs-jwt-understanding-oauth2.html
keycloak "response_type=code"
https://www.stefaanlippens.net/oauth-code-flow-pkce.html
istio K8S keycloak
https://github.com/geoffroyvergne/devops/tree/5a887ad5c05300e3c5d087c38337c27b0afcdb23/k8s/deployments/keycloak
istio keycloak
https://github.com/kameshsampath/istio-keycloak-demo/blob/master/openshift-files/keycloak.yaml
BOOK
Securing the Perimeter: Deploying Identity and Access Management with Free Open Source Software
export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')
# see https://www.ivonet.nl/2015/05/23/Keycloak-Docker/
# Data volume
docker pull busybox:1.31.1
docker run --name postgres-data -v /var/lib/postgresql/data busybox:1.31.1 true
docker pull postgres:9.6.17
# Postgres coupled to the datavolume
(docker rm postgres_9 || true)
docker run -d --name postgres_9 -p 15432:5432 --volumes-from postgres-data -e POSTGRES_DB=keycloak_db -e POSTGRES_DATABASE=keycloak_db -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=password -e POSTGRES_ROOT_PASSWORD=password postgres:9.6.17
docker logs -f postgres_9
# CHECK:
# sudo add-apt-repository "deb https://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
# sudo apt-get update
# sudo apt install postgresql-client-common postgresql-client-10 -y
# psql -h localhost -p 15432 -U keycloak keycloak_db
# Ctrl+D
cd to certs dir
# MUST EXIST
file httpbin.example.com.crt
file httpbin.example.com.crt
# Keycloak server image linking to the postgres image
docker pull jboss/keycloak:9.0.0
docker run --name keycloak-data -v /var/lib/keycloak/data busybox:1.31.1 true
export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')
(docker rm keycloak || true)
docker run --rm -it -p 8443:8443 -p 8080:8080 --name keycloak --memory="512m" --link postgres_9 --volumes-from keycloak-data \
-e PROXY_ADDRESS_FORWARDING=true \
-e DB_VENDOR=postgres -e DB_ADDR=$MY_IP:15432 -e DB_PORT=15432 \
-e DB_DATABASE=keycloak_db -e DB_SCHEMA=public \
-e DB_USER=keycloak -e DB_PASSWORD=password \
-e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin_pass \
-e KEYCLOAK_HTTPS_PORT=8443 -e KEYCLOAK_HTTP_PORT=8443 \
-e KEYCLOAK_HOSTNAME=keycloak.example.com \
-v httpbin.example.com.crt:/etc/x509/https/tls.crt \
-v httpbin.example.com.crt:/etc/x509/https/tls.key \
jboss/keycloak:9.0.0
# Uncomment the line below if you want to specify JDBC parameters. The parameter below is just an example, and it shouldn't be used in production without knowledge. It is highly recommended that you read the PostgreSQL JDBC driver documentation in order to use it.
# -e JDBC_PARAMS="ssl=true"
Add to host /etc/hosts file:
```bash
sudo -- sh -c -e "echo '127.0.0.1 keycloak.example.com' >> /etc/hosts";
sudo -- sh -c -e "echo '127.0.0.1 example.com' >> /etc/hosts";
cat /etc/hostsAdd to minikube /etc/hosts file:
export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')
minikube ssh -- sudo tee -a /etc/hosts <<EOF
127.0.0.1 localhost
127.0.1.1 minikube
127.0.0.1 example.com
$MY_IP keycloak.example.com
EOF
minikube ssh -- sudo cat /etc/hosts
minikube ssh -- ping keycloak.example.comOpen https://keycloak.example.com:8443/auth/admin/
Set sslRequired in realm settings
Navigate to https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/token-settings
Default Signature Algorithm RS256
Set Access Token Lifespan 60 min
Set Client login timeout 60 min
Navigate to https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/clients
OLDTODO~set up a client "pkce-test" (in the "master" realm) with access type "public"~~~
set up a client "pkce-test" (in the "master" realm) with access type "confidential"
set protocol to openid-connect
set Root URL to app url, like http://dev.local/
enable options Direct Access Grants and Authorization in the Settings section of newly created client. Also Access Type should be set to confidential
(to be able to use PKCE) and the "http://dev.local/*" as valid redirect URIs (which is a required field).
NOTE: replace dev.local with you app url
set Secret available under Credentials tab from Client section as "client_secret" in configmap
create a user "john_username" in https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/users with non-temporary password (credentials tab).
provider = "https://keycloak.example.com:8443/auth/realms/master"
username = "john_username"
password = "j000hn_pass" (not Temporary)
RemoveUpdate passwordfromRequired User Actions
TODO: Create code verifier and challenge as in https://www.stefaanlippens.net/oauth-code-flow-pkce.html#PKCE-code-verifier-and-challenge
Open https://keycloak.example.com:8443/auth/realms/master/.well-known/openid-configuration to see endpoints
Open https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/certs
copy it and replace " with "
set as "jwks" in configmap (yaml file, on istio side)
Follows
https://medium.com/@robert.broeckelmann/openid-connect-authorization-code-flow-with-red-hat-sso-d141dde4ed3f
https://gist.github.com/ataube/2dd3632eeb4f0b46286c87005dc20c74
http://www.janua.fr/keycloak-access-token-verification-example/
https://qiita.com/rururu_kenken/items/5a7b94146cf0a2eb537d
https://dzone.com/articles/oauth-20-authorisation-code-grant
Now lets test Keycloak Server OIDC URI Endpoints https://www.keycloak.org/docs/latest/server_admin/index.html#keycloak-server-oidc-uri-endpoints
Run in shell
echo "https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/auth?response_type=code&client_id=pkce-test&state=38610846-c64d-4fd2-8d9-b5fa2f976298&redirect_uri=https://keycloak.example.com:8443/authorization-code/callback&scope=openid profile User"
Open new in incognito browser window url generatedfrom command above
change master to realm name, redirect_uri, client_id, e.t.c.
login as
username = "john_username"
password = "j000hn_pass" (must be not Temporary from step above)
NOTE: state is generated.
TODO: provide state generator and validator
replace CODE_FROM_REQUEST below with answer to request above, just copy text after &code=
NOTE: If the redirect_uri was included in the authorization request, this value must be the same as the value used in that request.
export CODE_FROM_REQUEST=e4c91079-2c06-4157-a12d-c981dea7e6c3.0aa5bab3-a6e2-4d0a-babf-274adc241bd8.2aa47e2c-9dde-4b95-95d4-17c59ace967c
NOTE: client from keycloak settings
export CLIENT_SECRET=199c35ad-8e4b-4546-8e18-6f0d158eb364
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=authorization_code&client_id=pkce-test&client_secret=$CLIENT_SECRET&code=$CODE_FROM_REQUEST&redirect_uri=https://keycloak.example.com:8443/authorization-code/callback&scope=openid profile User&client_id=pkce-test&client_secret=$CLIENT_SECRET" https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/token
TODO: user must provide client_secret or not?
Example response:
{
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldU ...",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSld ...",
"token_type": "bearer",
"not-before-policy": 0,
"session_state": "04c1e2e4-e6a2-460f-841a-83ef532b01d8",
"scope": "email profile"
}
Decode the access_token in jwt.io
Open keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/certs
VERIFY SIGNATURE with Public key from rsa-generated (Realm Settings -> Keys)
NOTE: replace rsa-generated in production
Run keycloak under https
Follows
https://stackoverflow.com/questions/59446436/keycloak-keystore-and-java-keystore-with-https-redirect-loop
Google for `keycloak Free SSL Certificates from Let's Encrypt`
Generate certs as in https://stackoverflow.com/a/58801128
Run keycloak as in https://stackoverflow.com/a/58801128 , provide `KEYCLOAK_HOSTNAME`, `KEYCLOAK_HTTPS_PORT`, `KEYCLOAK_HTTP_PORT` and volumes
```bash
// ...
-p 8443:8443 --name keycloak --memory="512m" \
-e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin_pass \
-e KEYCLOAK_HTTPS_PORT=8443 -e KEYCLOAK_HTTP_PORT=8443 \
-e KEYCLOAK_HOSTNAME=keycloak.example.com \
-v /<path>/tls.crt:/etc/x509/https/tls.crt \
-v /<path>/tls.key:/etc/x509/https/tls.key \
// ...
Open /auth/admin/
Navigate to /auth/admin/master/console/#/realms/master/clients
Set sslRequired to true in realm settings
PostgreSQL database that is hosted outside of both of the Keycloak servers.
Follow
https://homelab.blog/blog/security/keycloak-part-2-setting-up-keycloak/
https://homelab.blog/blog/devops/Istio-OIDC-Config/
Deploy istio with custom certs
Google "istio letsencrypt httpsRedirect"
Follow
https://homelab.blog/blog/devops/Istio-OIDC-Config/
https://istio.io/blog/2019/custom-ingress-gateway/
https://github.com/stefanprodan/istio-gke/blob/master/docs/istio/05-letsencrypt-setup.md
https://istio.io/docs/tasks/security/citadel-config/plugin-ca-cert/
https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/
https://istio.io/docs/ops/configuration/security/root-transition/
https://istio.io/docs/tasks/security/citadel-config/ca-namespace-targeting/
https://istio.io/docs/tasks/security/authentication/https-overlay/
https://itsmetommy.com/2019/10/10/kubernetes-istio-cert-manager-gke/
kubectl create secret generic cacerts -n istio-system --from-file=samples/certs/ca-cert.pem \
--from-file=samples/certs/ca-key.pem --from-file=samples/certs/root-cert.pem \
--from-file=samples/certs/cert-chain.pem
istioctl manifest apply --set values.global.mtls.enabled=true,values.security.selfSigned=false --set values.global.controlPlaneSecurityEnabled=true
# CAUTION!
kubectl delete secret istio.defaultTBD:
Istio with OpenID, validation on app-side
https://github.com/IBM/cloud-native-starter/blob/master/documentation/DemoAuthentication.md
https://www.dangtrinh.com/2019/08/end-user-authentication-with-istio-and.html
https://keycloak.discourse.group/t/end-user-authentication-with-istio-and-keycloak/390
https://github.com/SaifRehman/ICP-Airways
https://github.com/vietnam-devs/coolstore-microservices
https://github.com/EdwinVW/pitstop
https://github.com/GoogleCloudPlatform/microservices-demo
https://github.com/selloween/docker-keycloak-letsencrypt
https://www.infracloud.io/service-mesh-demo-istio-sockshop/
NOTE: Deprecated. Websocket upgrades are done automatically starting from Istio 1.0.
read docs
https://github.com/solo-io/solo-docs/blob/9f9ba476fab0803fa7a7b2d74b26e3a1f7c3a325/supergloo/docs/v1/github.com/solo-io/supergloo/api/external/istio/networking/v1alpha3/virtual_service.proto.sk.md
https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/