Skip to content

Instantly share code, notes, and snippets.

@blockspacer
Last active October 11, 2022 11:29
Show Gist options
  • Select an option

  • Save blockspacer/0237dcdea981f78f8c3ea80b8b1d037d to your computer and use it in GitHub Desktop.

Select an option

Save blockspacer/0237dcdea981f78f8c3ea80b8b1d037d to your computer and use it in GitHub Desktop.
Istio C++
# About Docker
+ https://github.com/katopz/best-practices#docker
# About Kubernetes
+ pods are the base unit of operation for Kubernetes. Pods are the most visible, viable, and ephemeral units
that comprise one or more tightly coupled containers. Kubernetes does not operate
at the level of containers. There can be multiple pods in a single server node and
data sharing easily happens in between pods. Kubernetes automatically
provision and allocate pods for various services. Each pod has its own IP address
and shares the localhost and volumes.
+ Labels are typically the metadata that is attached to objects, including pods.
+ Services offer a low-overhead way to route all kinds of service requests to a set of pods to accomplish the requests.
Read:
+ https://docs.google.com/document/d/1E03-g0h3MgFlohXqPNYCjl7Pv_CA_l0zhD6K3MZ8O0M/edit?usp=sharing
+ https://www.freecodecamp.org/news/learn-kubernetes-in-under-3-hours-a-detailed-guide-to-orchestrating-containers-114ff420e882/
+ https://neuvector.com/network-security/advanced-kubernetes-networking/
+ https://layer5.io/landscape/
+ https://www.tfir.io/2019/05/17/whats-the-right-ingress-controller-for-my-kubernetes-environment/
# About Service Mesh
+ https://www.slideshare.net/dbryant_uk/microxchg-2018-what-is-a-service-mesh-do-i-need-one-when-developing-cloud-native-microservices
# About Cloud-enabled apps
The currently running massive and monolithic applications get
modernized and migrated to cloud environments to reap the distinct benefits of
the cloud paradigm
# About Cloud-native apps
This is all about designing, developing, debugging, delivering,
and deploying applications directly on cloud environments by intrinsically
leveraging the non-functional capabilities of cloud environments
# About microservices architecture (MSA)
Microservices are the architecture paradigm wherein a monolithic (silo) application is decomposed into small tiny micro applications which are packaged and deployed independently.
Read:
+ https://martinfowler.com/microservices/
+ https://www.zeolearn.com/magazine/microservices-architecture
+ https://microservices.io/patterns/microservices.html
+ https://docs.microsoft.com/en-us/azure/architecture/microservices/design/data-considerations
+ http://www.mammatustech.com/high-speed-microservices
+ https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md
# Microservice design principles
+ High cohesion among services: A microservice should have one single focus and
the sole responsibility for that action. It should not change as a result of other
related services. Services should be easily rewritable so that we can achieve
scalability, reliability, and flexibility. It should handle a single business function
and domain-specific functionality.
+ Autonomous service: A service should independently handle its work without
the help of any other services. It should not be tightly integrated with any other
service; it should remain loosely coupled in nature. By autonomous, we mean
that a microservice should not change because of the external components with
which it interacts. Autonomous services honor contracts and interfaces. They
should be stateless, independently changeable, independently deployable,
backwards compatible, and they should support concurrent development.
+ Business domain-centric service: Each individual service should perform or
represent a single business function. This could be a calculation of sales, tax,
income tax, or any other function related to a specific area. Each service should
bound or define its scope. Business-centric code can help to provide more
cohesion and make services more responsive to handle any changes in the
domain or business logic requirements.
+ Resilience: Resilience is a standard these days when providing a service to a
customer. Failure to provide resilience may result in another endpoint not
providing a response to your microservices. Designing your service in microformat helps to overcome failure. Our service should register itself during
startup and de-register itself upon failure. This should be part of a dynamic
discovery service, such as the auto-creation of a queue or the auto-removal of the
queue in a message queue. There could be a number of problems or exceptions
that a network-based service could encounter. It should be able to handle delays
and the unavailability of another service.
+ Observable service or functionality: Observability is another important design
principle while working on distributed microservices. When a complex
interconnected service breaks, it can take hours or days to isolate issues. We
should design our services in such a way that we can check the health of any
service by either showing its status on a health page or by sending it to a central
logging service such as Splunk, Logstash, syslogd, Logentries, Datadog, or Sumo
Logic. Observability is required to support reliable, scalable, and cost-effective
services and metrics to scale up, metrics to scale down, and metrics to alert the
team. This kind of monitoring and logging needs to be located at a central place.
In a containerized environment, auto deployment should be able to auto-detect
when a deployment fails so that it can be rolled back quickly to an older running
version. Observability can be related to CPU usage, memory usage, network
input/output metrics, disk metrics, the number of connections to a service, and so
on. All these metrics are easily available and measurable through tools such as
Check_MK, Nagios, New Relic, AppDynamics, StatsD, and Graphana.
+ Observability not only helps in terms of providing a technical solution but also so
that we can identify business decision-making, like the sales of a specific service
or the returns for a specific product.
Automation: Microservices also create challenges for an operation team with
regard to deployment, verifying functionality, and performing various types of
testing. There is now a wide range of automation tools available on market that
can easily be integrated to achieve automated deployment, verification, testing,
failure, and rollbacks. Some of the famous tools are Jenkins; Teamcity; Bamboo;
Git workflow plugins; GitLab CI/CD; UI test tools, such as Selenium, PhantomJS,
Nightwatch, BrowserStack; and many more. One important point here is that
while Docker changed the container market when it was developed, it was hard
to implement in a production environment, where a complete stack is required to
maintain it as a production-level service. There wasn't much clarity with regard
to monitoring or deployment. After that, Google released Borg in the form of
Kubernetes and changed the container market again by providing easy
deployment and rollback options with easy service and routing functionalities
that were perfect for production-grade deployments.
NOTE:
+ **Microservices prefer letting each service manage its own database** The key point it that the services should have no knowledge of each other's underlying database. https://github.com/katopz/best-practices/blob/master/best-practices-for-building-a-microservice-architecture.md#service-essentials-2
+ Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP.
+ Services can be developed and deployed independently of one another.
+ In the microservices world, the network is your biggest point of failure. In production you will fail if too much connections between. https://medium.com/@oprearocks/blasphemy-multiple-microservices-shared-database-f525025a8a81
+ 'micro' part doesn't mean replace every class with a network service, but componentize a monolithic application into sensibly sized components, each one dealing with an aspect of your program.
+ use In-memory service data
# Install curl with https support
```
git clone https://github.com/bagder/curl.git
# requires https://askubuntu.com/a/826891
sudo apt-get build-dep curl
cd curl
./buildconf
./configure --with-ssl --disable-shared
make -j8
# sudo make uninstall
sudo make install
curl -V
# Features: ... SSL ... TLS-SRP ...
```
# Install VirtualBox
Follow https://www.virtualbox.org/wiki/Downloads
# Install docker
Follow https://phoenixnap.com/kb/how-to-install-docker-on-ubuntu-18-04
Or under proxy, see https://gist.github.com/blockspacer/893b31e61c88f6899ffd0813111b3e41
# Install kubectl
Follow https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
```
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version --client
```
OR
```
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s http://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
```
# Install kubeadm
Follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
```
curl -SSL https://dl.k8s.io/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubeadm > ./kubeadm
chmod a+rx kubeadm
sudo mv kubeadm /usr/bin/kubeadm
kubeadm version
```
## Under proxy - setup Kubernetes The Hard Way
If under proxy - See https://wiki.christophchamp.com/index.php?title=Kubernetes/the-hard-way
# Install Minikube
Follow https://kubernetes.io/docs/setup/learning-environment/minikube/#installation
(For Windows: https://meteatamel.wordpress.com/2018/02/14/minikube-on-windows/)
```
curl -Lo minikube http://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
&& chmod +x minikube
sudo install minikube /usr/local/bin
minikube status
minikube stop
# OR minikube delete
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
--extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
--extra-config=apiserver.authorization-mode=RBAC \
--insecure-registry='localhost' \
--insecure-registry='127.0.0.1' \
--insecure-registry "192.168.39.0/24"
# OPTIONAL: open dashboard
minikube addons enable dashboard && kubectl get pods --all-namespaces | grep dashboard && sleep 15 && minikube dashboard
```
```
kubectl config use-context minikube
```
**NOTE: `minikube start --alsologtostderr --memory=16384 --cpus=4 --disk-size 25GB` with 16384 MB of memory and 4 CPUs. 16384 MB is sufficent to run Istio and bookinfo.**
```bash
# (Optional, recommended) If you want minikube to provide a load balancer for use by Istio, you can use the minikube tunnel feature. Run this command in a different terminal, because the minikube tunnel feature will block your terminal to output diagnostic information about the network:
minikube tunnel
# Sometimes minikube does not clean up the tunnel network properly. To force a proper cleanup:
minikube tunnel --cleanup
```
**NOTE: Use may want to run `minikube delete` before `minikube start`**
**NOTE: you can set proxy by `--docker-env`:**
```
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
--extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
--extra-config=apiserver.authorization-mode=RBAC \
--insecure-registry='localhost' \
--insecure-registry='127.0.0.1' \
--insecure-registry "192.168.39.0/24" \
--docker-env http_proxy=http://172.17.0.1:3128 \
--docker-env https_proxy=http://172.17.0.1:3128 \
--docker-env no_proxy=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru \
--docker-env HTTP_PROXY=http://172.17.0.1:3128 \
--docker-env HTTPS_PROXY=http://172.17.0.1:3128 \
--docker-env NO_PROXY=192.168.99.0/24,$(minikube ip),localhost,127.0.0.*,10.*,192.168.*,*.corp1.ru,*.corp.ru
# add corp certs to ~/.minikube/files/etc/ssl/certs
cp -r /etc/ssl/certs/* ~/.minikube/files/etc/ssl/certs
cp -r /usr/share/ca-certificates/* ~/.minikube/files/etc/ssl/certs
# cert files must have 750 permission
sudo chmod 750 ~/.minikube/files/etc/ssl/certs/*.crt
```
Now you can ssh into minikube and change `daemon.json` according to your proxy
```
minikube ssh
sudo mkdir -p /etc/docker
# NOTE: can't set "insecure-registries" due to `--insecure-registry` minikube arg
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"dns": ["127.0.0.53", "10.8.13.11", "10.8.13.12", "8.8.4.4", "8.8.8.8", "10.8.13.11", "10.8.13.12"],
"registry-mirrors":["https://docker.mirrors.ustc.edu.cn"],
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
}
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
# test pull under minikube
# docker pull grafana/grafana:6.4.3
exit
```
## Enable minikube addons
```bash
# see http://rastko.tech/kubernetes/2019/01/01/minikube-on-mac.html
minikube addons enable ingress
# Use `--insecure-registry='192.168.39.0/24'`
# see https://minikube.sigs.k8s.io/docs/tasks/docker_registry/
minikube addons enable registry
```
## Prepere minikube for istio
Follow on linux https://istio.io/docs/setup/platform-setup/minikube/
```bash
# monitor memory usage in minikube
minikube ssh -- sudo top
```
# Install istioctl
Follow on linux https://istio.io/docs
(For Windows: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/istio-install.md#windows)
```
curl -L https://git.io/getLatestIstio | sh -
cd istio*
echo 'export PATH=$(pwd)/bin:$PATH' >> ~/.bashrc
# OR
# sudo cp ./bin/istioctl /usr/local/bin/istioctl
# sudo chmod +x /usr/local/bin/istioctl
istioctl version
```
Istio releases http://gcsweb.istio.io/gcs/istio-release/releases/1.4.3/
## Apply istio profile
Follow https://istio.io/docs/setup/install/istioctl/
```bash
istioctl manifest apply --skip-confirmation
# OR
# istioctl manifest apply --set values.global.mtls.enabled=true,values.security.selfSigned=false --set values.global.controlPlaneSecurityEnabled=true
# see https://istio.io/docs/setup/getting-started/
kubectl get svc -n istio-system
# ensure corresponding Kubernetes pods are deployed and have a STATUS of Running
kubectl get pods -n istio-system
```
Make sure that all `istio-system` pods are running or completed before continuing. This can take several minutes when starting the pods for the first time. Be patient.
enable automatic sidecar injection:
```
kubectl label namespace default istio-injection=enabled
```
> if you want to exclude a specific pod from getting istio sidecar injected, add this to `Deployment` kind
```yaml
metadata:
annotations:
sidecar.istio.io/inject: "false"
```
# Add istioctl to PATH
(see https://github.com/IBM/istio101/blob/master/workshop/exercise-2/README.md or https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing)
```
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo.yaml
kubectl get svc -n istio-system
kubectl get pods -n istio-system
```
## NOTE: How to build docker images which will be seen by Kubernetes directly without having to push them anywhere
just run
```bash
# NOTE: Later, when we no longer wish to use the Minikube host, we can undo this change by running: eval $(minikube docker-env -u)
eval $(minikube docker-env)
```
and now you can build docker images which will be seen by Kubernetes directly without having to push them anywhere.
## Open istio dashboards
```bash
istioctl dashboard controlz # Open ControlZ web UI
istioctl dashboard envoy # Open Envoy admin web UI
istioctl dashboard grafana # Open Grafana web UI
istioctl dashboard jaeger # Open Jaeger web UI
istioctl dashboard kiali # Open Kiali web UI
istioctl dashboard prometheus # Open Prometheus web UI
istioctl dashboard zipkin # Open Zipkin web UI
```
See https://istio.io/docs/reference/commands/istioctl/#istioctl-dashboard
## NOTE: How to run Jaeger Dashboard
```bash
# in Istio >= 1.4.x
istioctl dashboard jaeger
```
OR if Istio < 1.4.x:
```bash
kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686
```
URL to Open Jaeger: http://localhost:16686
## NOTE: How to run Grafana Dashboard
```bash
# in Istio >= 1.4.x
istioctl dashboard grafana
```
OR if Istio < 1.4.x:
```bash
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000 &
```
URL to open Grafana: http://localhost:3000/dashboard/db/istio-mesh-dashboard
## NOTE: How to run Prometheus Dashboard
```bash
# in Istio >= 1.4.x
istioctl dashboard prometheus
```
OR if Istio < 1.4.x:
```bash
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &
```
URL to open Prometheus: http://localhost:9090
## NOTE: How to run Kiali
Kiali is Istio’s dashboard and this is one of the coolest features in 1.4.x: To open the Kiali dashboard you no longer need to execute complicated port-forwarding commands, simply type
```bash
# in Istio >= 1.4.x
istioctl dashboard kiali
```
OR if Istio < 1.4.x:
Run the following command to install Kiali:
```bash
bash <(curl -L http://git.io/getLatestKialiKubernetes)
```
Note: For some reason the script didn't work for me. I had to replace one line:
```bash
get_downloader
github_api_url="https://api.github.com/repos/kiali/kiali/releases/latest"
kiali_version_we_want="v0.15.0"
```
To launch Kiali you need the IP address and NodePort:
```bash
minikube ip
kubectl get svc -n istio-system kiali --output 'jsonpath={.spec.ports[*].nodePort}'
```
URL to open Kiali: https://[minikube-ip]:[kiali-nodeport]/kiali
## Run `istioctl verify-install`
```bash
istioctl verify-install
```
# Dev tools
Install nodejs
```bash
# sudo apt remove node npm
# sudo rm /usr/local/bin/node # must be removed
# sudo rm /usr/local/bin/npm # must be removed
# sudo rm -rf /usr/local/lib/node_modules # must be removed
OS_ARCH=x64 # $(uname -m)
NODE_V=v10.18.1
wget https://nodejs.org/dist/$NODE_V/node-$NODE_V-linux-$OS_ARCH.tar.gz
tar -xvf node-$NODE_V-linux-$OS_ARCH.tar.gz
cd node-$NODE_V-linux-$OS_ARCH
sudo cp -R * /usr/local/
sudo chown -R $USER /usr/local/lib/node_modules
cd -
# npm install npm -g # optional
node -v
npm -v
```
Under proxy: you may want to configure `~/.npmrc` like so https://stackoverflow.com/a/36929934
```bash
# NOTE: you may want to use NODE_TLS_REJECT_UNAUTHORIZED=0 under proxy during `npm install`
NODE_TLS_REJECT_UNAUTHORIZED=0 \
HTTP_PROXY=http://127.0.0.1:8088 \
HTTPS_PROXY=http://127.0.0.1:8088 \
npm install \
--unsafe-perm binding
```
Install protobuf from sources https://developers.google.com/protocol-buffers/docs/downloads and (if exists) remove old protobuf version `apt-get remove libprotobuf-dev`
NOTE: it is better to clone https://github.com/grpc/grpc/ repo and build protobuf from `grpc/third_party/protobuf`
Tested with `GRPC_RELEASE_TAG=v1.26.x`
```bash
python -V # Python 2.7 or newer
sudo apt-get install autoconf automake libtool curl make g++ unzip
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./configure --prefix=/usr
make
make check
sudo make install
sudo ldconfig # refresh shared library cache.
protoc --version
```
Install Protocol Buffers for Go https://github.com/golang/protobuf#installation
Install grpc (requres protobuf) https://github.com/grpc/grpc/blob/master/BUILDING.md
Install Helm https://helm.sh/docs/intro/install/
```bash
mkdir /tmp/helm
cd /tmp/helm
helm_version=v3.0.2-linux-amd64
wget https://get.helm.sh/helm-$helm_version.tar.gz
tar zxvf helm-$helm_version.tar.gz
sudo mv linux-amd64/helm /bin/helm
cd -
helm version
helm repo add stable https://kubernetes-charts.storage.googleapis.com
```
## Install Tiller (the Helm server-side component) into the Kubernetes cluster
see https://github.com/ruzickap/k8s-istio-demo#install-helm
## Install Rook Operator (Ceph storage for k8s):
see https://github.com/ruzickap/k8s-istio-demo#install-rook
## Install ElasticSearch, Kibana, FluentBit
see https://github.com/ruzickap/k8s-istio-demo#install-elasticsearch-kibana-fluentbit
## Check prerequisites
```bash
function _out() {
echo "$(date +'%F %H:%M:%S') $@"
}
function checkPrerequisites() {
MISSING_TOOLS=""
git --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} git"
curl --version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} curl"
which sed &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} sed"
docker -v &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} docker"
unzip -version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} unzip"
kubectl version --client=true &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} kubectl"
minikube version &> /dev/null || MISSING_TOOLS="${MISSING_TOOLS} minikube"
if [[ -n "$MISSING_TOOLS" ]]; then
_out "Some tools (${MISSING_TOOLS# }) could not be found, please install them first"
exit 1
else
_out You have all necessary prerequisites installed
fi
if ! kubectl describe namespace default | grep istio-injection=enabled > /dev/null ; then
_out "Istio automatic sidecar injection needs to be enabled. See documentation/SetupLocalEnvironment.md"
fi
}
checkPrerequisites
```
## Create app
https://blogs.vmware.com/opensource/2019/04/16/implementing-grpc-web-istio-envoy/
https://github.com/lucperkins/colossus/blob/1a487d9768d80094939a8171645227b64fd76b67/userinfo/userinfo-server.cc
https://grpc.io/blog/state-of-grpc-web/
https://venilnoronha.io/seamless-cloud-native-apps-with-grpc-web-and-istio & https://github.com/venilnoronha/grpc-web-istio-demo
https://docs.google.com/document/d/1Qo8o5C4UpGwMF7Mg02kLTaU4-xCSfJjLcnIFNveMEEA/edit?usp=sharing & https://github.com/saturnism/istio-by-example-java
https://github.com/grpc/grpc/tree/master/examples/cpp/load_balancing
https://istiobyexample.dev/grpc/
# Deploy the app to minikube
https://github.com/IBM/istio101/blob/master/workshop/exercise-3/README.md
https://meteatamel.wordpress.com/2018/04/24/istio-101-with-minikube/
+ https://dzone.com/articles/service-mesh-with-istio-on-kubernetes-in-5-steps
# Deploy the app to GKE
https://medium.com/@at_ishikawa/getting-started-to-develop-a-web-application-on-gke-with-istio-for-grpc-web-1d84e084fb01
# TODO: RBAC
https://rinormaloku.com/authorization-in-istio/
# TODO: CI/CD
https://github.com/castlemilk/kubernetes-cicd/tree/8505444338567cfaad5ab61496ad14da4f578657
# TODO: read gameontext
https://gameontext.gitbooks.io/gameon-gitbook/content/microservices/
# TODO: vagrant
https://github.com/gameontext/gameon/blob/master/Vagrantfile#L62
# TODO: read CoolStore
https://vietnam-devs.github.io/coolstore-microservices/#coolstore-website
# TODO: websockets
+ https://github.com/srinandan/istio-workshop/tree/7f8ebcebcc71bb0948ff8803444a70af1ef133f6/misc/websocket
+ https://github.com/vietnam-devs/coolstore-microservices/blob/f41b4957d2f4fb4201624f21a54f33ad4ea695b0/deploys/charts/coolstore-istio/templates/api-vs.yaml
+ https://github.com/gameontext/gameon/blob/e8b424e531f531a6b14226326ea236363c379a81/kubernetes/istio/istio-gatewayVirtualService.yaml
+ https://github.com/MatthieuSegret/yummy-phoenix-graphql/blob/f0b258293697b0b120ef8e8a3b3905043c998617/kubernetes/yummy/templates/networking/virtual-service-ws.yaml#L17
+ https://github.com/istio/istio/tree/master/samples/websockets
+ https://github.com/mukundha/istio-apigee-samples/blob/918fb72742ac20ac8a656362621b4ca15f39ba31/istio-manifests/tls-gateway.yaml#L33
+ https://github.com/DeerNation/deployment/blob/c2ef67e1004d39f48fd1b8ad7d4cdcfc5bcde230/kubernetes/virtual-services.yaml
+ https://medium.com/12-developer-labors/angular-chat-using-kubernetes-with-websockets-bb4d87bfe99a
+ https://habr.com/ru/post/351012/
+ https://hub.docker.com/r/ageapps/docker-chat/
+ https://medium.com/@faiyaz26/deploying-a-real-time-notification-system-on-kubernetes-part-1-e64af5c93a2b
## TODO: Running on Google Kubernetes Engine (GKE)
See https://github.com/GoogleCloudPlatform/microservices-demo#option-2-running-on-google-kubernetes-engine-gke
@blockspacer

blockspacer commented Dec 31, 2019

Copy link
Copy Markdown
Author

minikibe agones (make sure you use --kubernetes-version v1.12.10 and agones/release-1.2.0)

see:

wget --no-check-certificate https://raw.githubusercontent.com/NotGlop/docker-drag/master/docker_pull.py
sudo -E pip2 install requests --index-url=https://pypi.python.org/simple/ --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org
python docker_pull.py gcr.io/agones-images/udp-server:0.15                               
sudo -E docker load < agones-images_udp-server.tar
python docker_pull.py k8s.gcr.io/kube-apiserver:v1.16.0                          
sudo -E docker load < agones-images_udp-server.tar
minikube start --alsologtostderr --kubernetes-version v1.12.10 --memory=12288 --cpus=2 --disk-size 25GB --vm-driver virtualbox \
  --extra-config='apiserver.enable-admission-plugins=LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook' \
  --extra-config=apiserver.authorization-mode=RBAC \
  --insecure-registry='localhost' \
  --insecure-registry='127.0.0.1' \ \
  --insecure-registry "192.168.39.0/24" \
        --image-repository=gcr.io/google-containers –kubelet-insecure-tls --insecure-registry=gcr.io

To install Agones, a service account needs permission to create some special RBAC resource types.

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:default
kubectl create namespace agones-system
kubectl apply -f https://raw.githubusercontent.com/googleforgames/agones/release-1.2.0/install/yaml/install.yaml
kubectl create -f https://raw.githubusercontent.com/googleforgames/agones/release-1.2.0/examples/simple-udp/gameserver.yaml

To confirm Agones is up and running, run the following command:

kubectl describe --namespace agones-system pods
kubectl get gameservers 
minikube ip
kubectl describe gameserver
kubectl get gs 
kubectl get gameservers #список всех GameServer
minikube ip
kubectl describe gameserver 
kubectl get gs

Must print "ACK: hello", replace port based on kubectl get gs

echo "hello" | nc -u $(minikube ip) 7331

@blockspacer

blockspacer commented Jan 21, 2020

Copy link
Copy Markdown
Author

--- Istio using WSO2 identity server. https://medium.com/@balaajanthan/istio-enduser-authentication-with-wso2-identity-server-ba32a1941639 ---
The default WSO2 Identity server POD uses an image which requires a trial license from WSO2 because it has bugfix patches which are under a commercial license.

@blockspacer

Copy link
Copy Markdown
Author

Istio allows for JWT-based end-user authentication. We need to create a ‘Policy’ object that contains the identity service’s (IDCS) issuer name, and its jwks URL . We can use the script below to setup the policy (save it as ‘istio-authn-policy.yaml’ )

https://www.ateam-oracle.com/istio-%3A-end-user-authentication

@blockspacer

Copy link
Copy Markdown
Author

Bookinfo Using the Authservice for Token Acquisition
https://github.com/istio-ecosystem/authservice
An OIDC provider configured to support Authorization Code grant type. The urls and credentials for this provider will be needed to configure Authservice.
https://github.com/panva/node-oidc-provider

@blockspacer

Copy link
Copy Markdown
Author

Envoy Filter
Exactly the same behavior as above can be reached by using Envoy Filter to setup JWT Authentication. This option allows much more configuration flexibility(as listed in the Envoy documentation here), such as fully offline JWKS URI. A sample Envoy Filter configuration is provided below.

https://www.ateam-oracle.com/istio-%3A-end-user-authentication

@blockspacer

blockspacer commented Feb 26, 2020

Copy link
Copy Markdown
Author

Search query: "keycloak" "grant_type=authorization_code" "redirect_uri"

https://ria.ua.pt/bitstream/10773/23555/1/Disserta%C3%A7%C3%A3o.pdf

The Authorization Code Grant consists of the following calls (these examples came from the OAuth2 spec): https://community.apigee.com/articles/37103/saml2-vs-jwt-understanding-oauth2.html
keycloak "response_type=code"
https://www.stefaanlippens.net/oauth-code-flow-pkce.html
istio K8S keycloak
https://github.com/geoffroyvergne/devops/tree/5a887ad5c05300e3c5d087c38337c27b0afcdb23/k8s/deployments/keycloak
istio keycloak
https://github.com/kameshsampath/istio-keycloak-demo/blob/master/openshift-files/keycloak.yaml
BOOK
Securing the Perimeter: Deploying Identity and Access Management with Free Open Source Software

export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')

# see https://www.ivonet.nl/2015/05/23/Keycloak-Docker/

# Data volume
docker pull busybox:1.31.1
docker run --name postgres-data -v /var/lib/postgresql/data busybox:1.31.1 true

docker pull postgres:9.6.17
# Postgres coupled to the datavolume
(docker rm postgres_9 || true)
docker run -d --name postgres_9 -p 15432:5432 --volumes-from postgres-data -e POSTGRES_DB=keycloak_db -e POSTGRES_DATABASE=keycloak_db -e POSTGRES_USER=keycloak -e POSTGRES_PASSWORD=password -e POSTGRES_ROOT_PASSWORD=password postgres:9.6.17
docker logs -f postgres_9

# CHECK:
# sudo add-apt-repository "deb https://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
# sudo apt-get update
# sudo apt install postgresql-client-common postgresql-client-10 -y
# psql -h localhost -p 15432 -U keycloak keycloak_db
# Ctrl+D

cd to certs dir

# MUST EXIST
file httpbin.example.com.crt
file httpbin.example.com.crt

# Keycloak server image linking to the postgres image
docker pull jboss/keycloak:9.0.0
docker run --name keycloak-data -v /var/lib/keycloak/data busybox:1.31.1 true
export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')
(docker rm keycloak || true)
docker run --rm -it -p 8443:8443 -p 8080:8080 --name keycloak --memory="512m" --link postgres_9 --volumes-from keycloak-data \
    -e PROXY_ADDRESS_FORWARDING=true \
    -e DB_VENDOR=postgres -e DB_ADDR=$MY_IP:15432  -e DB_PORT=15432 \
    -e DB_DATABASE=keycloak_db -e DB_SCHEMA=public \
    -e DB_USER=keycloak -e DB_PASSWORD=password \
    -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin_pass \
    -e KEYCLOAK_HTTPS_PORT=8443  -e KEYCLOAK_HTTP_PORT=8443 \
    -e KEYCLOAK_HOSTNAME=keycloak.example.com \
    -v httpbin.example.com.crt:/etc/x509/https/tls.crt \
    -v httpbin.example.com.crt:/etc/x509/https/tls.key \
    jboss/keycloak:9.0.0

        # Uncomment the line below if you want to specify JDBC parameters. The parameter below is just an example, and it shouldn't be used in production without knowledge. It is highly recommended that you read the PostgreSQL JDBC driver documentation in order to use it.
        # -e JDBC_PARAMS="ssl=true"

Add to host /etc/hosts file:
```bash
sudo -- sh -c -e "echo '127.0.0.1 keycloak.example.com' >> /etc/hosts";
sudo -- sh -c -e "echo '127.0.0.1 example.com' >> /etc/hosts";
cat /etc/hosts

Add to minikube /etc/hosts file:

export MY_IP=$(ip route get 8.8.8.8 | sed -n '/src/{s/.*src *\([^ ]*\).*/\1/p;q}')
minikube ssh -- sudo tee -a /etc/hosts <<EOF
127.0.0.1       localhost
127.0.1.1       minikube
127.0.0.1 example.com
$MY_IP keycloak.example.com
EOF
minikube ssh -- sudo cat /etc/hosts
minikube ssh -- ping keycloak.example.com

Open https://keycloak.example.com:8443/auth/admin/

Set sslRequired in realm settings

Navigate to https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/token-settings
Default Signature Algorithm RS256
Set Access Token Lifespan 60 min
Set Client login timeout 60 min

Navigate to https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/clients
OLDTODO~set up a client "pkce-test" (in the "master" realm) with access type "public"~~~
set up a client "pkce-test" (in the "master" realm) with access type "confidential"
set protocol to openid-connect
set Root URL to app url, like http://dev.local/
enable options Direct Access Grants and Authorization in the Settings section of newly created client. Also Access Type should be set to confidential
(to be able to use PKCE) and the "http://dev.local/*" as valid redirect URIs (which is a required field).
NOTE: replace dev.local with you app url
set Secret available under Credentials tab from Client section as "client_secret" in configmap

create a user "john_username" in https://keycloak.example.com:8443/auth/admin/master/console/#/realms/master/users with non-temporary password (credentials tab).

provider = "https://keycloak.example.com:8443/auth/realms/master"
username = "john_username"
password = "j000hn_pass" (not Temporary)
Remove Update password from Required User Actions

TODO: Create code verifier and challenge as in https://www.stefaanlippens.net/oauth-code-flow-pkce.html#PKCE-code-verifier-and-challenge

Open https://keycloak.example.com:8443/auth/realms/master/.well-known/openid-configuration to see endpoints

Open https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/certs
copy it and replace " with "
set as "jwks" in configmap (yaml file, on istio side)

Follows
https://medium.com/@robert.broeckelmann/openid-connect-authorization-code-flow-with-red-hat-sso-d141dde4ed3f
https://gist.github.com/ataube/2dd3632eeb4f0b46286c87005dc20c74
http://www.janua.fr/keycloak-access-token-verification-example/
https://qiita.com/rururu_kenken/items/5a7b94146cf0a2eb537d
https://dzone.com/articles/oauth-20-authorisation-code-grant

Now lets test Keycloak Server OIDC URI Endpoints https://www.keycloak.org/docs/latest/server_admin/index.html#keycloak-server-oidc-uri-endpoints

Run in shell
echo "https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/auth?response_type=code&client_id=pkce-test&state=38610846-c64d-4fd2-8d9-b5fa2f976298&redirect_uri=https://keycloak.example.com:8443/authorization-code/callback&scope=openid profile User"
Open new in incognito browser window url generatedfrom command above

change master to realm name, redirect_uri, client_id, e.t.c.

login as

username = "john_username"
password = "j000hn_pass" (must be not Temporary from step above)

NOTE: state is generated.
TODO: provide state generator and validator

example response:
http://localhost:4200/authorization-code/callback?session_state=a4930924-fbc5-40cc-9e47-a5b0d4d357f4&code=cfe46307-d7d3-420f-b438-c5cbeccb4f13.a4930924-fbc5-40cc-9e47-a5b0d4d357f4.2e7a2cba-9b0e-4840-93db-bbd29999d0dc

replace CODE_FROM_REQUEST below with answer to request above, just copy text after &code=

NOTE: If the redirect_uri was included in the authorization request, this value must be the same as the value used in that request.

export CODE_FROM_REQUEST=e4c91079-2c06-4157-a12d-c981dea7e6c3.0aa5bab3-a6e2-4d0a-babf-274adc241bd8.2aa47e2c-9dde-4b95-95d4-17c59ace967c

NOTE: client from keycloak settings

export CLIENT_SECRET=199c35ad-8e4b-4546-8e18-6f0d158eb364
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=authorization_code&client_id=pkce-test&client_secret=$CLIENT_SECRET&code=$CODE_FROM_REQUEST&redirect_uri=https://keycloak.example.com:8443/authorization-code/callback&scope=openid profile User&client_id=pkce-test&client_secret=$CLIENT_SECRET" https://keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/token

TODO: user must provide client_secret or not?

Example response:

{
"access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldU ...",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSld ...",
"token_type": "bearer",
"not-before-policy": 0,
"session_state": "04c1e2e4-e6a2-460f-841a-83ef532b01d8",
"scope": "email profile"
}

Decode the access_token in jwt.io
Open keycloak.example.com:8443/auth/realms/master/protocol/openid-connect/certs
VERIFY SIGNATURE with Public key from rsa-generated (Realm Settings -> Keys)

NOTE: replace rsa-generated in production


Run keycloak under https

Follows
https://stackoverflow.com/questions/59446436/keycloak-keystore-and-java-keystore-with-https-redirect-loop

Google for `keycloak Free SSL Certificates from Let's Encrypt`

Generate certs as in https://stackoverflow.com/a/58801128

Run keycloak as in https://stackoverflow.com/a/58801128 , provide `KEYCLOAK_HOSTNAME`, `KEYCLOAK_HTTPS_PORT`, `KEYCLOAK_HTTP_PORT` and volumes
```bash
// ...
 -p 8443:8443 --name keycloak --memory="512m" \
    -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin_pass \
    -e KEYCLOAK_HTTPS_PORT=8443  -e KEYCLOAK_HTTP_PORT=8443 \
    -e KEYCLOAK_HOSTNAME=keycloak.example.com \
-v /<path>/tls.crt:/etc/x509/https/tls.crt \
  -v /<path>/tls.key:/etc/x509/https/tls.key \
// ...

Open /auth/admin/
Navigate to /auth/admin/master/console/#/realms/master/clients
Set sslRequired to true in realm settings

PostgreSQL database that is hosted outside of both of the Keycloak servers.

Follow
https://homelab.blog/blog/security/keycloak-part-2-setting-up-keycloak/
https://homelab.blog/blog/devops/Istio-OIDC-Config/

Deploy istio with custom certs

Google "istio letsencrypt httpsRedirect"

Follow
https://homelab.blog/blog/devops/Istio-OIDC-Config/
https://istio.io/blog/2019/custom-ingress-gateway/
https://github.com/stefanprodan/istio-gke/blob/master/docs/istio/05-letsencrypt-setup.md
https://istio.io/docs/tasks/security/citadel-config/plugin-ca-cert/
https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/
https://istio.io/docs/ops/configuration/security/root-transition/
https://istio.io/docs/tasks/security/citadel-config/ca-namespace-targeting/
https://istio.io/docs/tasks/security/authentication/https-overlay/
https://itsmetommy.com/2019/10/10/kubernetes-istio-cert-manager-gke/

kubectl create secret generic cacerts -n istio-system --from-file=samples/certs/ca-cert.pem \
    --from-file=samples/certs/ca-key.pem --from-file=samples/certs/root-cert.pem \
    --from-file=samples/certs/cert-chain.pem

istioctl manifest apply --set values.global.mtls.enabled=true,values.security.selfSigned=false --set values.global.controlPlaneSecurityEnabled=true

# CAUTION!
 kubectl delete secret istio.default

@blockspacer

blockspacer commented Feb 28, 2020

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment