Flink delegation token K8S test.md

Prerequisites:

• Install any container system like docker/Virtualbox/hyperkit
• Install minikube
• mkdir -p ${HOME}/share
• chmod 777 ${HOME}/share
• minikube start --mount-string="${HOME}/share:/share" --mount
• kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=default:default
• kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Clone the necessary repos:

cd
git clone [email protected]:gaborgsomogyi/docker-kdc.git
git clone [email protected]:gaborgsomogyi/docker-hdfs.git
git clone [email protected]:gaborgsomogyi/docker-hbase.git
git clone [email protected]:gaborgsomogyi/docker-hive.git
git clone [email protected]:gaborgsomogyi/flink.git
git clone [email protected]:gaborgsomogyi/flink-test-java-delegation-token-provider.git
git clone [email protected]:gaborgsomogyi/flink-test-java-job.git
git clone [email protected]:gaborgsomogyi/flink-docker.git
git clone [email protected]:gaborgsomogyi/flink-kubernetes-operator.git

Docker KDC console:

cd ${HOME}/docker-kdc
eval $(minikube docker-env)
docker-compose build
kubectl apply -f kdc.yaml

Docker HDFS console:

cd ${HOME}/docker-hdfs
cp *.xml ${HOME}/share
eval $(minikube docker-env)
docker build -t gaborgsomogyi/hdfs:latest .
kubectl apply -f hdfs.yaml

Docker HBase console:

cd ${HOME}/docker-hbase
cp *.xml ${HOME}/share
eval $(minikube docker-env)
docker build -t gaborgsomogyi/hbase:latest .
kubectl apply -f hbase.yaml

Docker Hive console:

cd ${HOME}/docker-hive
cp *.xml ${HOME}/share
eval $(minikube docker-env)
docker build -t gaborgsomogyi/hive:latest .
kubectl apply -f hive.yaml

Flink console:

cd ${HOME}/flink
mvn clean install -DskipTests -Dfast -Dscala-2.12

Delegation token provider console:

cd ${HOME}/flink-test-java-delegation-token-provider
mvn clean install
cp target/flink-test-java-delegation-token-provider-1.0.jar ${HOME}/flink/flink-dist/target/flink-1.18-SNAPSHOT-bin/flink-1.18-SNAPSHOT/lib/

Test java job console:

cd ${HOME}/flink-test-java-job
mvn clean install
cp target/flink-test-java-job-1.0.jar ${HOME}/flink/flink-dist/target/flink-1.18-SNAPSHOT-bin/flink-1.18-SNAPSHOT/examples/

Flink console:

mkdir -p ${HOME}/flink/flink-dist/target/flink-1.18-SNAPSHOT-bin/flink-1.18-SNAPSHOT/plugins/s3-fs-hadoop
cp flink-filesystems/flink-s3-fs-hadoop/target/flink-s3-fs-hadoop-1.18-SNAPSHOT.jar ${HOME}/flink/flink-dist/target/flink-1.18-SNAPSHOT-bin/flink-1.18-SNAPSHOT/plugins/s3-fs-hadoop
cd ${HOME}/flink/flink-dist/target/flink-1.18-SNAPSHOT-bin
tar -cvzf flink.tgz flink-1.18-SNAPSHOT
cp flink.tgz ${HOME}/flink-docker/latest/java11-debian

Flink docker console:

cd ${HOME}/flink-docker/latest/java11-debian
eval $(minikube docker-env)
docker build -t gaborgsomogyi/flink .

Flink k8s operator console:

cd ${HOME}/flink-kubernetes-operator
git checkout dt_test
eval $(minikube docker-env)
DOCKER_BUILDKIT=1 docker build . -t flink-kubernetes-operator
helm install flink-kubernetes-operator helm/flink-kubernetes-operator --set webhook.create=false"

Start an application with keytab on K8S

Flink k8s operator console:
kubectl apply -f examples/delegation-token-test.yaml

Start an application with ticket cache on K8S

Flink k8s operator console:
Remove the following from examples/delegation-token-test.yaml:

security.kerberos.login.keytab: ...
security.kerberos.login.principal: ...

kubectl apply -f examples/delegation-token-test.yaml

Watch the application

k9s

Stop K8S session

kubectl delete -f examples/delegation-token-test.yaml
kubectl delete pod/hive
kubectl delete pod/hbase
kubectl delete pod/hdfs
kubectl delete pod/kdc