setup airflow on docker container in ubuntu 20.04 server

airflow_on_docker

1. inbstall docker on ubuntu 20.04
# https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04
2. install airflow in docker
# https://hub.docker.com/r/puckel/docker-airflow
3. work with airflow
# https://airflow.apache.org/docs/apache-airflow/stable/start.html
4. create a user:
airflow users create -r Admin -u admin -e [email protected] -f admin -l user -p test


# outline VPN to docker container:
1. https://blog.ssdnodes.com/blog/outline-vpn-tutorial-vps/
2. ufw status
3. sudo ufw allow 1024:65535/tcp



# airflow form
https://medium.com/maisonsdumonde/road-to-add-form-for-airflows-dag-1dcf2e7583ef





# vs code on docker container:
https://hub.docker.com/r/codercom/code-server
0. go to the working directory
1. docker run -d -p 8081:8080 -v "$PWD:/home/coder/project" -u "$(id -u):$(id -g)" codercom/code-server
or
docker run -it -p 8081:8080 -v "$PWD:/home/coder/project" -u "$(id -u):$(id -g)" codercom/code-server
2. docker exec -it charming_dhawan /bin/bash
3. cat ~/.config/code-server/config.yaml
## remove docker container:
docker rm "container name"



# firewall adjustments:
ufw staus
sudo apt install net-tools


#biuld docker image
Run the command below from the folder containing Dockerfile
docker build --rm -t airflow-image .


sudo apt install python3.9
comvert python3 to python on ubuntu
update-alternatives --install /usr/bin/python python /usr/bin/python3 10
sudo apt install python3-pip
virtualenv --python=c:\Python25\python.exe envname



install airflow
grep dags_folder airflow.cfg 
airflow db reset
cd ../airflow-materials/data-pipelines/hellow_world.py dags/

stop airflow
kill <pid>
airflow webserver -p 8080 -D
airflow scheduler -D
bash to container with root privilages:
docker exec -u 0 -it mycontainer bash

airflow@cdcf48897012:~$ /entrypoint.sh airflow resetdb


build image:----------------
docker-compose up -d --build







kubernetes:
view the cluter:
doctl kubernetes cluster kubeconfig show k8s-1-19-3-do-2-sfo2-1609932770154
save the cluster:
doctl kubernetes cluster kubeconfig save 60b8d334-614c-4a1c-a49b-032707a5e244
delete the cluster:
kubectl config delete-cluster do-sfo2-k8s-1-19-3-do-2-sfo2-1609878022308
delete the cluster context from kubeconfig:
kubectl config delete-context do-sfo2-k8s-1-19-3-do-2-sfo2-1609878022308



download files using scp scp root@<server.ip>:/tmp/Downloads/p3/Udemy_Financial_Engineering_and_Artificial_Intelligence_in_Python_2020-11.part3_Downloadly.ir.rar /d/trash/trash/udemy



# push a docker image to docker hub:
build the image:  
docker build  -t kub-first-app .  
change the tag:  
docker tag kub-first-app visl/kub-first-app  
push the image:  
docker push visl/kub-first-app  


deploy docker image to k8s cluster:
kubectl create deployment first-app --image=visl/kub-first-app
create a service to expose container:
kubectl expose deployment first-app --type=LoadBalancer --port=8080