install_awx_operator.md

Install AWX-operator on Ubuntu 20 using Minikube

I’ve had a few requests for this demo as It seems like a lot of people are having issues setting up the new version of AWX. Before v18, (v17) you could simply install AWX on a server with enough resources. Now, the preferred way to install AWX is via the AWX-operator. So you need a Kubernetes or OpenShift cluster. For this demo, I’m using Minikube.

Subscribe To Me On YouTube: https://bit.ly/lon_sub

(if you get any errors, check the bottom of this page for the fix)

I’m using minikube because it’s a single node cluster and it keeps the price down. I did try this on a t2.medium, but there just weren’t enough resources to get it working.

This demo will cover the following:

• Install and setup kubectl and docker.
• Installing and setting up Minikube in EC2 [t3a.xlarge – 4 x cpu / 16gb memory]
  • ami-0194c3e07668a7e36
  • You might be able to get away with less resources)
• Setup the AWX-operator (latest version – 0.10.0) and deploy.
• Get the AWX admin password.
• Forward the service port so we can access it from the internet and use minikube tunnel.
• Login to AWX and take a quick tour.
• All commands will be added into the description.

Make sure you have enough FREE Disk space. 10GB+ please

Links:

https://github.com/ansible/awx-operator https://github.com/ansible/awx-operator/tags

If you like the demo, hit subscribe for more videos like this and give it a like.

Commands:

setup the repo's:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version --client
sudo apt-get update -y &&  sudo apt-get install -y docker.io

Install minikube:

There is an issue with the latest version (1.22) Please install 1.21 for now.

curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/```

Use this one below for the time being:

curl -Lo minikube https://github.com/kubernetes/minikube/releases/download/v1.21.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
sudo usermod -aG docker $USER
LOGOUT & BACK IN AGAIN
groups $USER

minikube start

minikube start --addons=ingress --cpus=2 --install-addons=true --kubernetes-version=stable --memory=6g

Check its working:

kubectl get nodes
kubectl get pods
kubectl get pods -A

Install the AWX Operator:

kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.10.0/deploy/awx-operator.yaml
kubectl get pods

Create the deployment file:

vi awx-demo.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  ingress_type: none
  hostname: awx-demo.example.com

Run the deployment:

kubectl apply -f awx-demo.yml
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"

WAIT A FEW MINS...

Get the Admin user password:

kubectl get secrets
kubectl get secret awx-demo-admin-password -o jsonpath="{.data.password}" | base64 --decode

Expose the deployment:

kubectl expose deployment awx-demo --type=LoadBalancer --port=8080

Minikube tunnel

On a new session, start the minikube tunnel:

minikube tunnel

Enable AWX to be access via the Internet:

kubectl port-forward svc/awx-demo-service --address 0.0.0.0 30886:80

Now visit https://your_ip:high_port

You may need to update your FW rules to be able to connect to the AWX login screen

---

--- ISSUES SECTION ---

1. Starting minikube tunnel - Exiting due to GUEST_STATUS: state: unknown state "minikube": docker container inspect minikube --format=: exit status 1 ANS: sudo chmod 666 /var/run/docker.sock ; sudo usermod -aG docker ${USER}

2. If you see the message: ImagePullBackOff or ErrImagePull when you run kubectl get pods, run the following command to see what the issue is:

kubectl describe pods <my-pod> # Output from "kubectl get pods" command for the pod with the issue.

I've seen a few issues where the server runs out of space so it should be easy to fix.

3. If you have PENDING resources, try running a describe of the resource. If you see this at the bottom you are lacking in resources: ( kubectl describe pods <my-pod> # Output from kubectl get pods )

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  14s (x19 over 23m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

This link has some more information: https://containersolutions.github.io/runbooks/posts/kubernetes/0-nodes-available-insufficient/

I reinstall it again and not luck. These are my logs:

NAME READY STATUS RESTARTS AGE
pod/awx-demo-7744b86597-d45q4 4/4 Running 0 13m
pod/awx-demo-postgres-0 1/1 Running 0 13m
pod/awx-operator-5dd757f594-7x98q 1/1 Running 0 13m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-demo LoadBalancer 10.99.22.95 10.99.22.95 8080:32194/TCP 11m
service/awx-demo-postgres ClusterIP None 5432/TCP 28m
service/awx-demo-service NodePort 10.100.237.74 80:30717/TCP 27m
service/awx-operator-metrics ClusterIP 10.105.183.10 8383/TCP,8686/TCP 13m
service/kubernetes ClusterIP 10.96.0.1 443/TCP 28m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx-demo 1/1 1 1 2d3h
deployment.apps/awx-operator 1/1 1 1 2d3h

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-demo-7744b86597 1 1 1 64m
replicaset.apps/awx-demo-7bbb564887 0 0 0 2d3h
replicaset.apps/awx-operator-5dd757f594 1 1 1 65m
replicaset.apps/awx-operator-849b7d7d9d 0 0 0 2d3h

NAME READY AGE
statefulset.apps/awx-demo-postgres 1/1 2d3h

test:~$ kubectl describe pods awx-demo-7744b86597-d45q4
Name: awx-demo-7744b86597-d45q4
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Fri, 08 Oct 2021 13:42:20 +1000
Labels: app.kubernetes.io/component=awx
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=awx-demo
app.kubernetes.io/part-of=awx-demo
app.kubernetes.io/version=19.2.0
pod-template-hash=7744b86597
Annotations:
Status: Running
IP: 172.17.0.2
IPs:
IP: 172.17.0.2
Controlled By: ReplicaSet/awx-demo-7744b86597
Containers:
redis:
Container ID: docker://d6ed2245db5ea3c7ad124155d02cdcb258ce232606a75f4c9e721048b7a3699a
Image: docker.io/redis:latest
Image ID: docker-pullable://redis@sha256:2d692e20ba38bf5382d497ee975f0b3389abee73e030cc3315d78c180bf19d66
Port:
Host Port:
Args:
redis-server
/etc/redis.conf
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Environment:
Mounts:
/data from awx-demo-redis-data (rw)
/etc/redis.conf from awx-demo-redis-config (ro,path="redis.conf")
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
awx-demo-web:
Container ID: docker://8ef9d812e55379dfdc0bf2812f11aaf0dbcbf4dc4ed398ba3e9bf6614b3a69ef
Image: quay.io/ansible/awx:19.2.0
Image ID: docker-pullable://quay.io/ansible/awx@sha256:f7cdabee0da2ea195e3dab8a8b39f3f5f1f32f0d2ee3d0ac561ec7d640d7042d
Port: 8052/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Requests:
cpu: 1
memory: 2Gi
Environment:
MY_POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/etc/nginx/nginx.conf from awx-demo-nginx-conf (ro,path="nginx.conf")
/etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
/etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
/etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
/etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
/etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/lib/awx/rsyslog from rsyslog-dir (rw)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
/var/run/supervisor from supervisor-socket (rw)
awx-demo-task:
Container ID: docker://19710ce432b9f9cbc9a4d1ddac026554e502a68ebdd58a3325d787a0fa13da12
Image: quay.io/ansible/awx:19.2.0
Image ID: docker-pullable://quay.io/ansible/awx@sha256:f7cdabee0da2ea195e3dab8a8b39f3f5f1f32f0d2ee3d0ac561ec7d640d7042d
Port:
Host Port:
Args:
/usr/bin/launch_awx_task.sh
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Requests:
cpu: 500m
memory: 1Gi
Environment:
SUPERVISOR_WEB_CONFIG_PATH: /etc/supervisord.conf
AWX_SKIP_MIGRATIONS: 1
MY_POD_UID: (v1:metadata.uid)
MY_POD_IP: (v1:status.podIP)
MY_POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/etc/tower/SECRET_KEY from awx-demo-secret-key (ro,path="SECRET_KEY")
/etc/tower/conf.d/credentials.py from awx-demo-application-credentials (ro,path="credentials.py")
/etc/tower/conf.d/execution_environments.py from awx-demo-application-credentials (ro,path="execution_environments.py")
/etc/tower/conf.d/ldap.py from awx-demo-application-credentials (ro,path="ldap.py")
/etc/tower/settings.py from awx-demo-settings (ro,path="settings.py")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/lib/awx/rsyslog from rsyslog-dir (rw)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/receptor from receptor-socket (rw)
/var/run/redis from awx-demo-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
/var/run/supervisor from supervisor-socket (rw)
awx-demo-ee:
Container ID: docker://b90510fcb1ba2af2d1fc4f455bbd1e336f3882674eeed33109cdf892ecbba638
Image: quay.io/ansible/awx-ee:0.3.0
Image ID: docker-pullable://quay.io/ansible/awx-ee@sha256:885facada773ef85bfd4fc952a268f3d6e4331d5d134e79c54bb2bb201f81968
Port:
Host Port:
Args:
receptor
--config
/etc/receptor.conf
State: Running
Started: Fri, 08 Oct 2021 13:42:22 +1000
Ready: True
Restart Count: 0
Environment:
Mounts:
/etc/receptor.conf from awx-demo-receptor-config (ro,path="receptor.conf")
/var/lib/awx/projects from awx-demo-projects (rw)
/var/run/receptor from receptor-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from awx-demo-token-4b6tl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
awx-demo-application-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-app-credentials
Optional: false
awx-demo-secret-key:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-secret-key
Optional: false
awx-demo-settings:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-redis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-redis-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-redis-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
supervisor-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
rsyslog-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
receptor-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
rsyslog-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-receptor-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-demo-awx-configmap
Optional: false
awx-demo-projects:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
awx-demo-token-4b6tl:
Type: Secret (a volume populated by a Secret)
SecretName: awx-demo-token-4b6tl
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 14m default-scheduler Successfully assigned default/awx-demo-7744b86597-d45q4 to minikube
Normal Pulled 14m kubelet Container image "docker.io/redis:latest" already present on machine
Normal Started 14m kubelet Started container awx-demo-web
Normal Started 14m kubelet Started container redis
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx:19.2.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-web
Normal Created 14m kubelet Created container redis
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx:19.2.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-task
Normal Started 14m kubelet Started container awx-demo-task
Normal Pulled 14m kubelet Container image "quay.io/ansible/awx-ee:0.3.0" already present on machine
Normal Created 14m kubelet Created container awx-demo-ee
Normal Started 14m kubelet Started container awx-demo-ee

Firewall
test:~$ sudo ufw status
Status: active

To Action From

---

30710:30720/tcp ALLOW Anywhere

Did you create a larger VM to install AWX on [more CPU and more memory]? I would advise you delete the original VM, create a new one with enough resources and then follow the video again.

hi, trying to use callback to provision my ec2 instance curl -H 'Content-Type: application/json' --data '{"host_config_key": "key"}' http://IP:30877/api/v2/job_templates/13/callback/ but I get this error: No matching host could be found!. I added HTTP_X_FORWARDED_FOR to configuration but it still doesn't work.

Followed your instruction but only the awx-demo-postgres comes up. But no error message.
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME READY STATUS RESTARTS AGE
awx-demo-postgres-0 1/1 Running 0 35m

Followed your instruction but only the awx-demo-postgres comes up. But no error message. kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" NAME READY STATUS RESTARTS AGE awx-demo-postgres-0 1/1 Running 0 35m

Using this one now. Very good.
https://github.com/kurokobo/awx-on-k3s

Followed your instruction but only the awx-demo-postgres comes up. But no error message. kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" NAME READY STATUS RESTARTS AGE awx-demo-postgres-0 1/1 Running 0 35m

Hi Max18-100. I've not looked at this for a long time! The thing that came up many times before was the VM not having enough resources. In my demo, I used a t3a.xlarge – 4 x cpu / 16gb memory in AWS. You will need at least 4 CPUs and 8GB memory for it to work properly.

I did try to get a T2.medium working (2CPU X 4GB) but it wasn't enough.

Does your VM have enough resources? Alternatively try the K3 install.: https://github.com/kurokobo/awx-on-k3s. I have another video with a demo on my channel here: https://www.youtube.com/watch?v=xVOnBDvpbK0