development-oriented overview of the Ansible execution environment ecosystem

AWX execution environment image parents

The first half of this table lists base images for execution environments (EE).

Quay page (`ansible/*`)	Tag	Repository (`ansible/*`)	Default branch
	`quay.io/centos/centos:8`		`main`
python-base	`quay.io/ansible/python-base:latest`	python-base-image	`main`
python-builder	`quay.io/ansible/python-builder:latest`	python-builder-image	`main`
ansible-builder	`quay.io/ansible/ansible-builder:latest`	ansible-builder	`devel`
ansible-runner	`quay.io/ansible/ansible-runner:latest` `quay.io/ansible/ansible-runner:stable-2.9-devel`	ansible-runner	`devel`
	--- execution environments ---
awx-ee	`quay.io/ansible/awx-ee:latest`	awx-ee	`devel`
network-ee	`quay.io/ansible/network-ee:latest` `quay.io/ansible/network-ee:stable-2.9`	network-ee	`main`

ansible-builder is used to create the Containerfile/Dockerfile and build context for an EE. The output from ansible-builder is checked into source, and then automation will build and push images for release of awx-ee and network-ee.

Documentation pages on execution environments:

The execution environments section of ansible-runner docs
All of the ansible-builder docs
First blog post on ansible-builder

Note that the ansible-runner image tag is devel, but it will change to latest once its 2.0 version is released.

Managing git Repositories

These are 5 separate repos, and you can make it 6 if you also want to build the network-ee.

Zuul builds in the images in quay.io, but these are instructions to do it on your own. Keeping all 5 or 6 repos up-to-date is unreasonable without automation. For this problem I am using the mu repo manager:

# your desired folder to contain git repos
mkdir repos
cd repos
# clone all the repos manually, git clone https://github.com/ansible/ansible-runner.git, etc.
pip install mu-repo
mu group add ee --empty
mu register python-base-image python-builder-image ansible-core-image ansible-runner awx-ee

This gets you the initial setup. The flow for maintaining the git state looks like this:

mu status
mu pull --rebase origin

This is tollerant to different default branches, which is necessary here. I rename the remote "origin" to "ansible", as a personal preference.

Building the build chain

Working from your root folder container the git repositories on the desired branches (probably all main/devel), you can source this script. Doing this will build all of the base images.

podman build -f python-base-image/Containerfile python-base-image -t quay.io/ansible/python-base:latest
podman build -f python-builder-image/Containerfile python-builder-image -t quay.io/ansible/python-builder:latest
podman build -f ansible-core-image/Dockerfile ansible-core-image -t quay.io/ansible/ansible-core:latest
podman build -f ansible-runner/Dockerfile ansible-runner -t quay.io/ansible/ansible-runner:devel

Building the base images should be rarely needed. This is mainly for development purposes. A example situation where this is needed is when the ultimate base image changes, say, from Fedora to CentOS. That will change every image from the python-base and python-builder images on downward.

Building an Execution Environment

If you do not yet have the base images on your computer, building an EE will pull them from quay.io.

There are 2 ways of building from an EE's repo. The more "complete" way is to use ansible-builder. If you have unversioned collections in your requirements.yml and those upstreams change something, this will pick up those changes and can change the files in the EE's source.

ansible-builder build -v3 -c awx-ee -f awx-ee/execution-environment.yml -t quay.io/ansible/awx-ee:latest

Alternative for last step is that you use podman directly instead of ansible-builder. This will not pick up requirement changes from upstream collections.

podman build -f awx-ee/Dockerfile awx-ee -t quay.io/ansible/awx-ee:latest

Otherwise, the two commands should give the same result. The use of ansible-builder build is more development oriented.

Building stable-2.9 image

To build with Ansible 2.9, you run the same commands but with different options. I did this by checking out the ansible-runner branch pabelanger:temp/stable-2.9-image which is for PR ansible/ansible-runner#590. That has since been merged, so these steps should work for using the devel or main branch of all the involved repos.

This requires modifying the collection requirements because Ansible 2.9 does not support SCM collection requirements. I did it this way:

diff --git a/sources/requirements.yml b/sources/requirements.yml
index 753d038..00bea84 100644
--- a/sources/requirements.yml
+++ b/sources/requirements.yml
@@ -8,14 +8,10 @@ collections:
   - name: theforeman.foreman  # has requirements.txt (which -r to another file)
   - name: google.cloud  # has requirements.txt, mainly for google-auth
   # forked from opendev.org
-  - name: https://github.com/AlanCoding/ansible-collections-openstack.git
-    version: ee_req_install
-    type: git
+  - name: openstack.cloud
   - name: community.vmware  # has requirements.txt, but may add pyvcloud
-  - name: https://github.com/shanemcd/ovirt-ansible-collection
-    type: git
-  - name: https://github.com/ansible-collections/community.kubernetes.git
-    type: git
+  # - name: ovirt.ovirt
+  - name: community.kubernetes
   # adds openshift python lib
   # needs kubectl for yum / dnf / apt-get
   # needs to install snap, then use snap to install helm

Again, from the repos root:

podman build -f ansible-runner/Dockerfile ansible-runner -t quay.io/ansible/ansible-runner:stable-2.9-devel --build-arg ANSIBLE_BRANCH=stable-2.9
ansible-builder build -v3 -c awx-ee -f awx-ee/execution-environment.yml -t quay.io/ansible/awx-ee:stable-2.9-latest -b quay.io/ansible/ansible-runner:stable-2.9-devel

Result

As of running 1/14/2021, strictly using the main branches of devel and main.

$ podman images
REPOSITORY                         TAG               IMAGE ID      CREATED         SIZE
quay.io/ansible/awx-ee             latest            b5a04133055b  59 seconds ago  1.07 GB
quay.io/ansible/ansible-core       latest            ed37f30d6b10  12 minutes ago  519 MB
quay.io/ansible/python-builder     latest            f13a13deb2b9  2 days ago      376 MB
quay.io/ansible/python-base        latest            825988f1c736  2 days ago      357 MB
quay.io/ansible/ansible-runner     devel             30bbfe2f32b5  3 weeks ago     441 MB
quay.io/ansible/ansible-runner     stable-2.9-devel  a79f7da2619e  4 weeks ago     530 MB
quay.io/centos/centos              8                 300e315adb2f  5 weeks ago     217 MB

Zuul Dependencies

All of these 5 repos (including ansible-builder) will build images as a part of CI, and are connected in Zuul.

That means that you may, hypothetically, be able to make a pull request to ansible/awx-ee that depends on a PR in all the other 4 repos, although in practice the lower-level images should change infrequently.

Example of running CI with a linked PR ansible/ansible-runner#594

Depends-On: ansible/python-builder-image#17

Using this syntax, the CI will use a parent image from the other repo for checks.

Ansible python-builder-image script functionality

The image quay.io/ansible/python-builder has scripts which are central to ansible-builder functionality. Those are inside of the repo:

$ tree python-builder-image/scripts/
python-builder-image/scripts/
├── assemble
├── get-extras-packages
└── install-from-bindep

0 directories, 3 files

The assemble and install-from-bindep files are invoked in ansible-builder Containerfiles. The get-extras-package is called from inside the assemble script.

Inputs / Outputs is summarized here:

Location inside container	Takes inputs from	Outputs files to
/usr/local/bin/assemble	/tmp/src/requirements.txt	/output/bindep/run.txt
	/tmp/src/bindep.txt	/output/bindep/epel.txt
	(extras??)
/usr/local/bin/get-extras-packages
/output/install-from-bindep	/output/requirements.txt	`<python-site-packages>`
	/output/bindep/run.txt	`<dnf install location>`
	/output/bindep/epel.txt	`<dnf install location>`

These are employed in different stages.

The /tmp/src/ location is populated by the Containerfile, ultimately coming from the ansible-builder introspect script. Inside of the builder stage, requirements inside of /tmp/src/ are processed and then modified requirements are written to /output. Then the /output folder is copied from the builder stage to the final stage.

An exception to this flow is that the install-from-bindep is already inside the /output folder, and thus is also copied to the final stage. In the final stage, that script is ran which then runs dnf and pip installs.

Commands ran by scripts

To produce these, some hacks were added on top of the python-builder image. Without this, there are lots of 1,000s of lines of code which are difficult to read.

For the RUN assemble layer in the builder stage, from the awx-ee runs:

cd /tmp/src

# These lines process the bindep inputs and make them ready for dnf
bindep -l newline | sort >> /output/bindep/run.txt || true
bindep -l newline -b epel | sort >> /output/bindep/stage.txt || true
grep -Fxvf /output/bindep/run.txt /output/bindep/stage.txt >> /output/bindep/epel.txt || true
rm -rf /output/bindep/stage.txt

# Install the system compile dependencies
compile_packages=$(bindep -b compile || true)
dnf install -y ${compile_packages}

# Cache the python requirements to /output/wheels
/tmp/venv/bin/pip install --cache-dir=/output/wheels -r /tmp/src/requirements.txt
cp /tmp/src/requirements.txt /output/requirements.txt

For the RUN /output/install-from-bindep && rm -rf /output/wheels layer in the final stage of the build:

dnf install -y $(cat /output/bindep/run.txt)
dnf install -y --enablerepo epel $(cat /output/bindep/epel.txt)
pip3 install --cache-dir=/output/wheels -r /output/requirements.txt

Locations and internal constructs

This attempts to summarize "stuff" that you need to follow the build flow.

the /tmp/src folder - semi-processed requirements
- bindep.txt is the main bindep manifest
  - this is processed and written to /output/bindep/*.txt, depending on the profile
  - the entries without a profile are saved to /output/bindep/run.txt
- requirments.txt is the python manifest
the /tmp/venv folder
- python install location used by assemble script, an intermediary
the /output folder
- wheels folder is the pip cache, populated by the builder stage and used by the final stage
- requirements.txt python requirements copied unmodified from /tmp/src
- bindep/run.txt simple dnf install input
- bindep/epel.txt input for dnf install with epel repo enabled
- packages.txt is not used for execution environments

AlanCoding/Ansible_EE_images.md