This gist is about a one-off process to convert the tower_* modules in Ansible core
into the AWX collection, and some miscelaneous tasks like cherry picking content in the
interium period.
We started out using the content collector tool, with some modifications. This extremely hacky bash script is what I used to document my steps. In the end, we did not use the tree created from this tool, but I did use it to check that what we had is right by diffing between the files it created and what my more manual process created.
To keep the commits from the original modules, I used this Stack Overflow copy pasta:
git filter-branch --index-filter 'git rm --cached -qr --ignore-unmatch -- . && git reset -q $GIT_COMMIT -- lib/ansible/modules/web_infrastructure/ansible_tower lib/ansible/module_utils/ansible_tower.py lib/ansible/plugins/inventory/tower.py' --prune-empty -- --all
This took 2 hours and 16 minutes to finish running. From output of that, I created this repo:
https://github.com/AlanCoding/ansible-tower-content-export
To merge this content back into AWX, I first did it this way:
git checkout devel
git checkout -b awx_modules_merge
git remote add module_content https://github.com/AlanCoding/ansible-tower-content-export.git
git merge module_content/devel --allow-unrelated-histories
I did not like the git "unrelated history", and rebased the branch. This re-parented the top commit so it looked like a more ordinary AWX change. Obviously, if you are starting fresh with a new collection repo, this shouldn't matter to you.
After the merge / rebase was finished, the files had to be moved into the standard
directory structure for collections (to allow building). This was a bit of a
headache, but eventually I got one nice clean commit to move all the files to
where the needed to be, via git mv commands.
Another topic - how do I list all new Ansible commits that modified any relevant files or directories? I never got a good answer to that. Best available option was to filter PRs by label.
There were several commits we have pulled in because they were merged into Ansible core after the AWX collection was created. Introducing the collection meant that we were basically maintaining a fork.
The cherry picking process involved 2 repos:
- Ansible itself is added as a remote to the content-export repo
- commit is cherry-picked there
- substantial modifications are made to remove content from other places think of docfragments, other modules, other parts of Ansible, etc.
- The content-export repo is added as an AWX remote
- the 2nd version of the commit is cherry-picked there
This worked surprisingly well, because the diff would ride through the file renames. In addition to porting merged fixes, we were also able to adopt a few high-quality open and closed PRs from Ansible core.
In our case, the module_utils imports had to be changed as:
# old imports
from ansible.module_utils.ansible_tower import TowerModule, tower_auth_config, tower_check_mode
# what it became
from ..module_utils.ansible_tower import TowerModule, tower_auth_config, tower_check_modeI believe it is probably a good idea for all collection maintainers to use this syntax.
So far, ensuring that we have the collection version of the modules installed has been a manual process. I apply this diff to the Ansible install that I'm using and verify that a module run will show this output:
diff --git a/lib/ansible/modules/web_infrastructure/ansible_tower/tower_organization.py b/lib/ansible/modules/web_infrastructure/ansible_tower/tower_organization.py
index bba58d8894..f3e9586e69 100644
--- a/lib/ansible/modules/web_infrastructure/ansible_tower/tower_organization.py
+++ b/lib/ansible/modules/web_infrastructure/ansible_tower/tower_organization.py
@@ -88,6 +88,7 @@ def main():
module.fail_json(msg='Failed to update the organization: {0}'.format(excinfo), changed=False)
json_output['changed'] = result['changed']
+ json_output['this_is_the_old_one'] = True
module.exit_json(**json_output)
Then I install the collection and verify that the output is not there. This is not ideal, and other collection maintainers have pointed this out too.
When testing, I prefer to use custom directories local to the project.
# In module source directory
ansible-galaxy collection build --output-path=alan
ansible-galaxy collection install alan/awx-awx-0.0.1.tar.gz -p alan
# In directory with test playbook
ANSIBLE_COLLECTIONS_PATHS=<rel_path>/alan/ ansible-playbook -i localhost, -e ansible_python_interpreter=$(which python) tower_module.yml
To test that it worked, I used a simple example playbook.
Your mileage may vary, but that's the general pattern.
I expect that --output-path in the build command is not necessary in most cases.
We used the same directory for the build and install here (but not the source...)
use whatever you want it shouldn't matter.
You can also create a directory structured symlinked to your source so that
you do not have to either build or install it.
That's useful for testing with playbooks (but not anything via ansible-test).
A proposal was eventually made to have a Makefile target which setups up
the symlink from ~/.ansible/collections to source, via make symlink_collection.
The ansible-doc command should also be tested wherever you did this build / install stuff.
ANSIBLE_COLLECTIONS_PATHS=alan ansible-doc -M alan/ansible_collections/awx/awx/plugins/modules/ -t module awx.awx.tower_organization
See my issue about using non-FQCN module names.
I've pretty much given up on that and accepted that any use of ansible-doc, in particular,
will require awx.awx.tower_organization and not tower_organization.
FQCN - Fully Qualified Collection Name
Here again, you should hack something into Ansible source code so that you know that it is picking up your modules and not the un-migrated ones.
Originally we wanted to keep the doc fragments as extends_documentation_fragment: tower.
However, from issues on this subject, there seems to be no plans at all for a "relative"
reference to doc fragments. I also held out hope that maybe we could just leave the
primary doc fragment inside of core. But I got this repo to test with:
https://github.com/bcoca/ansible-minmal
This sends a pretty clear message, all of the content will be purged. So a solution for the doc fragments has to be developed.
In our case, we had an additional complication that we wanted to be able to change the namespace. From the testing perspective, this is so that I could test upload to:
https://galaxy.ansible.com/alancoding/awx
In other words, both of these cases required some dynamic process of replacing lines in files before building. 2 different solutions were used for both:
- To substitute multiple values, we used a
galaxy.yml.j2template file - To cope with the doc fragment issue, we used a general find + replace in the modules directory
Both of these solutions were implemented in an Ansible playbook,
inside the awx_collection folder in the AWX PR.
With installing and manual testing taken care of, that leads us into publishing. Some of this is lightly documented in the AWX folder. More ancilary tasks related to this are documented in this other gist.
the good stuff is in AlanCoding/utility-playbooks, roles/publish-collection/tasks/main.yml.
This will run back over all the templating steps, then run the ansible-galaxy
commands of build and then publish.
Some of that is still in flux, and I will be adding the doc_fragments templating to it.
At first I did this wrong, trying to mark deprecated,
and that became very confusing.
The process converged on adding a simple migrated_to key in the .github/BOTMETA.yml file.
A Pull Request for this was opened and merged.
In 2020, that is now concretely the standard practice.
For AWX, I developed a wild hacky solution for testing that combines virtual environments, mocks requests, and uses the existing unit test suite to build responses. Pretty soon this will be simplified, because the need for installing tower-cli will be eliminated.
I would consider that closer to "unit" testing. Some collections have integration
tests, and some have unit tests like sensu.go.
That sensu.go link is a good example of how another collection is using a test client for unit tests.
This is most similar to what is being done in AWX, but with some additional piping.
A Makefile target for sanity testing was added which is make test_collection_sanity.
Over time, the relevance of this has increased.
We have had several issues where Ansible devel introduced a new rule.
This is going to continue to be a thing, since ansible-test is the best available megaphone to broadcast
needed changes to collections for deprecations and quality standards.
The construction of this sanity test target is ugly. Several issues have to be worked around. I put a great amount of detail in the issue about ignoring git content which is not checked in. It is still true that you can run sanity tests outside a git repo and have it work. The primary intention is probably that you first build, then install, then run the tests.
In the early days of collections, there were a couple of other issues, related to not recognizing collection module_utils right. It also took time to get it compatible with relative imports. But those have since been resolved, and I have been able to remove several hacks related to replacing the relative import.
Notable ones I have found are:
- https://github.com/ansible/ansible_collections_f5
- https://github.com/ansible/ansible_collections_google
- https://github.com/sensu/sensu-go-ansible
Some notable differences...
module imports can just be done with the fully-qualified namespace in most cases.
from ansible_collections.sensu.sensu_go.plugins.module_utils.base import sensu_argument_spec, AnsibleSensuClientThis should not cause any problems.
Also, doc fragments are rarely set up in a correct way. One notable exception:
extends_documentation_fragment:
- sensu.sensu_go.base
Full list of collections: https://galaxy.ansible.com/api/v2/collections/
The f5 collection does several things differently than our DOCUMENTION string.
notes:
- Requires BIG-IP >= 13.1.0
extends_documentation_fragment: f5
The documentation fragment probably isn't going to work. Also, the requirement differs from our pattern, which is:
requirements:
- ansible-tower-cli >= 3.0.2
On the requirements issue, I have no idea who is right, or if it's even possible to be right.
Most repos seem to hard-code in their version into galaxy.yml.
For f5, they mark special versions like:
0.9.0-alpha.build31.0.0-rc1
The AWX collection used a -beta
series for a while before the official first release.
If you want to dive much deeper into collection-to-collection dependencies,
I made a test repo with several examples.
If you use a -beta version number, dependency resolution will not work, so be warned.
If you subsequently release a A.B.C version afterwards, it doesn't pose any problems.
There are also a few other scenarios that prevent resolving dependencies or installing as a dependency.
So certainly collections need to be blacklisted if you were considering them as a dependency,
or else your own collection will be un-installable.
The actual use of collection dependencies is relatively niche, so this shouldn't actually affect anyone in practice.
https://github.com/AlanCoding/collection-dependencies-demo