topcheck.py

README.md

What we can currently detect

• Duplicate match expressions
  • If merged, duplicate expressions will break state and pillar top file rendering
  • This has bitten us before, and is hard to detect with the files being so large
• Invalid/misspelled match types
  • e.g. - match: prce instead of pcre
  • Will break state and pillar top file rendering, even if the match expression under which it is found does not apply to your box
    • This is because each expression must be evaluated to see if it applies to a given minion

Requires

There are two requirements:

1. Python 3
   • I believe that there is no syntax newer than 3.2, so all of our GitHub Action Runners should be able to support this script
2. Salt is installed on the box
   • Our GitHub Action Runners have our single-executable salt-minion installed

Problem

However, the requirement that Salt is installed raises a problem; due to how Salt is packaged, the Python 3 environment that executes the script may not be the one in which Salt is installed.

In both cases, the solution to this issue involves first detecting the type of installation, and then using os.execvpe() to re-execute the script using an interpreter within the environment where Salt is installed.

The python family of os.exec* functions are equivalent to their counterparts in the C stdlib, and act like the exec command in shell scripting; the currently-running process is replaced in the process table by the new one.

One idiosyncrasy of os.execvpe() that is not well-documented on the Python docs is that the list of args passed as the 2nd argument must contain the entire argv. Thus, the first argument to os.execvpe() would be an executable that provides a Python interpreter, and the first element of the args passed to it would be the interpreter to invoke. For example:

import os
from typing import Any, Dict, List

command: List[str] = ["python3", "/path/to/script.py", "arg1", "arg2"]
env: Dict[str, Any] = os.environ.copy()
env["SOME_NEW_ENV_VAR"] = "value"

os.execvpe(command[0], command, env)

Single-executable distribution of Salt

Within Linode, we distribute salt-minion packages as a single executable zipapp, using shiv.

The salt codebase and its depchain are part of the zipapp. This means that they are not in a directory like /usr/lib/python3.x/site-packages.

To work around this, the script is re-executed with the SHIV_INTERPRETER environment variable set to true, which alters the behavior of the executable to instead present a Python interpreter within the shiv environment.

Upstream Salt Packages

Newer Salt releases are packaged upstream with a bundled Python, all under /opt/saltstack. These are called "onedir" packages.

This means that the salt codebase and depchain are located under /opt/saltstack, again not at the expected location.

To work around this, the script looks for either salt or salt-call in the system's PATH. If found, it checks to see if the executable is actually a symlink. A "onedir" package keeps everything, including the salt, salt-call, salt-key, etc. scripts under /opt/saltstack, and symlinks them into /usr/bin to ensure they are found in the shell's PATH. The symlink's path is resolved, and then a python3 executable is looked for at a path relative to the location of the linked script. If found, the script is re-executed via that Python interpreter.