Python installation candidate handling

README.md

states.yml lists some possible states we can consider for installable candidates and how we can create new artifacts that give us more information during the installation process.

Conventions:

1. If no action applies, than it's a failure
2. If no conditional state within an action applies, then it's a failure

One possible implementation may look like:

Represent a Candidate as a proxy object with an implementation like

class Candidate:
    @property
    def name(self):
        try:
            return self.obj.name
        except AttributeError:
            self._advance()
        return self.name

    @property
    def metadata(self):
        try:
            return self.obj.metadata
        except AttributeError:
            self._advance()
        return self.metadata

    def _advance(self):
        self.obj = next(self.obj)

where self.obj is an object of a type that represents one of the states above. Then using a property not provided by the current state advances it as-needed, doing any download or build as required.

states.yml

---
- name: Remote unnamed VCS
  example: git+https://github.com/pypa/pip.git
  actions:
  - name: download
    to_state: local unnamed vcs
  provides: []
  tags: [user-provided]

- name: Remote named VCS
  example: git+https://github.com/pypa/pip.git#egg=pip[all]-19.2.3
  actions:
  - name: download
    to_state: local named vcs
  provides:
  - name
  - name: version
    optional: True
    source: "#egg=..."
  - name: desired_extras
    optional: True
    source: "#egg=..."
  tags: [user-provided]

- name: Remote editable named VCS
  example: -e git+https://github.com/pypa/pip.git#egg=pip
  actions:
  - name: download
    to_state: local editable named vcs
  provides: [name]
  tags: [user-provided]

- name: Remote sdist
  example: https://files.pythonhosted.org/packages/00/9e/4c83a0950d8bdec0b4ca72afd2f9cea92d08eb7c1a768363f2ea458d08b4/pip-19.2.3.tar.gz
  actions:
  - name: download
    to_state: local sdist
  provides: [name, version]
  tags: [user-provided]
  notes:
  - Can be directly provided or returned from an index query if the user gave 'pip'

- name: Remote wheel
  example: https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl 
  actions:
  - name: download
    to_state: local wheel
  provides: [name, version, wheel_tag]
  tags: [user-provided]
  notes:
  - Can be directly provided or returned from an index query if the user gave 'pip'

- name: Local non-editable directory
  example: ./pip
  actions:
  - name: copy
    to_state: unpacked sources
    action: copy to temporary directory
    notes:
    - this is the current pip behavior
  - name: build sdist in place
    to_state: local sdist
    notes:
    - this is the proposed behavior, see discussion on https://github.com/pypa/pip/issues/2195
    - transitioning directly to local sdist means doing either a PEP 517 build_sdist or legacy setup.py sdist
  provides: []
  tags: [user-provided]

- name: Local editable directory
  example: -e ./pip
  actions:
  - name: query
    to_states:
    - name: local editable legacy project
      condition: setup.py is present
  provides: []
  tags: [user-provided]

- name: Local wheel
  example: dist/pip-19.2.3-py2.py3-none-any.whl
  actions:
  - name: unpack
    to_state: unpacked wheel
  provides: [name, version, wheel_tag]
  tags: [user-provided]

- name: Local editable legacy project
  actions:
  - name: install
    to_state: installed editable distribution

- name: Unpacked wheel
  actions:
  - name: install
    to_state: installed overwritable distribution
  provides: [name, version, metadata, dependencies]
  tags: [not-user-provided]

- name: Local sdist
  actions:
  - name: unpack
    to_state: unpacked sources
  provides: [name, version]
  tags: [user-provided]

- name: Local editable named VCS
  example: -e git+file://$PWD/pip#egg=pip
  actions:
  - name: query
    to_states:
    - name: local editable legacy project
      condition: if setup.py present
  provides:
  - name
  - name: version
    optional: True
  - name: desired_extras
    optional: True
  tags: [user-provided]

- name: Local unnamed VCS
  example: git+file://$PWD/pip
  actions:
  - name: query
    to_states:
    - name: local modern project
      condition: if pyproject.toml present
    - name: local legacy project
      condition: if setup.py present
  provides: []
  notes:
  - the reason for going directly to modern/legacy project (as opposed to
    treating this as unpacked sources) is because there may be a subdirectory
    indicated, so the behavior for determining the directory is different
  tags: [user-provided]

- name: Local named VCS
  example: git+file://$PWD/pip#egg=pip
  actions:
  - name: query
    to_states:
    - name: local modern project
      condition: if pyproject.toml present
    - name: local legacy project
      condition: if setup.py present
  provides:
  - name
  - name: version
    optional: True
    source: "#egg=..."
  - name: desired_extras
    optional: True
  tags: [user-provided]

- name: Local archive
  example: ./package.zip
  actions:
  - name: unpack
    to_state: local non-editable directory
  provides: []
  tags: [user-provided]

- name: Remote archive
  example: https://company.com/repo/package.zip
  actions:
  - name: download
    to_state: local archive
  provides: []
  tags: [user-provided]

- name: Installed overridable distribution
  notes:
  - Could represent a globally-installed package when we're installing to
    a user site or virtual environment.
  - TODO

- name: Installed overwritable distribution
  example: .venv/lib/pythonX.Y/site-packages/foo.dist-info (or egg-info)
  provides: [name, version, metadata, uninstallable]
  notes:
  - extras are missing
  - https://github.com/pypa/packaging-problems/issues/215#issuecomment-504569821

- name: Installed editable distribution
  example: .venv/lib/pythonX.Y/site-packages/foo.egg-link

- name: Unpacked wheel
  actions:
  - name: install
    action: copy files, generate
    to_state: installed overwritable distribution
  provides:
  - name: name
    source: .dist-info/METADATA
  - name: version
    source: .dist-info/METADATA
  - name: metadata
    source: .dist-info/METADATA
  tags: [not-user-provided]

- name: Unpacked sources
  example: /tmp/pip-<random>/pip
  actions:
  - name: query
    to_states:
    - name: local legacy project
      condition: setup.py present
    - name: local modern project
      condition: pyproject.toml present
  provides:
  - name: name
    optional: True
    notes:
    - if our previous state provided it (e.g. sdist)
  - name: version
    optional: True
    notes:
    - if our previous state provided it (e.g. sdist)
  - name: desired_extras
    optional: True
    notes:
    - if our previous state provided it (e.g. sdist)
  notes:
  - Should we depend on PKG-INFO? No. In the future we can use it if there’s
    a spec that says it can be trusted to have all relevant details (including
    dependencies)
  tags: [not-user-provided]

- name: Local legacy project
  example: /tmp/pip-<random>/pip (with setup.py)
  actions:
  - name: build
    to_state: local wheel
    when: "'wheel' is installed"
    action: setup.py bdist_wheel
  - name: install
    to_state: installed overwritable distribution
    action: setup.py install
  provides:
  - name: name
    source: setup.py egg_info
  - name: version
    source: setup.py egg_info
  - name: metadata
    source: setup.py egg_info
  notes:
  - if name and version were also provided by the successor of this state,
    then they would be validated against the ones returned by egg_info
  tags: [not-user-provided]

- name: Local modern project
  example: /tmp/pip-<random>/pip (with pyproject.toml)
  actions:
  - name: build
    action: PEP 517 `build_wheel` hook
    to_state: local wheel
  provides:
  - name: metadata
    when: "`prepare_metadata_for_build_wheel` is implemented"
    source: .dist-info generated by prepare_metadata_for_build_wheel
  tags: [not-user-provided]

verify.py

import os
from contextlib import contextmanager
from pathlib import Path

import yaml


obj = yaml.safe_load(
    Path(__file__).parent.joinpath('stages.yml').read_text(encoding='utf-8')
)

states_by_name = {}


for state in obj:
    states_by_name[state['name'].lower()] = state


@contextmanager
def error_reporter(name):
    try:
        yield
    except:
        print(f"Error processing {name}")
        raise


for state in obj:
    name = state['name']
    actions = state.get('actions', [])
    with error_reporter(name):
        for action in actions:
            try:
                to_states = [action['to_state']]
            except KeyError:
                to_states = [s['name'] for s in action['to_states']]
            for to_state_name in to_states:
                if to_state_name not in states_by_name:
 

- name: Unfetched remote archive
  example: https://company.com/repo/package.zip
  actions:
    download:
      result: Local archive
  provides: []

- name: Local legacy project
  example: /tmp/pip-<random>/pip (with setup.py)
  actions:
    build:
      if: `wheel` is installed
      do: setup.py bdist_wheel
      result: Local wheel
    install:
      if: `wheel` is NOT installed
      do: setup.py install
      result: Installed overwritable distribution
  provides: [name, version, metadata]

Noting that "unpacked wheel" shows up twice here: 10th and 20th element.

Alsoooo modern projects, can't be installed directly. They have to go through being built into a wheel.