Skip to content

Instantly share code, notes, and snippets.

@pfmoore
Created March 12, 2020 13:48
Show Gist options
  • Save pfmoore/36c3d3ace6ec38ce42515ff540411634 to your computer and use it in GitHub Desktop.
Save pfmoore/36c3d3ace6ec38ce42515ff540411634 to your computer and use it in GitHub Desktop.
Lifecycle of an InstallRequirement
==================================
Three types of InstallRequirement:
* A specifier, something like "pip>=19.0" or just "pip".
* A location, which could be:
- A URL - https://example.com/wheels/foo-1.0-py3-none-any.whl
- A directory - D:\Wheels\foo-1.0-py3-none-any.whl
- A VCS URL - git+https://github.com/pypa/pip
* An editable, something like "-e ."
There are also lots of other terms, like "direct". We'll come to them...
Creating an InstallRequirement
==============================
Lots of exciting parsing goes on, but it all ends up at one of the
constructors in pip._internal.req.constructors:
* install_req_from_editable
* install_req_from_line
Constructing the resolver uses make_req_from_req_string. TODO: go back
to this path and check. For now, ignore it for the sake of my sanity.
line:
return InstallRequirement(
parts.requirement,
* comes_from,
* link=parts.link,
markers=parts.markers,
* use_pep517=use_pep517,
* isolated=isolated,
* install_options=options.get("install_options", []) if options else [],
* global_options=options.get("global_options", []) if options else [],
* hash_options=options.get("hashes", {}) if options else {},
* wheel_cache=wheel_cache,
* constraint=constraint,
* extras=parts.extras,
)
editable:
return InstallRequirement(
parts.requirement,
* comes_from,
source_dir=source_dir,
editable=True,
* link=parts.link,
* constraint=constraint,
* use_pep517=use_pep517,
* isolated=isolated,
* install_options=options.get("install_options", []) if options else [],
* global_options=options.get("global_options", []) if options else [],
* hash_options=options.get("hashes", {}) if options else {},
* wheel_cache=wheel_cache,
* extras=parts.extras,
)
Note that the first argument is a Requirement (but not an InstallRequirement!)
This is from pip._vendor.packaging.requirements.
So we are now at the constructor of the class.
Most stuff just bunged into attributes.
* source_dir is normpath(abspath())
* if req and req.url (PEP 508 URL req) then set link to req.url if not given explicitly
* local_file_path is from link, if it's a file
* extras come from req if not set
* markers come from req if not set
Plus a chunk of (mutable) state:
* satisfied_by
* should_reinstall
* _temp_build_dir
Set in ensure_build_location, which is used in ensure_has_source_dir
to set source_dir if unset. Which in turn is called in prepare.
* install_succeeded
* prepared
Set in resolver, *not* in operations.prepare!!!
Only used by resolver...!!!
* is_direct
Set in req_command...
Means "Was specified by the user in the command line or requirement/constraint file"
* successfully_downloaded
* build_env
Set in pip._internal.distributions.sdist
* metadata_directory
Set by prepare_metadata() from _generate_metadata()
* pyproject_requires
* requirements_to_check
* pep517_backend
* use_pep517
These 4 set by load_pyproject_toml from pip._internal.distributions.sdist
Install Command
===============
Question - what is a requirement tracker??
get_requirements - from args. Note check_supported_wheels, true unless --target.
* Parse constraint files. Reqs have constraint=true
* Add rqs from command line
* Add editable reqs
* Add reqs from requirement files
All these reqs are is_direct=True
Question - what state are requirements in at this point??
Answer - freshly built, no significant (lifecycle) methods called.
resolve.
Question - what state are requirements in at this point??
Go on to build and install steps here.
Resolve Process
===============
_resolve_one
Set prepared
_get_abstract_dist_for, get_pkg_resources_distribution
** What is an "unnamed" requirement (quotes from comment in the code - hence the question!)? Do we care?
It's an artifact of the resolver and reuqirement sets.
* Some requirements (e.g., ".") don't have a defined name until we get the metadata.
* It's possible to have unnamed requirements as *dependencies* (foo depends on ".")?? Maybe?
* Need to rescan to get the name. Not sure if rescanning rather than preparing now is
just because the existing resolver sucks, or if there's a better reason...
_get_abstract_dist_for
Editable reqs get prepared. That's it for them. Simples.
req.ensure_has_source_dir
update_editable
_get_prepared_distribution
_check_skip_installed. Sets satisfied_by.
*This is the code that makes installed requirements!*
So it's not until this point that we know if this is an install or an upgrade.
populate_link
Calls the finder to find the requirement (if .link not set)
Checks for a cached wheel and uses that if possible
prepare_linked_requirement
req.ensure_has_source_dir
unpack_url
_get_prepared_distribution
<< req.archive() on VCS source dir >> - only of interest because it's the place req.archive is used.
I'm going to say at this point that I consider "prepared" as the "complete" point for the resolver, because preparing
returns an abstract_dist, which can be converted into a pkg_resources distribution.
But if you *lose* that abstract_dist, you can't get it back without preparing again...
_get_prepared_distribution(req,tracker,finder,build_isolation)
make_distribution_for_install_requirement
tracker.track(req)
prepare_distribution_metadata(finder, build_isolation)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment