Export a Conda environment with --from-history, but also append Pip-installed dependencies

conda_env_export.py

"""
Export a Conda environment with --from-history, but also append
Pip-installed dependencies

Exports only manually-installed dependencies, excluding build versions, but
including Pip-installed dependencies.

Lots of issues requesting this functionality in the Conda issue tracker,
e.g. https://github.com/conda/conda/issues/9628

TODO (?): support command-line flags -n and -p

MIT License:
    
    Copyright (c) 2021 @gwerbin

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
"""
import re
import subprocess
import sys

import yaml


def export_env(history_only=False, include_builds=False):
    """ Capture `conda env export` output """
    cmd = ['conda', 'env', 'export']
    if history_only:
        cmd.append('--from-history')
        if include_builds:
            raise ValueError('Cannot include build versions with "from history" mode')
    if not include_builds:
        cmd.append('--no-builds')
    cp = subprocess.run(cmd, stdout=subprocess.PIPE)
    try:
        cp.check_returncode()
    except:
        raise
    else:
        return yaml.safe_load(cp.stdout)


def _is_history_dep(d, history_deps):
    if not isinstance(d, str):
        return False
    d_prefix = re.sub(r'=.*', '', d)
    return d_prefix in history_deps


def _get_pip_deps(full_deps):
    for dep in full_deps:
        if isinstance(dep, dict) and 'pip' in dep:
            return dep


def _combine_env_data(env_data_full, env_data_hist):
    deps_full = env_data_full['dependencies']
    deps_hist = env_data_hist['dependencies']
    deps = [dep for dep in deps_full if _is_history_dep(dep, deps_hist)]

    pip_deps = _get_pip_deps(deps_full)

    env_data = {}
    env_data['channels'] = env_data_full['channels']
    env_data['dependencies'] = deps
    env_data['dependencies'].append(pip_deps)

    return env_data


def main():
    env_data_full = export_env()
    env_data_hist = export_env(history_only=True)
    env_data = _combine_env_data(env_data_full, env_data_hist)
    yaml.dump(env_data, sys.stdout)
    print('Warning: this output might contain packages installed from non-public sources, e.g. a Git repository. '
          'You should review and test the output to make sure it works with `conda env create -f`, '
          'and make changes as required.\n'
          'For example, `conda-env-export` itself is not currently uploaded to PyPI, and it must be removed from '
          'the output file, or else `conda create -f` will fail.', file=sys.stderr)


if __name__ == '__main__':
    main()

requirements.txt

pyyaml

This is lovely! Would be great if this otherwise kept to the conda env export format, e.g. in terms of indentation.

This is nice!
How do I modify this if I would like to include name in yml?

Thanks @gwerbin !

How do I modify this if I would like to include name in yml?

add env_data['name'] = env_data_full['name'] after line 58.

did this at: https://gist.github.com/etihwnad/55c974b691fea8105b8b52ac376b83ed

I post your codes on here, thank you for your lovely work!

Thx for this lovely script!
For others also working on Windows: I had to modify line 29 into cp = subprocess.run(cmd, stdout=subprocess.PIPE, shell=True) to make it runnable.

Thx for this lovely script! For others also working on Windows: I had to modify line 29 into cp = subprocess.run(cmd, stdout=subprocess.PIPE, shell=True) to make it runnable.

I'm surprised to see that you needed shell=True. I had originally developed this script on Windows (although that was quite a while ago now), so it should work as-is on that platform.

Does conda env now require shell integration on some platforms?

---

Note that you can use subprocess.check_output() to simplify the code a bit:

    output = subprocess.check_output(cmd)
    return yaml.safe_load(output)

I will want to test this of course, but I'll update the main script when I am sure that I didn't make a mistake in the above.

Made my own adjustments to this:

1. accepts -o/--output [filename] for where to write the file
2. keeps the --from-history dependencies which are more platform agnostic than the default dependencies. The end results is the from-history dependecies + the pip installs

Thanks for sharing this useful gist here. I found it last year and adapted for a specific use case. I just realize I never published the code, so here it is:

https://github.com/andresberejnoi/Conda-Tools

The main problems with conda env export that I was trying to solve for my specific case were these:

• Not including pip installations when using flag --from-history
• When using flag --from-history, package versions are often not
  included (depending on whether the user manually typed them)
• When using --from-history flag, third-party channels are not included
  in the file.