Skip to content

Instantly share code, notes, and snippets.

@msarahan

msarahan/conda-build-3-demo.ipynb

Last active May 17, 2017 15:41
Show Gist options
  • Save msarahan/7c7bc8e8ba2b46306cfb011365d21b59 to your computer and use it in GitHub Desktop.
Save msarahan/7c7bc8e8ba2b46306cfb011365d21b59 to your computer and use it in GitHub Desktop.
Download ZIP
Conda-build 3 demo notebook
Display the source blob
Display the rendered blob
Raw
conda-build-3-demo.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conda Build 3 new features demo"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Handling version compatibility is one of the hardest challenges in building software. Up to now, conda-build provided helpful tools in terms of the ability to constrain or pin versions in recipes. The limiting thing about this capability was that it entailed editing a lot of recipes. Conda-build 3 introduces a new scheme for controlling version constraints, which enhances behavior in two ways. First, you can now set versions in an external file, and you can provide lists of versions for conda-build to loop over. Matrix builds are now much simpler and no longer require an external tool, such as conda-build-all. Second, there have been several new Jinja2 functions added, which allow recipe authors to express their constraints relative to the versions of packages installed at build time. This dynamic expression greatly cuts down on the need for editing recipes.\n",
"\n",
"Each of these developments have enabled interesting new capabilities for cross-compiling, as well as improving package compatibility by adding more intelligent constraints."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This document is intended as a quick overview of new features in conda-build 3. For more information, see the docs PR at https://github.com/conda/conda-docs/pull/414\n",
"\n",
"These demos use conda-build's python API to render and build recipes. That API currently does not have a docs page, but is pretty self explanatory. See the source at https://github.com/conda/conda-build/blob/master/conda_build/api.py"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from conda_build import api\n",
"import os\n",
"from pprint import pprint"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, set up some helper functions that will output recipe contents in a nice-to-read way:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def print_yamls(recipe, **kwargs):\n",
" yamls = [api.output_yaml(m[0]) \n",
" for m in api.render(recipe, verbose=False, permit_unsatisfiable_variants=True, **kwargs)]\n",
" for yaml in yamls:\n",
" print(yaml)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def print_outputs(recipe, **kwargs):\n",
" pprint(api.get_output_file_paths(recipe, verbose=False, **kwargs))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Most of the new functionality revolves around much more powerful use of jinja2 templates. The core idea is that there is now a separate configuration file that can be used to insert many different entries into your meta.yaml files."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: abc\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - something {{ something }}\r\n",
" run:\r\n",
" - something {{ something }}\r\n"
]
}
],
"source": [
"!cat 01_basic_templating/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"something:\r\n",
" - 1.0\r\n",
" - 2.0\r\n"
]
}
],
"source": [
"# The configuration is hierarchical - it can draw from many config files. One place they can live is alongside meta.yaml:\n",
"!cat 01_basic_templating/conda_build_config.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we have one slot in meta.yaml, and two values for that one slot, we should end up with two output packages:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h78957b4_0.tar.bz2',\n",
" '/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h8be4e46_0.tar.bz2']\n"
]
}
],
"source": [
"print_outputs('01_basic_templating/')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - something 1.0\n",
" run:\n",
" - something 1.0\n",
"extra:\n",
" final: false\n",
"\n",
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - something 2.0\n",
" run:\n",
" - something 2.0\n",
"extra:\n",
" final: false\n",
"\n"
]
}
],
"source": [
"print_yamls('01_basic_templating/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, that's fun already. But wait, there's more!\n",
"\n",
"People are used to not needing to explicitly pin numpy. Conda-build 3 supports that, as well as explicit pinning."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: abc\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - python\r\n",
" run:\r\n",
" - python\r\n"
]
}
],
"source": [
"!cat 02_python_version/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"python:\r\n",
" - 2.7\r\n",
" - 3.5\r\n"
]
}
],
"source": [
"!cat 02_python_version/conda_build_config.yaml"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py27h53b4115_0\n",
"requirements:\n",
" build:\n",
" - pip 9.0.1 py27_1\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - openssl 1.0.2k 2\n",
" - setuptools 27.2.0 py27_0\n",
" - python 2.7.13 0\n",
" - zlib 1.2.8 3\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py27_0\n",
" run:\n",
" - python >=2.7,<2.8\n",
"extra:\n",
" final: true\n",
"\n",
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py35h6c99163_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - setuptools 27.2.0 py35_0\n",
" - openssl 1.0.2k 2\n",
" - zlib 1.2.8 3\n",
" - python 3.5.3 1\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py35_0\n",
" - pip 9.0.1 py35_1\n",
" run:\n",
" - python >=3.5,<3.6\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('02_python_version/')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py34hb8b720c_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - python 3.4.5 0\n",
" - pip 9.0.1 py34_1\n",
" - openssl 1.0.2k 2\n",
" - setuptools 27.2.0 py34_0\n",
" - zlib 1.2.8 3\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py34_0\n",
" run:\n",
" - python >=3.4,<3.5\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"# Setting environment variables overrides the conda_build_config.yaml. This preserves older, well-established behavior.\n",
"os.environ[\"CONDA_PY\"] = \"3.4\"\n",
"print_yamls('02_python_version/')\n",
"del os.environ['CONDA_PY']"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: abc\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py36ha934722_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - wheel 0.29.0 py36_0\n",
" - openssl 1.0.2k 2\n",
" - python 3.6.1 2\n",
" - zlib 1.2.8 3\n",
" - setuptools 27.2.0 py36_0\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - pip 9.0.1 py36_1\n",
" run:\n",
" - python >=3.6,<3.7\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"# passing python as an argument (CLI or to the API) also overrides conda_build_config.yaml\n",
"print_yamls('02_python_version/', python=\"3.6\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Wait a minute - what is that ``h7d013e7`` gobbledygook in the build/string field?\n",
"\n",
"Conda-build 3 aims to generalize pinning/constraints. Such constraints differentiate a package. For example, in the past, we have had things like py27np111 in filenames. This is the same idea, just generalized. Since we can't readily put every possible constraint into the filename, we have kept the old ones, but added the hash as a general solution.\n",
"\n",
"There's more information about what goes into a hash at https://github.com/conda/conda-docs/pull/414/files#diff-d1a5dbb2f223de4935d93990676c7074R919\n",
"\n",
"Let's take a look at how to inspect the hash contents of a built package."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Solving package specifications: .\n",
"Package plan for environment '/Users/msarahan/miniconda3/conda-bld/abc_1495035536701/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla':\n",
"\n",
"The following NEW packages will be INSTALLED:\n",
"\n",
" openssl: 1.0.2k-2 \n",
" pip: 9.0.1-py36_1 \n",
" python: 3.6.1-2 \n",
" readline: 6.2-2 \n",
" setuptools: 27.2.0-py36_0\n",
" sqlite: 3.13.0-0 \n",
" tk: 8.5.18-0 \n",
" wheel: 0.29.0-py36_0\n",
" xz: 5.2.2-1 \n",
" zlib: 1.2.8-3 \n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for abc\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"source tree in: /Users/msarahan/miniconda3/conda-bld/abc_1495035536701/work\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"BUILD START: ['abc-1.0-py36ha934722_0.tar.bz2']\n",
"Solving package specifications: .\n",
"Package plan for environment '/Users/msarahan/miniconda3/conda-bld/abc_1495035536701/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla':\n",
"\n",
"The following NEW packages will be INSTALLED:\n",
"\n",
" openssl: 1.0.2k-2 \n",
" pip: 9.0.1-py36_1 \n",
" python: 3.6.1-2 \n",
" readline: 6.2-2 \n",
" setuptools: 27.2.0-py36_0\n",
" sqlite: 3.13.0-0 \n",
" tk: 8.5.18-0 \n",
" wheel: 0.29.0-py36_0\n",
" xz: 5.2.2-1 \n",
" zlib: 1.2.8-3 \n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Packaging abc-1.0-py36ha934722_0\n",
"INFO conda_build.build:bundle_conda(663): Packaging abc-1.0-py36ha934722_0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"number of files: 0\n",
"Fixing permissions\n",
"Fixing permissions\n",
"updating: abc-1.0-py36ha934722_0.tar.bz2\n",
"Fetching package metadata ...\n",
"Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36ha934722_0.tar.bz2\n",
"# Automatic uploading is disabled\n",
"# If you want to upload package(s) to anaconda.org later, type:\n",
"\n",
"anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36ha934722_0.tar.bz2\n",
"\n",
"# To have conda build upload to anaconda.org automatically, use\n",
"# $ conda config --set anaconda_upload yes\n",
"\n",
"anaconda_upload is not set. Not uploading wheels: []\n",
"/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36ha934722_0.tar.bz2\n"
]
}
],
"source": [
"outputs = api.build('02_python_version/', python=\"3.6\", \n",
" anaconda_upload=False)\n",
"pkg_file = outputs[0]\n",
"print(pkg_file)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'abc-1.0-py36ha934722_0': {'files': ['info/recipe/conda_build_config.yaml'],\n",
" 'recipe': {'requirements': {'build': ['openssl '\n",
" '1.0.2k 2',\n",
" 'pip 9.0.1 '\n",
" 'py36_1',\n",
" 'python '\n",
" '3.6.1 2',\n",
" 'readline '\n",
" '6.2 2',\n",
" 'setuptools '\n",
" '27.2.0 '\n",
" 'py36_0',\n",
" 'sqlite '\n",
" '3.13.0 0',\n",
" 'tk 8.5.18 0',\n",
" 'wheel '\n",
" '0.29.0 '\n",
" 'py36_0',\n",
" 'xz 5.2.2 1',\n",
" 'zlib 1.2.8 '\n",
" '3'],\n",
" 'run': ['python '\n",
" '>=3.6,<3.7']}}}}\n"
]
}
],
"source": [
"# using command line here just to show you that this command exists.\n",
"!conda inspect hash-inputs ~/miniconda3/conda-bld/osx-64/abc-1.0-py36ha934722_0.tar.bz2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When numpy gets involved, things get more fun. The simplest use of numpy is just for its python interface - not using its C header. For such usage, we don't need to pin numpy in the output packages."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: numpy_matrix\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - python\r\n",
" - numpy\r\n",
" run:\r\n",
" - python\r\n",
" - numpy\r\n"
]
}
],
"source": [
"!cat 03_numpy_matrix/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"python:\r\n",
" - 2.7\r\n",
" - 3.5\r\n",
"numpy:\r\n",
" - 1.10\r\n",
" - 1.11\r\n",
"ignore_version:\r\n",
" - numpy\r\n"
]
}
],
"source": [
"!cat 03_numpy_matrix/conda_build_config.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, so I'm hoping that we get only two packages here - one for each Python version, but not one for each numpy version."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for numpy_matrix\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"['/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-py27hb175994_0.tar.bz2',\n",
" '/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-py35hbae5bf6_0.tar.bz2']\n"
]
}
],
"source": [
"pprint(api.get_output_file_paths('03_numpy_matrix/', verbose=False))"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py27hb175994_0\n",
"requirements:\n",
" build:\n",
" - pip 9.0.1 py27_1\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - openssl 1.0.2k 2\n",
" - setuptools 27.2.0 py27_0\n",
" - python 2.7.13 0\n",
" - zlib 1.2.8 3\n",
" - readline 6.2 2\n",
" - numpy\n",
" - wheel 0.29.0 py27_0\n",
" run:\n",
" - numpy\n",
" - python >=2.7,<2.8\n",
"extra:\n",
" final: true\n",
"\n",
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: py35hbae5bf6_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - setuptools 27.2.0 py35_0\n",
" - numpy\n",
" - openssl 1.0.2k 2\n",
" - zlib 1.2.8 3\n",
" - python 3.5.3 1\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py35_0\n",
" - pip 9.0.1 py35_1\n",
" run:\n",
" - python >=3.5,<3.6\n",
" - numpy\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('03_numpy_matrix/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What about when we do use numpy's C API, and we do need to pin it so that a particular version is available at runtime?"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: numpy_matrix\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - python {{ python }}\r\n",
" - numpy {{ numpy }}\r\n",
" run:\r\n",
" - python\r\n",
" - numpy\r\n"
]
}
],
"source": [
"!cat 04_numpy_matrix_pinned/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"python:\r\n",
" - 2.7\r\n",
" - 3.5\r\n",
"numpy:\r\n",
" - 1.10\r\n",
" - 1.11\r\n",
"\r\n",
"pin_run_as_build:\r\n",
" numpy:\r\n",
" min_pin: 'x.x.x'\r\n",
" max_pin: 'x.x'\r\n"
]
}
],
"source": [
"!cat 04_numpy_matrix_pinned/conda_build_config.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"pin_run_as_build is a special extra key in the config file. It is a generalization of the ``x.x`` concept that existed for numpy since 2015. There's more information at https://github.com/conda/conda-docs/pull/414/files#diff-d1a5dbb2f223de4935d93990676c7074R757\n",
"\n",
"Each x indicates another level of pinning in the output recipe."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-np110py27hd1eac1c_0.tar.bz2',\n",
" '/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-np110py35h947cb85_0.tar.bz2',\n",
" '/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-np111py27hb43cb11_0.tar.bz2',\n",
" '/Users/msarahan/miniconda3/conda-bld/osx-64/numpy_matrix-1.0-np111py35h9fe35bf_0.tar.bz2']\n"
]
}
],
"source": [
"print_outputs('04_numpy_matrix_pinned/')"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: np110py27hd1eac1c_0\n",
"requirements:\n",
" build:\n",
" - pip 9.0.1 py27_1\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - mkl 11.3.3 0\n",
" - openssl 1.0.2k 2\n",
" - numpy 1.10.4 py27_2\n",
" - setuptools 27.2.0 py27_0\n",
" - python 2.7.13 0\n",
" - zlib 1.2.8 3\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py27_0\n",
" run:\n",
" - numpy >=1.10.4,<1.11\n",
" - python >=2.7,<2.8\n",
"extra:\n",
" final: true\n",
"\n",
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: np111py27hb43cb11_0\n",
"requirements:\n",
" build:\n",
" - pip 9.0.1 py27_1\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - openssl 1.0.2k 2\n",
" - setuptools 27.2.0 py27_0\n",
" - mkl 2017.0.1 0\n",
" - python 2.7.13 0\n",
" - zlib 1.2.8 3\n",
" - numpy 1.11.3 py27_0\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py27_0\n",
" run:\n",
" - python >=2.7,<2.8\n",
" - numpy >=1.11.3,<1.12\n",
"extra:\n",
" final: true\n",
"\n",
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: np110py35h947cb85_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - numpy 1.10.4 py35_2\n",
" - setuptools 27.2.0 py35_0\n",
" - mkl 11.3.3 0\n",
" - openssl 1.0.2k 2\n",
" - zlib 1.2.8 3\n",
" - python 3.5.3 1\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py35_0\n",
" - pip 9.0.1 py35_1\n",
" run:\n",
" - python >=3.5,<3.6\n",
" - numpy >=1.10.4,<1.11\n",
"extra:\n",
" final: true\n",
"\n",
"package:\n",
" name: numpy_matrix\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: np111py35h9fe35bf_0\n",
"requirements:\n",
" build:\n",
" - sqlite 3.13.0 0\n",
" - tk 8.5.18 0\n",
" - setuptools 27.2.0 py35_0\n",
" - openssl 1.0.2k 2\n",
" - mkl 2017.0.1 0\n",
" - zlib 1.2.8 3\n",
" - python 3.5.3 1\n",
" - xz 5.2.2 1\n",
" - readline 6.2 2\n",
" - wheel 0.29.0 py35_0\n",
" - numpy 1.11.3 py35_0\n",
" - pip 9.0.1 py35_1\n",
" run:\n",
" - python >=3.5,<3.6\n",
" - numpy >=1.11.3,<1.12\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('04_numpy_matrix_pinned/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, you've now seen several examples of \"pinning expressions\" - things like x.x that apply pins more or less stringently. Let's take a look at how we can control the relationship of these constraints. Before now you could certainly accomplish pinning, it just took more work. Now you can define your pinning expressions, and then change your target versions only in one config file."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: compatible\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - libpng\r\n",
" run:\r\n",
" - {{ pin_compatible('libpng') }}\r\n"
]
}
],
"source": [
"!cat 05_compatible/meta.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is effectively saying \"add a runtime libpng constraint that follows conda-build's default behavior, relative to the version of libpng that was used at build time\"\n",
"\n",
"The default behavior is: exact version match lower bound (\"x.x.x.x.x.x.x\"), next major version upper bound (\"x\")"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for compatible\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: compatible\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: h3d53989_0\n",
"requirements:\n",
" build:\n",
" - zlib 1.2.8 3\n",
" - libpng 1.6.27 0\n",
" run:\n",
" - libpng >=1.6.27,<2\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('05_compatible/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These constraints are completely customizable with pinning expressions:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: compatible\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - libpng\r\n",
" run:\r\n",
" - {{ pin_compatible('libpng', max_pin='x.x') }}\r\n"
]
}
],
"source": [
"!cat 06_compatible_custom/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: compatible\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: hbb16060_0\n",
"requirements:\n",
" build:\n",
" - zlib 1.2.8 3\n",
" - libpng 1.6.27 0\n",
" run:\n",
" - libpng >=1.6.27,<1.7\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('06_compatible_custom/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, you can also manually specify version bounds. These supersede any relative constraints."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: compatible\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - libpng\r\n",
" run:\r\n",
" - {{ pin_compatible('libpng', max_pin='x.x', min_pin=None) }}\r\n"
]
}
],
"source": [
"!cat 07_compatible_custom_lower_upper/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: compatible\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: hf4d5716_0\n",
"requirements:\n",
" build:\n",
" - zlib 1.2.8 3\n",
" - libpng 1.6.27 0\n",
" run:\n",
" - libpng <1.7\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('07_compatible_custom_lower_upper/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Much of the development of conda-build 3 has been inspired by improving the compiler toolchain situation. Conda-build 3 adds special support for more dynamic specification of compilers."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: cross\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - {{ compiler('c') }}\r\n"
]
}
],
"source": [
"!cat 08_compiler/meta.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By replacing any actual compiler with this jinja2 function, we're free to swap in different compilers based on the contents of the conda_build_config.yaml file (or other variant configuration). Rather than saying \"I need gcc,\" we are saying \"I need a C compiler.\"\n",
"\n",
"By doing so, recipes are much more dynamic, and conda-build also helps to keep your recipes in line with respect to runtimes."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for cross\n",
"WARNING conda_build.metadata:finalize_outputs_pass(574): Could not finalize metadata due to missing dependencies: ['clang_osx-109-x86_64']\n",
"Returning non-final recipe; one or more dependencies was unsatisfiable.\n",
"WARNING conda_build.api:render(56): Returning non-final recipe; one or more dependencies was unsatisfiable.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: cross\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - clang_osx-109-x86_64\n",
" run: []\n",
"extra:\n",
" final: false\n",
"\n"
]
}
],
"source": [
"print_yamls('08_compiler/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Conda-build 3 is very good at allowing cross compiling. For example, we have the same placeholder compiler recipe:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: cross\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - {{ compiler('c') }}\r\n"
]
}
],
"source": [
"!cat 09_cross/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"c_compiler:\r\n",
" - gcc\r\n",
"target_platform:\r\n",
" - linux-64\r\n",
" - linux-cos5-64\r\n",
" - linux-aarch64\r\n"
]
}
],
"source": [
"# but by adding in a base compiler name, and target platforms, we can make a build matrix\n",
"# This is not magic, the compiler packages must already exist. Conda-build is only following a naming scheme.\n",
"!cat 09_cross/conda_build_config.yaml"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING conda_build.metadata:finalize_outputs_pass(574): Could not finalize metadata due to missing dependencies: ['gcc_linux-64']\n",
"WARNING conda_build.api:render(56): Returning non-final recipe; one or more dependencies was unsatisfiable.\n",
"WARNING conda_build.metadata:finalize_outputs_pass(574): Could not finalize metadata due to missing dependencies: ['gcc_linux-cos5-64']\n",
"WARNING conda_build.api:render(56): Returning non-final recipe; one or more dependencies was unsatisfiable.\n",
"WARNING conda_build.metadata:finalize_outputs_pass(574): Could not finalize metadata due to missing dependencies: ['gcc_linux-aarch64']\n",
"WARNING conda_build.api:render(56): Returning non-final recipe; one or more dependencies was unsatisfiable.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: cross\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - gcc_linux-64\n",
" run: []\n",
"extra:\n",
" final: false\n",
"\n",
"package:\n",
" name: cross\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - gcc_linux-cos5-64\n",
" run: []\n",
"extra:\n",
" final: false\n",
"\n",
"package:\n",
" name: cross\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
"requirements:\n",
" build:\n",
" - gcc_linux-aarch64\n",
" run: []\n",
"extra:\n",
" final: false\n",
"\n"
]
}
],
"source": [
"print_yamls('09_cross/')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, it is frequently a problem to remember to add runtime dependencies. Sometimes the recipe author is not entirely familiar with the lower level code, and has no idea about runtime dependencies. Other times, it's just a pain to keep versions of runtime dependencies in line. Conda-build 3 introduces a way of storing the required runtime dependencies *on the package providing the dependency at build time.*\n",
"\n",
"For example, using g++ in a non-static configuration will require that the end-user have a sufficiently new libstdc++ runtime library available at runtime. Many people don't currently include this in their recipes. Sometimes the system libstdc++ is adequate, but often not. By imposing the downstream dependency, we can make sure that people don't forget the runtime dependency."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: package_has_run_exports\r\n",
" version: 1.0\r\n",
"\r\n",
"build:\r\n",
" run_exports:\r\n",
" - bzip2 {{ pin_compatible('bzip2') }}\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - bzip2\r\n"
]
}
],
"source": [
"# First, a package that provides some library. \n",
"# When anyone uses this library, they need to include the appropriate runtime.\n",
"!cat 10_runtimes/uses_run_exports/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\r\n",
" name: package_consuming_run_exports\r\n",
" version: 1.0\r\n",
"\r\n",
"requirements:\r\n",
" build:\r\n",
" - package_has_run_exports\r\n"
]
}
],
"source": [
"# This is the simple downstream package that uses the library provided in the previous recipe.\n",
"!cat 10_runtimes/consumes_exports/meta.yaml"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Solving package specifications: .\n",
"Package plan for environment '/Users/msarahan/miniconda3/conda-bld/package_has_run_exports_1495035602303/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla':\n",
"\n",
"The following NEW packages will be INSTALLED:\n",
"\n",
" bzip2: 1.0.6-3\n",
"\n",
"source tree in: /Users/msarahan/miniconda3/conda-bld/package_has_run_exports_1495035602303/work\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for package_has_run_exports\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"Solving package specifications: .\n",
"BUILD START: ['package_has_run_exports-1.0-h3715afa_0.tar.bz2']\n",
"Solving package specifications: .\n",
"Package plan for environment '/Users/msarahan/miniconda3/conda-bld/package_has_run_exports_1495035602303/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla':\n",
"\n",
"The following NEW packages will be INSTALLED:\n",
"\n",
" bzip2: 1.0.6-3\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.build:bundle_conda(663): Packaging package_has_run_exports-1.0-h3715afa_0\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"number of files: 0\n",
"Fixing permissions\n",
"Fixing permissions\n",
"updating: package_has_run_exports-1.0-h3715afa_0.tar.bz2\n",
"Fetching package metadata ...\n",
"Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-h3715afa_0.tar.bz2\n",
"# Automatic uploading is disabled\n",
"# If you want to upload package(s) to anaconda.org later, type:\n",
"\n",
"anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-h3715afa_0.tar.bz2\n",
"\n",
"# To have conda build upload to anaconda.org automatically, use\n",
"# $ conda config --set anaconda_upload yes\n",
"\n",
"anaconda_upload is not set. Not uploading wheels: []\n"
]
},
{
"data": {
"text/plain": [
"['/Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-h3715afa_0.tar.bz2']"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# let's build the former package first.\n",
"api.build('10_runtimes/uses_run_exports', anaconda_upload=False)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO conda_build.metadata:finalize_outputs_pass(548): Attempting to finalize metadata for package_consuming_run_exports\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"package:\n",
" name: package_consuming_run_exports\n",
" version: '1.0'\n",
"build:\n",
" noarch: ''\n",
" noarch_python: false\n",
" string: hddc2032_0\n",
"requirements:\n",
" build:\n",
" - package_has_run_exports 1.0 h3715afa_0\n",
" run:\n",
" - bzip2 bzip2 >=1.0.6,<2\n",
"extra:\n",
" final: true\n",
"\n"
]
}
],
"source": [
"print_yamls('10_runtimes/consumes_exports')"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"In the above recipe, note that bzip2 has been added as a runtime dependency, and is pinned according to conda-build's default pin_compatible scheme."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment