Skip to content

Instantly share code, notes, and snippets.

@usrbinkat
Created April 14, 2026 18:10
Show Gist options
  • Select an option

  • Save usrbinkat/71fe23cf52ba0eeb74809ef97e94f477 to your computer and use it in GitHub Desktop.

Select an option

Save usrbinkat/71fe23cf52ba0eeb74809ef97e94f477 to your computer and use it in GitHub Desktop.
Pulumi + Python + Nix

Pulumi + Python + Nix: Complete Implementation Specification

Purpose: Exhaustive implementor-level reference for building, migrating, and maintaining Pulumi Python infrastructure projects using Nix for hermetic dependency management. Replaces pip, uv, virtualenv, and pyproject.toml-based dependency resolution with a single python.withPackages derivation that is reproducible, offline-capable, and free of PATH collisions.

Audience: Any developer or AI agent with zero prior Nix, pip, uv, or Pulumi knowledge. Every concept is defined from first principles. Every code snippet is complete and copy-paste viable.

Scope: Greenfield implementation and migration from existing pip/uv projects.


Table of Contents

  1. Concepts and Terminology
  2. Why This Migration Exists
  3. Architecture Overview
  4. File Layout
  5. Step-by-Step Implementation
  6. The mkPulumiPypiPackage Pattern
  7. Finding PyPI Wheel URLs and Hashes
  8. The PATH Collision Problem
  9. The Two-Pass Import Pattern
  10. The PULUMI_PYTHON_CMD Wrapper
  11. Pyright Integration
  12. Environment Variables Reference
  13. Migration Checklist: pip/uv → Nix
  14. Debugging
  15. Common Failure Modes
  16. Native (Non-Pure) Python Packages
  17. Adding a New Provider SDK
  18. Adding a New Runtime Dependency
  19. Updating Package Versions
  20. Offline and Air-Gapped Builds
  21. CI/CD Integration
  22. Complete Reference Implementation

1. Concepts and Terminology

Python Packaging

Term Definition
pip Python's default package installer. Downloads packages from PyPI, installs into a directory.
uv A fast pip replacement written in Rust. Drop-in compatible with pip but 10-100x faster.
virtualenv / venv An isolated Python environment with its own site-packages. Created by python -m venv .venv.
pyproject.toml Modern Python project metadata file. Declares dependencies, build system, tool configs.
wheel A pre-built Python package distribution format (.whl file). No compilation needed at install time.
sdist A source distribution. Requires compilation at install time (may need C compiler, headers).
pure Python wheel A wheel with tag py3-none-any — runs on any Python 3, any OS, any architecture. No native code.
native wheel A wheel with tag like cp313-cp313-manylinux_2_17_x86_64 — compiled for specific Python version, OS, and CPU.
site-packages Directory where Python finds installed packages. import foo searches here.
PyPI Python Package Index (pypi.org). The public repository of Python packages.

Nix Packaging

Term Definition
Nix A purely functional package manager. Every package is a function of its inputs, producing a unique store path.
nixpkgs The Nix package repository. Contains 100,000+ packages including Python packages under python3Packages.*.
derivation A build recipe in Nix. Specifies inputs, build commands, and outputs. Produces a /nix/store/<hash>-<name> path.
flake A Nix project with a flake.nix file declaring inputs (dependencies) and outputs (packages, shells, modules).
devshell A development environment created by nix develop. Provides tools and environment variables without installation.
overlay A function that modifies or extends the nixpkgs package set. Applied during import nixpkgs { overlays = [...]; }.
mkShell Nix function for creating development shells. Accepts nativeBuildInputs (tools) and buildInputs (libraries).
python.withPackages Nix function that creates a Python environment with specific packages available for import. Returns a single derivation.
buildPythonPackage Nix function that builds a Python package from source or wheel. Used for packages not in nixpkgs.
fetchurl Nix function that downloads a file from a URL and verifies its hash. Used to fetch wheels from PyPI.
symlinkJoin Nix function that merges multiple derivations into one by symlinking their contents. Used to combine pulumi CLI components.
writeShellScriptBin Nix function that creates a shell script in /nix/store as an executable. Used for wrapper scripts.
nativeBuildInputs Packages available at build time and in the devshell PATH. This is where you put tools.
propagatedBuildInputs Dependencies that are automatically available to downstream consumers. Used for Python package deps.
shellHook Shell code executed when entering a nix develop shell. Used for PATH manipulation and env setup.
env Attribute set of environment variables exported by mkShell. Set once, available in shell.

Pulumi

Term Definition
Pulumi Infrastructure as Code tool. Programs in Python/Go/TypeScript/etc. define cloud resources declaratively.
Pulumi CLI The pulumi command-line binary (written in Go). Orchestrates deployments.
Language plugin Pulumi component that interprets your program. pulumi-language-python runs Python programs.
Provider SDK Python package that defines resources for a specific cloud/service (e.g., pulumi-kubernetes).
Stack A Pulumi deployment instance with its own state and configuration (e.g., dev, staging, prod).
PULUMI_PYTHON_CMD Environment variable telling Pulumi which Python interpreter to use. Overrides auto-detection.
Pulumi.yaml Project configuration file. Declares runtime (python), main module path, and optional virtualenv path.

2. Why This Migration Exists

Problems with pip/uv in Nix environments

  1. Non-hermetic: pip fetches packages from PyPI at runtime. Different machines get different versions. Network failures break builds.

  2. PATH collision: Nix devshells and pip virtualenvs both provide python3 binaries. mkShell prepends all package bin/ directories to PATH including transitive dependencies. When python.withPackages is used alongside a venv, two python3 binaries compete. The wrong one wins, and import pulumi fails.

  3. Pyright resolution failure: Pulumi's Go-based language plugin runs python -m pyright to typecheck. If pyright is only a standalone Node binary on PATH (from nixpkgs), Pulumi can't find it. It must be pip-visible (installed in the Python environment as a package).

  4. Duplicate Python interpreters: python.withPackages creates a wrapper script (python3) that knows about installed packages AND includes the bare python3 from the underlying derivation as a build input. mkShell puts both in PATH. The bare one has no packages.

  5. PULUMI_PYTHON_CMD mismatch: Pulumi auto-detects Python by searching PATH. If it finds the bare interpreter instead of the withPackages wrapper, pulumi up fails with ModuleNotFoundError: No module named 'pulumi'.

  6. Virtualenv path doubling: Pulumi resolves virtualenv paths differently during pulumi install (relative to Pulumi.yaml directory) vs pulumi up (relative to main: directory). This causes the path to be concatenated twice, creating non-existent directories.

What Nix-native provides

  • Hermetic: All packages are content-addressed in /nix/store. Identical inputs always produce identical outputs.
  • Offline-capable: After first build, all packages are cached locally. No network needed.
  • Single interpreter: One python.withPackages call, one python3 binary, one site-packages.
  • No venv: No .venv/ directory, no UV_PROJECT_ENVIRONMENT, no PULUMI_PYTHON_VIRTUALENV.
  • Reproducible: flake.lock pins every input. Same lock file = same environment on every machine.

3. Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                        Dependency Resolution Flow                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  pyproject.toml (OLD — reference only, not used by Nix)                    │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │ [project.dependencies]                                          │        │
│  │   pulumi >= 3.0                                                 │        │
│  │   pulumi-kubernetes >= 4.0                                      │        │
│  │   pydantic >= 2.0                                               │        │
│  │   ...                                                           │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                          │                                                  │
│                    Audit + Classify                                         │
│                          │                                                  │
│                          ▼                                                  │
│  pulumi.nix (NEW — Nix-native dependency specification)                    │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │ pythonDeps = ps: [                                              │        │
│  │   ps.pulumi              # from nixpkgs                        │        │
│  │   pulumiKubernetes       # from PyPI wheel (not in nixpkgs)    │        │
│  │   ps.pydantic            # from nixpkgs                        │        │
│  │   pyrightPkg             # from PyPI wheel (Pulumi requirement)│        │
│  │ ];                                                              │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                          │                                                  │
│                    Merge into single env                                    │
│                          │                                                  │
│                          ▼                                                  │
│  languages.nix (Aggregation layer)                                         │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │ pythonEnv = python313.withPackages (ps: [                       │        │
│  │   ps.ipython                    # dev tools                    │        │
│  │   ps.ruff                       # linter                      │        │
│  │ ] ++ pulumiDeps.pythonDeps ps); # <-- Pulumi deps merged here │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                          │                                                  │
│                    Single derivation                                       │
│                          │                                                  │
│                          ▼                                                  │
│  /nix/store/<hash>-python3-3.13.12-env/bin/python3                         │
│  ┌─────────────────────────────────────────────────────────────────┐        │
│  │ This ONE binary can import:                                     │        │
│  │   pulumi, pulumi_kubernetes, pulumi_tls, pydantic, pyright,    │        │
│  │   ipython, ruff, bcrypt, semver, gitpython, kubernetes, ...    │        │
│  └─────────────────────────────────────────────────────────────────┘        │
│                          │                                                  │
│                    Used by:                                                │
│                          │                                                  │
│             ┌────────────┼────────────┐                                    │
│             ▼            ▼            ▼                                    │
│         devshell     pulumi CLI    pyright                                 │
│         PATH         wrapper       typechecker                            │
│                      (PULUMI_      (python -m                             │
│                      PYTHON_CMD)   pyright)                               │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

4. File Layout

your-project/
├── flake.nix                          # Flake entry point
├── flake.lock                         # Pinned input versions
├── src/
│   ├── packages/
│   │   ├── pulumi.nix                 # Pulumi Python deps + CLI wrapper
│   │   ├── languages.nix             # Aggregates all language packages
│   │   └── default.nix               # Package set entry point
│   └── devshells/
│       ├── base.nix                   # Base devshell
│       ├── full.nix                   # All languages
│       └── default.nix               # Shell exports
├── infrastructure/
│   └── pulumi/
│       ├── Pulumi.yaml                # Pulumi project config
│       ├── src/
│       │   ├── __main__.py            # Entry point
│       │   └── ...                    # Your IaC code
│       └── stacks/
│           ├── Pulumi.dev.yaml
│           └── Pulumi.prod.yaml
├── pyproject.toml                     # LEGACY — kept for reference only
└── .envrc                             # direnv configuration

Key principle: pyproject.toml becomes a reference document only. Nix is the source of truth for all Python dependencies. You may keep pyproject.toml for editor tooling (pyright config, ruff config) but its [project.dependencies] section is not used.


5. Step-by-Step Implementation

5.1 Auditing Existing Dependencies

Start by extracting every Python dependency from your current project:

# From pyproject.toml
grep -A 100 '^\[project\]' pyproject.toml | grep -A 100 '^dependencies' | head -50

# From requirements.txt (if using pip)
cat requirements.txt

# From uv.lock (if using uv)
uv pip list --format=columns

# From an active venv
pip list --format=columns

# From Pulumi.yaml (virtualenv path)
grep virtualenv Pulumi.yaml

Create a list of every package with its version:

pulumi==3.192.0
pulumi-kubernetes==4.23.0
pulumi-tls==5.2.1
pulumi-docker-build==0.0.14
pulumi-cloudflare==6.14.0
pulumi-random==4.16.8
pydantic==2.11.0
bcrypt==4.3.0
semver==3.1.0
gitpython==3.1.44
kubernetes==33.1.0
pyyaml==6.0.2
pyright==1.1.407

5.2 Classifying Packages

For each package, determine where it comes from:

# Check if package exists in nixpkgs
nix eval nixpkgs#python313Packages.pulumi.version 2>/dev/null
nix eval nixpkgs#python313Packages.pydantic.version 2>/dev/null
nix eval nixpkgs#python313Packages.pulumi-kubernetes.version 2>/dev/null

Classify into three categories:

Category Example How to provide
In nixpkgs pulumi, pydantic, pyyaml, bcrypt ps.pulumi in pythonDeps
Not in nixpkgs, pure Python wheel pulumi-kubernetes, pulumi-tls mkPulumiPypiPackage
Not in nixpkgs, native wheel (rare for Pulumi deps) buildPythonPackage with autoPatchelfHook

5.3 Building pulumi.nix

This is the core file. It declares all Pulumi-related Python dependencies and the CLI wrapper.

# src/packages/pulumi.nix
#
# Pulumi Python Environment (NixOS-native with python.withPackages)
#
# This file provides:
#   1. pythonDeps — function returning Python packages for withPackages
#   2. package — Pulumi CLI with PULUMI_PYTHON_CMD wrapper
#   3. env — environment variables for devshells
#
# Architecture:
#   - pythonDeps is a FUNCTION that takes the python package set (ps)
#     and returns a list of packages. This is merged into the single
#     python.withPackages call in languages.nix.
#   - package is a symlinkJoin of the Pulumi CLI, its wrapper, and
#     the language plugin. It requires pythonEnv to be provided.
#   - The two-pass import pattern (see section 9) breaks the circular
#     dependency between pythonDeps and pythonEnv.
#
# Adding a new provider:
#   1. Check nixpkgs: nix eval nixpkgs#python313Packages.pulumi-foo
#   2. If found: add ps."pulumi-foo" to pythonDeps
#   3. If not found: add mkPulumiPypiPackage entry (see section 6)
#   4. Run: nix develop (rebuilds the environment)

{ pkgs, pythonEnv ? null }:

let
  # =========================================================================
  # PyPI Wheel Builder
  # =========================================================================
  # All Pulumi provider SDKs are pure Python wheels (py3-none-any).
  # They share identical propagatedBuildInputs: parver, pulumi, semver.
  # This helper avoids repeating the same buildPythonPackage boilerplate.
  #
  # For native (non-pure) wheels, see section 16.
  mkPulumiPypiPackage = { pname, version, url, hash, meta ? {} }:
    pkgs.python313Packages.buildPythonPackage {
      inherit pname version;
      format = "wheel";

      src = pkgs.fetchurl {
        inherit url hash;
      };

      # All Pulumi provider SDKs depend on these at runtime.
      # parver: PEP 440 version parsing
      # pulumi: Core SDK (resource registration, stack references)
      # semver: Semantic versioning
      propagatedBuildInputs = with pkgs.python313Packages; [
        parver
        pulumi
        semver
      ];

      # Provider SDK tests require a running Pulumi engine.
      # They cannot run in the Nix sandbox.
      doCheck = false;

      # Verify the package is importable after installation.
      # This catches broken wheels and missing dependencies.
      pythonImportsCheck = [ pname ];

      inherit meta;
    };

  # =========================================================================
  # Provider SDKs from PyPI (not available in nixpkgs)
  # =========================================================================
  # To find wheel URLs: https://pypi.org/project/PACKAGE_NAME/VERSION/#files
  # Select the "py3-none-any.whl" file. Copy the URL.
  #
  # To get the hash:
  #   nix-prefetch-url --type sha256 "URL"
  #   Then convert: nix hash convert --hash-algo sha256 --to sri <hash>
  #
  # Or use: nix store prefetch-file --hash-type sha256 "URL"

  pulumiKubernetes = mkPulumiPypiPackage {
    pname = "pulumi_kubernetes";
    version = "4.23.0";
    url = "https://files.pythonhosted.org/packages/cc/00/983975f1bcf02601f12b4afb60a4dfdf9f81ab11cbf2493aa349781684ae/pulumi_kubernetes-4.23.0-py3-none-any.whl";
    hash = "sha256-REPLACE_WITH_ACTUAL_HASH";
    meta.description = "Pulumi Kubernetes provider SDK";
  };

  pulumiTls = mkPulumiPypiPackage {
    pname = "pulumi_tls";
    version = "5.2.1";
    url = "https://files.pythonhosted.org/packages/3c/12/f5035bbfe624279a4838ec824ec8a6d34e9351f6780def8c2ab0ce965a0c/pulumi_tls-5.2.1-py3-none-any.whl";
    hash = "sha256-REPLACE_WITH_ACTUAL_HASH";
    meta.description = "Pulumi TLS provider SDK";
  };

  pulumiDockerBuild = mkPulumiPypiPackage {
    pname = "pulumi_docker_build";
    version = "0.0.14";
    url = "https://files.pythonhosted.org/packages/72/a0/7f8b89aed8ef77aa16dde4d3bff9254035a77e597d21c450f8c32d3db6c8/pulumi_docker_build-0.0.14-py3-none-any.whl";
    hash = "sha256-REPLACE_WITH_ACTUAL_HASH";
    meta.description = "Pulumi Docker Build provider SDK";
  };

  pulumiCloudflare = mkPulumiPypiPackage {
    pname = "pulumi_cloudflare";
    version = "6.14.0";
    url = "https://files.pythonhosted.org/packages/68/97/e6f11adbd919ad41ab920b137b7a711e3eb1e8130602a4c29d0ae86d329f/pulumi_cloudflare-6.14.0-py3-none-any.whl";
    hash = "sha256-REPLACE_WITH_ACTUAL_HASH";
    meta.description = "Pulumi Cloudflare provider SDK";
  };

  # =========================================================================
  # Pyright (Type Checker)
  # =========================================================================
  # Pulumi's language plugin requires pyright to be pip-visible (installed
  # in the Python environment as a package), not just on PATH as a standalone
  # CLI binary.
  #
  # The PyPI pyright package is a thin Python wrapper that invokes the
  # node-based pyright binary. It requires nodeenv and typing-extensions.
  #
  # nixpkgs provides pyright as a standalone Node binary. That works for
  # editors (VSCode, Neovim) but NOT for Pulumi's internal typechecker
  # invocation which does: python -m pyright
  pyrightPkg = pkgs.python313Packages.buildPythonPackage {
    pname = "pyright";
    version = "1.1.407";
    format = "wheel";

    src = pkgs.fetchurl {
      url = "https://files.pythonhosted.org/packages/dc/93/b69052907d032b00c40cb656d21438ec00b3a471733de137a3f65a49a0a0/pyright-1.1.407-py3-none-any.whl";
      hash = "sha256-REPLACE_WITH_ACTUAL_HASH";
    };

    propagatedBuildInputs = with pkgs.python313Packages; [
      nodeenv
      typing-extensions
    ];

    meta = {
      description = "Python command line wrapper for pyright type checker";
      homepage = "https://github.com/microsoft/pyright";
    };
  };

in
{
  # =========================================================================
  # pythonDeps: Function returning Python packages for withPackages
  # =========================================================================
  # This is NOT a derivation — it's a function that takes the python
  # package set (ps) and returns a list. It's called inside
  # python.withPackages in languages.nix.
  #
  # Why a function? Because the python package set (ps) is provided by
  # python.withPackages, not by us. We can't reference ps.pulumi
  # outside of a withPackages call.

  pythonDeps = ps: [
    # ─── Pulumi Core SDK ──────────────────────────────────────────
    # From nixpkgs (check version: nix eval nixpkgs#python313Packages.pulumi.version)
    ps.pulumi

    # ─── Provider SDKs from nixpkgs ───────────────────────────────
    ps."pulumi-random"

    # ─── Provider SDKs from PyPI wheels ───────────────────────────
    # These are NOT in nixpkgs python313Packages.
    # Built above using mkPulumiPypiPackage.
    pulumiKubernetes
    pulumiTls
    pulumiDockerBuild
    pulumiCloudflare

    # ─── Runtime Dependencies ─────────────────────────────────────
    # From pyproject.toml [project.dependencies] or requirements.txt.
    # Each must be verified: nix eval nixpkgs#python313Packages.PACKAGE.version
    ps.pydantic
    ps.bcrypt
    ps.semver
    ps.gitpython
    ps.kubernetes
    ps.pyyaml

    # ─── Type Checker ─────────────────────────────────────────────
    # Must be pip-visible for Pulumi's typechecker option.
    pyrightPkg
  ];

  # Re-export for languages.nix
  inherit pythonDeps;

  # =========================================================================
  # package: Pulumi CLI with Python wrapper
  # =========================================================================
  # This is only usable after pythonEnv is provided (second-pass import).
  # See section 9 for the two-pass pattern.
  #
  # The wrapper sets PULUMI_PYTHON_CMD to point at the withPackages
  # interpreter, ensuring Pulumi uses the correct Python with all
  # provider SDKs available.

  package = let
    # Assert: pythonEnv must be provided. This fails at eval time if
    # pulumi.nix is imported without pythonEnv (first-pass import
    # should not access .package).
    resolvedPythonEnv = assert pythonEnv != null; pythonEnv;

    # Shell script wrapper that sets PULUMI_PYTHON_CMD before exec'ing
    # the real pulumi binary. This ensures the Go-based language plugin
    # spawns the correct Python interpreter.
    pulumiWrapper = pkgs.writeShellScriptBin "pulumi" ''
      export PULUMI_PYTHON_CMD="${resolvedPythonEnv}/bin/python"
      exec ${pkgs.pulumi}/bin/pulumi "$@"
    '';

  in pkgs.symlinkJoin {
    name = "pulumi-with-python";
    paths = [
      pulumiWrapper                          # Wrapped CLI (takes priority)
      pkgs.pulumi                            # Original CLI (for other binaries like pulumi-watch)
      pkgs.pulumictl                         # Pulumi utilities
      pkgs.pulumiPackages.pulumi-python      # Python language plugin
      # NOTE: pythonEnv is NOT in paths — it's in the devshell's
      # nativeBuildInputs and prepended to PATH via shellHook.
    ];
    meta = {
      description = "Pulumi IaC with NixOS-native Python environment";
      homepage = "https://pulumi.com";
    };
  };

  # =========================================================================
  # env: Environment variables for devshells
  # =========================================================================
  # These are merged into the devshell's env attribute set.
  env = let
    resolvedPythonEnv = assert pythonEnv != null; pythonEnv;
  in {
    # Tell Pulumi which Python to use. Without this, Pulumi searches PATH
    # and may find the wrong python3 (bare interpreter without packages).
    PULUMI_PYTHON_CMD = "${resolvedPythonEnv}/bin/python";
  };
}

5.4 Building languages.nix

This file creates the single Python environment shared by all consumers.

# src/packages/languages.nix
#
# Single source of truth for all language environments.
# The pythonEnv derivation created here is the ONLY Python interpreter
# in the devshell. All other consumers (Pulumi, pyright, devtools)
# use this same derivation.

{ pkgs, ... }:

let
  # =========================================================================
  # Language version definitions (from your versions.nix or inline)
  # =========================================================================
  langs = {
    python = { version = "313"; display = "3.13"; };
    # go = { version = "1.25"; display = "1.25"; };
    # node = { version = "22"; display = "22"; };
    # rust = { version = "1.92.0"; display = "1.92.0"; };
  };

  # =========================================================================
  # Two-Pass Import for Pulumi
  # =========================================================================
  # First pass: import pulumi.nix WITHOUT pythonEnv.
  # This gives us access to pythonDeps (a function) but NOT package/env
  # (which require pythonEnv).
  pulumiDeps = import ./pulumi.nix { inherit pkgs; };

  # =========================================================================
  # Python Environment (SINGLE derivation)
  # =========================================================================
  # ALL Python packages go here. Do NOT create multiple withPackages calls.
  # Multiple calls = multiple python3 binaries in PATH = broken imports.
  pythonEnv = pkgs."python${langs.python.version}".withPackages (ps: [
    # ─── Development Tools ────────────────────────────────────────
    ps.ipython          # Enhanced REPL
    ps.ruff             # Linter + formatter

    # ─── Pulumi Dependencies ──────────────────────────────────────
    # Merged from pulumi.nix via the pythonDeps function.
    # This includes: core SDK, provider SDKs, runtime deps, pyright.
  ] ++ pulumiDeps.pythonDeps ps);

  # =========================================================================
  # Second pass: import pulumi.nix WITH pythonEnv
  # =========================================================================
  # Now pulumi.nix can create the CLI wrapper and env variables
  # that reference the actual Python interpreter path.
  pulumiPkg = import ./pulumi.nix { inherit pkgs pythonEnv; };

  # =========================================================================
  # Package lists for devshells
  # =========================================================================
  pythonPackages = [
    pythonEnv
    # Do NOT add other Python packages here. They go in withPackages above.
  ];

in
{
  # Exported for devshells
  inherit pythonEnv pythonPackages;
  inherit pulumiPkg;

  # All language packages combined
  packages = pythonPackages;
  # ++ goPackages ++ nodePackages ++ rustPackages;  # Add as needed

  # Environment variables
  env = {
    # Tell uv/pip to use the Nix-provided Python, not download their own.
    UV_SYSTEM_PYTHON = "1";
    # Don't write .pyc files (clutter in Nix store paths).
    PYTHONDONTWRITEBYTECODE = "1";
  } // pulumiPkg.env;
}

5.5 Building the Devshell

# src/devshells/full.nix
#
# Full development shell with all languages and tools.

{ pkgs, languages, ... }:

pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    # Language environments
  ] ++ languages.pythonPackages
    ++ [ languages.pulumiPkg.package ];  # Pulumi CLI with wrapper

  # =========================================================================
  # CRITICAL: PATH ordering fix
  # =========================================================================
  # mkShell prepends each package's bin/ to PATH INCLUDING transitive deps.
  # python.withPackages adds both:
  #   1. /nix/store/<hash>-python3-3.13.12-env/bin/python3  (wrapper with packages)
  #   2. /nix/store/<hash>-python3-3.13.12/bin/python3      (bare, NO packages)
  #
  # The bare one can win PATH priority. When it does:
  #   - `python3 -c "import pulumi"` → ModuleNotFoundError
  #   - `pulumi up` → "No module named 'pulumi'"
  #   - `python -m pyright` → "No module named 'pyright'"
  #
  # Fix: prepend pythonEnv FIRST in shellHook. This ensures the wrapper
  # (with all packages) always wins.
  shellHook = ''
    export PATH="${languages.pythonEnv}/bin:$PATH"
  '';

  env = languages.env;
}

5.6 Integrating with flake.nix

# flake.nix (relevant sections only)
{
  outputs = { nixpkgs, ... }:
    let
      pkgs = import nixpkgs {
        system = "x86_64-linux";
        config.allowUnfree = true;
      };

      languages = import ./src/packages/languages.nix { inherit pkgs; };
    in {
      devShells.x86_64-linux.default = import ./src/devshells/full.nix {
        inherit pkgs languages;
      };
    };
}

6. The mkPulumiPypiPackage Pattern

When to use

Use mkPulumiPypiPackage when a Pulumi provider SDK is:

  • NOT available in nixpkgs (nix eval nixpkgs#python313Packages.PACKAGE fails)
  • A pure Python wheel (py3-none-any.whl on PyPI)
  • Has standard Pulumi provider dependencies (parver, pulumi, semver)

How it works

mkPulumiPypiPackage = { pname, version, url, hash, meta ? {} }:
  pkgs.python313Packages.buildPythonPackage {
    inherit pname version;

    # "wheel" format tells Nix to install the .whl directly
    # without running setup.py or building from source.
    format = "wheel";

    # fetchurl downloads the file and verifies the hash.
    # If the hash doesn't match, the build fails immediately.
    src = pkgs.fetchurl {
      inherit url hash;
    };

    # These are available at runtime (import time) for the package.
    # All Pulumi providers need these three.
    propagatedBuildInputs = with pkgs.python313Packages; [
      parver    # PEP 440 version parsing
      pulumi    # Core SDK
      semver    # Semantic versioning
    ];

    # Skip tests — they require a running Pulumi engine.
    doCheck = false;

    # Verify the import works after installation.
    # pname = "pulumi_kubernetes" → import pulumi_kubernetes
    pythonImportsCheck = [ pname ];

    inherit meta;
  };

What each field does

Field Type Purpose
pname string Python package name as it appears in import (underscores, e.g., pulumi_kubernetes)
version string Exact version (e.g., "4.23.0")
url string Full URL to the .whl file on PyPI
hash string SRI hash (sha256-...) of the .whl file
meta.description string Human-readable description (optional)
format "wheel" Tells buildPythonPackage to install the wheel directly
propagatedBuildInputs list Runtime dependencies (available to importers)
doCheck bool Whether to run tests (false for providers)
pythonImportsCheck list Package names to try importing after install

7. Finding PyPI Wheel URLs and Hashes

Method 1: PyPI website

  1. Go to https://pypi.org/project/PACKAGE_NAME/VERSION/#files
  2. Find the file ending in py3-none-any.whl
  3. Copy the URL
  4. Download and hash:
# Download the wheel
curl -L -o /tmp/package.whl "URL"

# Get the Nix SRI hash
nix hash file --sri /tmp/package.whl
# Output: sha256-AbCdEf...

# Or if using older nix:
sha256sum /tmp/package.whl
# Then convert:
nix hash convert --hash-algo sha256 --to sri SHA256_HEX_HERE

Method 2: nix-prefetch-url

# This downloads, caches in nix store, and prints the hash
nix-prefetch-url "https://files.pythonhosted.org/packages/.../package-1.0.0-py3-none-any.whl"
# Output: 0abc123def456...  (base32)

# Convert to SRI format for use in fetchurl
nix hash convert --hash-algo sha256 --to sri 0abc123def456...
# Output: sha256-AbCdEf...

Method 3: nix store prefetch-file

nix store prefetch-file --hash-type sha256 \
  "https://files.pythonhosted.org/packages/.../package-1.0.0-py3-none-any.whl"
# Output includes the hash in SRI format

Method 4: Let Nix tell you (intentional failure)

# Set hash to empty string — Nix will fail and tell you the correct hash
src = pkgs.fetchurl {
  url = "https://files.pythonhosted.org/packages/.../package.whl";
  hash = "";  # Nix will error with: "got: sha256-CORRECT_HASH_HERE"
};
nix develop  # Will fail with the correct hash
# error: hash mismatch in fixed-output derivation
#   specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
#   got:       sha256-AbCdEfGhIjKlMnOpQrStUvWxYz0123456789ABCD=

Copy the got: hash into your hash field.


8. The PATH Collision Problem

What happens

When you put pythonEnv (from python.withPackages) in nativeBuildInputs:

mkShell {
  nativeBuildInputs = [ pythonEnv otherPackage ];
}

Nix constructs PATH by iterating all packages AND their transitive dependencies. pythonEnv depends on the base python3 interpreter (it wraps it). Both get added to PATH:

PATH=/nix/store/xxx-python3-3.13.12-env/bin:    ← wrapper (has packages)
     /nix/store/yyy-python3-3.13.12/bin:        ← bare (NO packages)
     /nix/store/zzz-other-package/bin:
     ...

The first python3 in PATH wins. Whether the wrapper or bare one comes first depends on Nix's topological sort of the dependency graph — it's not deterministic across different nixpkgs versions or package sets.

The fix

Prepend the wrapper explicitly in shellHook:

shellHook = ''
  export PATH="${pythonEnv}/bin:$PATH"
'';

This guarantees the wrapper is ALWAYS first, regardless of how Nix sorts the rest of PATH.

How to verify

# In your devshell:
which python3
# Should show: /nix/store/xxx-python3-3.13.12-env/bin/python3
# NOT: /nix/store/yyy-python3-3.13.12/bin/python3

# The -env suffix means it's the wrapper with packages.
# Verify:
python3 -c "import pulumi; print(pulumi.__version__)"
# Should print the version, not ModuleNotFoundError

9. The Two-Pass Import Pattern

The problem

pulumi.nix needs to:

  1. Export pythonDeps (a function) for languages.nix to merge into withPackages
  2. Export package (Pulumi CLI wrapper) that references the final pythonEnv

But pythonEnv is created by languages.nix which calls pythonDeps... circular dependency.

The solution

Import pulumi.nix twice:

# languages.nix

# FIRST PASS: get pythonDeps function (pythonEnv not needed)
pulumiDeps = import ./pulumi.nix { inherit pkgs; };
# pulumiDeps.pythonDeps is available
# pulumiDeps.package would fail (assert pythonEnv != null)

# Create pythonEnv using the deps from first pass
pythonEnv = pkgs.python313.withPackages (ps: [
  ps.ipython
] ++ pulumiDeps.pythonDeps ps);

# SECOND PASS: now provide pythonEnv for the wrapper
pulumiPkg = import ./pulumi.nix { inherit pkgs pythonEnv; };
# pulumiPkg.package is now available (pythonEnv != null)
# pulumiPkg.env is now available

Why assert and not if

# In pulumi.nix:
package = let
  resolvedPythonEnv = assert pythonEnv != null; pythonEnv;
in ...

assert fails at evaluation time with a clear error if someone tries to access .package from a first-pass import. This is intentional — it prevents silent bugs where the wrapper points at null.


10. The PULUMI_PYTHON_CMD Wrapper

What Pulumi does without it

  1. pulumi up starts the Pulumi engine (Go binary)
  2. Engine spawns the Python language plugin (pulumi-language-python)
  3. Language plugin searches PATH for python3 or python
  4. It finds the first python3 in PATH
  5. It runs python3 /path/to/your/__main__.py
  6. If python3 is the bare interpreter: ModuleNotFoundError: No module named 'pulumi'

What the wrapper does

#!/nix/store/.../bin/pulumi
export PULUMI_PYTHON_CMD="/nix/store/xxx-python3-3.13.12-env/bin/python"
exec /nix/store/yyy-pulumi-3.192.0/bin/pulumi "$@"

The language plugin checks PULUMI_PYTHON_CMD first, before searching PATH. This guarantees it uses the withPackages interpreter.

Why symlinkJoin

The wrapper script is a single binary (pulumi). But Pulumi ships many binaries:

  • pulumi — main CLI
  • pulumi-language-python — Python language plugin
  • pulumictl — utilities

symlinkJoin merges them all into one directory. The wrapper's pulumi takes priority over the original (symlinks resolve in order).


11. Pyright Integration

The problem

Pyright exists in two forms:

Form Package How it works Works for Pulumi?
Standalone Node binary nixpkgs#pyright Runs as pyright CLI NO
Python package PyPI pyright Runs as python -m pyright YES

Pulumi's language plugin runs pyright via python -m pyright. The standalone Node binary is not in Python's site-packages, so python -m pyright fails with No module named 'pyright'.

The fix

Build pyright from its PyPI wheel and include it in pythonDeps:

pyrightPkg = pkgs.python313Packages.buildPythonPackage {
  pname = "pyright";
  version = "1.1.407";
  format = "wheel";
  src = pkgs.fetchurl {
    url = "https://files.pythonhosted.org/packages/.../pyright-1.1.407-py3-none-any.whl";
    hash = "sha256-...";
  };
  propagatedBuildInputs = with pkgs.python313Packages; [
    nodeenv          # pyright needs Node.js at runtime
    typing-extensions
  ];
};

This makes python -m pyright work inside the Nix-provided Python environment.

Verification

# In devshell:
python3 -m pyright --version
# Output: pyright 1.1.407

# NOT:
pyright --version
# This is the standalone Node binary (different thing, also works but not for Pulumi)

12. Environment Variables Reference

Variable Value Set Where Purpose
PULUMI_PYTHON_CMD /nix/store/.../python3 pulumi.nix env Tell Pulumi which Python to use
UV_SYSTEM_PYTHON "1" languages.nix env Prevent uv from downloading Python
PYTHONDONTWRITEBYTECODE "1" languages.nix env No .pyc files in nix store
PYRIGHT_PYTHON_FORCE_VERSION "latest" .env file Prevent pyright from complaining about version

Variables to DELETE when migrating

Variable Why remove
PULUMI_PYTHON_VIRTUALENV No venv. Pulumi uses PULUMI_PYTHON_CMD instead.
UV_PROJECT_ENVIRONMENT No venv. Python packages are in nix store.
VIRTUAL_ENV No venv.
PYTHONPATH Don't set manually. withPackages handles site-packages.

13. Migration Checklist: pip/uv → Nix

[ ] 1. Audit: List all Python dependencies from pyproject.toml/requirements.txt
[ ] 2. Classify: For each package, check if it's in nixpkgs python313Packages
[ ] 3. Build: Create pulumi.nix with pythonDeps function
[ ] 4. Fetch: Get PyPI wheel URLs and hashes for packages not in nixpkgs
[ ] 5. Merge: Create languages.nix with single python.withPackages call
[ ] 6. Shell: Create devshell with PATH fix in shellHook
[ ] 7. Verify: `nix develop` → `python3 -c "import pulumi"` → success
[ ] 8. Verify: `pulumi preview` works without errors
[ ] 9. Verify: `python -m pyright --version` works
[ ] 10. Clean: Remove .venv/ directory
[ ] 11. Clean: Remove UV_PROJECT_ENVIRONMENT from .envrc
[ ] 12. Clean: Remove PULUMI_PYTHON_VIRTUALENV from .env files
[ ] 13. Clean: Remove virtualenv from Pulumi.yaml (or set to empty)
[ ] 14. Update: Set PULUMI_PYTHON_CMD in Pulumi.yaml or rely on env var
[ ] 15. Test: Full `pulumi up` against a real stack
[ ] 16. Commit: All changes

14. Debugging

Python can't find packages

# Which python3 is being used?
which python3
# Should be: /nix/store/xxx-python3-3.13.12-env/bin/python3 (note: -env suffix)

# What packages are available?
python3 -c "import sys; print('\n'.join(sys.path))"

# Can we import pulumi?
python3 -c "import pulumi; print(pulumi.__version__)"

# Can we import a provider?
python3 -c "import pulumi_kubernetes; print('ok')"

Pulumi can't find Python

# What Python does Pulumi see?
echo $PULUMI_PYTHON_CMD

# Does that Python have packages?
$PULUMI_PYTHON_CMD -c "import pulumi; print('ok')"

# Force verbose output:
PULUMI_DEBUG_COMMANDS=1 pulumi preview

Hash mismatch when adding a package

error: hash mismatch in fixed-output derivation
  specified: sha256-AAAA...
  got:       sha256-BBBB...

Copy the got: hash and replace the specified hash in your fetchurl call.

"No module named 'pyright'" during pulumi up

Pyright is not in your pythonDeps. Add pyrightPkg to the list. See section 11.

Multiple python3 binaries in PATH

# List all python3 binaries in PATH
type -a python3

# Should show ONE entry (the -env wrapper)
# If you see two, your shellHook PATH prepend is missing or wrong

15. Common Failure Modes

Symptom Cause Fix
ModuleNotFoundError: No module named 'pulumi' Wrong python3 in PATH (bare, not wrapper) Add export PATH="${pythonEnv}/bin:$PATH" to shellHook
pulumi up hangs then fails with Python error PULUMI_PYTHON_CMD not set or points to bare python3 Set PULUMI_PYTHON_CMD in pulumi.nix env
python -m pyright fails pyright not in withPackages Add pyrightPkg to pythonDeps
Hash mismatch on nix develop PyPI wheel was updated (same version, different file) Re-fetch hash (see section 7)
assert failure accessing .package First-pass import trying to access .package Only access .package from second-pass import
pulumi install creates a .venv PULUMI_PYTHON_VIRTUALENV still set Remove from .env and Pulumi.yaml
Two different Python versions in PATH Multiple withPackages calls Use ONE withPackages call in languages.nix

16. Native (Non-Pure) Python Packages

Most Pulumi provider SDKs are pure Python (py3-none-any). But some runtime deps may have native extensions (C/C++/Rust).

Identifying native packages

# Check the wheel filename on PyPI
# Pure:   package-1.0.0-py3-none-any.whl
# Native: package-1.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Building native packages

If the package is in nixpkgs, just use ps.PACKAGE — nixpkgs handles the build.

If not in nixpkgs:

nativePackage = pkgs.python313Packages.buildPythonPackage {
  pname = "my-native-package";
  version = "1.0.0";

  # For native packages, use sdist (source distribution) not wheel
  src = pkgs.fetchPypi {
    pname = "my-native-package";
    version = "1.0.0";
    hash = "sha256-...";
  };

  # Native build dependencies
  nativeBuildInputs = [
    pkgs.python313Packages.setuptools
    pkgs.python313Packages.wheel
    pkgs.python313Packages.cython  # If needed
  ];

  # System libraries needed at build time
  buildInputs = [
    pkgs.openssl  # Example: if package links against libssl
    pkgs.libffi
  ];

  # Runtime dependencies
  propagatedBuildInputs = with pkgs.python313Packages; [
    cffi
  ];
};

Using manylinux wheels with autoPatchelf

For pre-compiled Linux wheels that aren't in nixpkgs:

nativeFromWheel = pkgs.python313Packages.buildPythonPackage {
  pname = "my-package";
  version = "1.0.0";
  format = "wheel";

  src = pkgs.fetchurl {
    url = "https://files.pythonhosted.org/packages/.../my_package-1.0.0-cp313-cp313-manylinux_2_17_x86_64.whl";
    hash = "sha256-...";
  };

  # autoPatchelfHook fixes the RPATH of native libraries
  # to point at nix store paths instead of /usr/lib
  nativeBuildInputs = [ pkgs.autoPatchelfHook ];

  # System libraries the native code links against
  buildInputs = [
    pkgs.stdenv.cc.cc.lib  # libstdc++
    pkgs.openssl           # libssl, libcrypto
    pkgs.zlib              # libz
  ];
};

17. Adding a New Provider SDK

Scenario: Add pulumi-aws to your project

# Step 1: Check if it's in nixpkgs
nix eval nixpkgs#python313Packages.pulumi-aws.version 2>/dev/null
# If this prints a version → use ps."pulumi-aws" in pythonDeps
# If this fails → build from PyPI wheel

# Step 2: Find the wheel on PyPI
# Go to: https://pypi.org/project/pulumi-aws/VERSION/#files
# Find: pulumi_aws-VERSION-py3-none-any.whl
# Copy the URL

# Step 3: Get the hash
nix store prefetch-file --hash-type sha256 \
  "https://files.pythonhosted.org/packages/.../pulumi_aws-7.7.0-py3-none-any.whl"

# Step 4: Add to pulumi.nix

Add to pulumi.nix:

pulumiAws = mkPulumiPypiPackage {
  pname = "pulumi_aws";
  version = "7.7.0";
  url = "https://files.pythonhosted.org/packages/.../pulumi_aws-7.7.0-py3-none-any.whl";
  hash = "sha256-HASH_FROM_STEP_3";
  meta.description = "Pulumi AWS provider SDK";
};

Add to pythonDeps:

pythonDeps = ps: [
  # ... existing deps ...
  pulumiAws          # NEW
];

Rebuild:

nix develop
python3 -c "import pulumi_aws; print('ok')"

18. Adding a New Runtime Dependency

Scenario: Add requests to your Pulumi program

# Step 1: Check nixpkgs
nix eval nixpkgs#python313Packages.requests.version
# Output: "2.32.3" → it's in nixpkgs

# Step 2: Add to pythonDeps in pulumi.nix
pythonDeps = ps: [
  # ... existing deps ...
  ps.requests         # NEW — HTTP library
];
# Step 3: Rebuild
nix develop
python3 -c "import requests; print(requests.__version__)"

19. Updating Package Versions

Updating a nixpkgs package

nixpkgs packages update when you update your flake inputs:

nix flake update nixpkgs
nix develop
python3 -c "import pulumi; print(pulumi.__version__)"

Updating a PyPI wheel package

  1. Find the new version on PyPI
  2. Get the new wheel URL
  3. Get the new hash
  4. Update version, url, hash in mkPulumiPypiPackage
# Example: update pulumi-kubernetes from 4.23.0 to 4.24.0
# 1. Go to https://pypi.org/project/pulumi-kubernetes/4.24.0/#files
# 2. Copy py3-none-any.whl URL
# 3. Get hash:
nix store prefetch-file --hash-type sha256 "NEW_URL"
# 4. Update pulumi.nix

Verifying after update

nix develop
python3 -c "import pulumi_kubernetes; print(pulumi_kubernetes.__version__)"
# Should show new version

pulumi preview -s dev
# Should work without errors

20. Offline and Air-Gapped Builds

How Nix enables offline

After the first nix develop succeeds, all packages are cached in /nix/store. Subsequent nix develop invocations don't require network access.

For fully air-gapped environments

  1. Build on a connected machine: nix develop
  2. Export the closure: nix copy --to /path/to/portable-store .#devShells.x86_64-linux.default
  3. Transfer the store to the air-gapped machine
  4. Import: nix copy --from /path/to/portable-store .#devShells.x86_64-linux.default

PyPI wheels and air-gap

The wheel URLs in fetchurl are fetched once and content-addressed in the nix store. After first fetch, the URL is never contacted again. The hash verification ensures integrity.


21. CI/CD Integration

GitHub Actions

- uses: cachix/install-nix-action@v31
- run: nix develop --command pulumi preview -s prod

Pulumi.yaml for Nix

name: my-infrastructure
runtime:
  name: python
  options:
    # Do NOT set virtualenv — Nix handles Python packages
    typechecker: pyright
description: Infrastructure managed by Pulumi with Nix-native Python
main: src/

What NOT to put in Pulumi.yaml

# DELETE these — they conflict with Nix
runtime:
  options:
    virtualenv: .venv          # DELETE
    toolchain: pip              # DELETE

22. Complete Reference Implementation

Minimal working pulumi.nix

{ pkgs, pythonEnv ? null }:

let
  mkPypiPkg = { pname, version, url, hash }:
    pkgs.python313Packages.buildPythonPackage {
      inherit pname version;
      format = "wheel";
      src = pkgs.fetchurl { inherit url hash; };
      propagatedBuildInputs = with pkgs.python313Packages; [ parver pulumi semver ];
      doCheck = false;
      pythonImportsCheck = [ pname ];
    };

  pulumiK8s = mkPypiPkg {
    pname = "pulumi_kubernetes";
    version = "4.23.0";
    url = "https://files.pythonhosted.org/packages/.../pulumi_kubernetes-4.23.0-py3-none-any.whl";
    hash = "sha256-...";
  };

in {
  pythonDeps = ps: [
    ps.pulumi
    ps."pulumi-random"
    pulumiK8s
    ps.pydantic
    ps.pyyaml
  ];

  inherit pythonDeps;

  package = let
    env = assert pythonEnv != null; pythonEnv;
    wrapper = pkgs.writeShellScriptBin "pulumi" ''
      export PULUMI_PYTHON_CMD="${env}/bin/python"
      exec ${pkgs.pulumi}/bin/pulumi "$@"
    '';
  in pkgs.symlinkJoin {
    name = "pulumi-with-python";
    paths = [ wrapper pkgs.pulumi pkgs.pulumiPackages.pulumi-python ];
  };

  env = let env = assert pythonEnv != null; pythonEnv; in {
    PULUMI_PYTHON_CMD = "${env}/bin/python";
  };
}

Minimal working languages.nix

{ pkgs }:

let
  pulumiDeps = import ./pulumi.nix { inherit pkgs; };

  pythonEnv = pkgs.python313.withPackages (ps: [
    ps.ipython
  ] ++ pulumiDeps.pythonDeps ps);

  pulumiPkg = import ./pulumi.nix { inherit pkgs pythonEnv; };

in {
  inherit pythonEnv pulumiPkg;
  pythonPackages = [ pythonEnv ];
  env = {
    UV_SYSTEM_PYTHON = "1";
    PYTHONDONTWRITEBYTECODE = "1";
  } // pulumiPkg.env;
}

Minimal working devshell

{ pkgs, languages }:

pkgs.mkShell {
  nativeBuildInputs = languages.pythonPackages ++ [ languages.pulumiPkg.package ];
  shellHook = ''export PATH="${languages.pythonEnv}/bin:$PATH"'';
  env = languages.env;
}

Verification commands

# Enter devshell
nix develop

# Verify Python
which python3                                    # Should show -env path
python3 -c "import pulumi; print(pulumi.__version__)"
python3 -c "import pulumi_kubernetes; print('ok')"
python3 -m pyright --version                     # Should work

# Verify Pulumi
which pulumi                                     # Should show wrapper
echo $PULUMI_PYTHON_CMD                          # Should show -env python3
pulumi version                                   # Should show Pulumi version
pulumi preview -s dev                            # Should work

# Verify no venv contamination
echo $VIRTUAL_ENV                                # Should be empty
echo $PULUMI_PYTHON_VIRTUALENV                   # Should be empty
ls .venv 2>/dev/null                             # Should not exist

End of specification. This document covers the complete implementation surface for Nix-native Pulumi Python projects, from first principles through production deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment