Python / Pytorch version problems debugging guide

python-pytorch-version-conflicts-debuggin-guide.md

Known Dependencies Management

This guide covers managing Python package dependencies when you know the working versions.

1. Using Requirements.txt

# Generate requirements.txt with specific versions
pip freeze > requirements.txt

Your requirements.txt can include version specifiers:

torch==2.1.2  # Exact version
numpy>=1.20.0  # Minimum version
pandas~=1.4.0  # Compatible release (same as >=1.4.0, <1.5.0)

2. Virtual Environments

Always use virtual environments for project isolation:

# Create new virtual environment
python -m venv myproject_env

# Activate it (Unix/macOS)
source myproject_env/bin/activate
# OR Windows
myproject_env\Scripts\activate

3. Using pip-tools

Enhanced dependency management:

# Install pip-tools
pip install pip-tools

# Create requirements.in with direct dependencies
# Example requirements.in contents:
# torch
# xformers
# numpy

# Generate locked requirements.txt
pip-compile requirements.in

# Install from requirements.txt
pip-sync

4. Dependency Checking

# Check for conflicts
pip check

# Detailed dependency tree
pip install pipdeptree
pipdeptree -p torch  # Check specific package

5. Documentation Best Practices

Document dependencies with their purposes:

# requirements.txt
torch==2.1.2  # Deep learning framework
torchvision==0.16.2  # Computer vision utilities
xformers==0.0.22.post7  # Memory-efficient transformers

6. Conda Environments

For conda users:

# environment.yml
name: myproject
channels:
  - pytorch
  - conda-forge
dependencies:
  - python=3.10
  - torch=2.1.2
  - torchvision=0.16.2
  - xformers=0.0.22

Conda environment commands:

# Export environment
conda env export > environment.yml

# Create from file
conda env create -f environment.yml

---

Unknown Dependencies Management

This guide covers managing Python package dependencies when working with code that doesn't specify versions.

1. Installation Order Strategy

Follow this order for interdependent packages:

# 1. Core framework (e.g., PyTorch)
# 2. Framework extensions (e.g., torchvision)
# 3. Dependent packages (e.g., transformers)

2. Debugging Tools

# Check installed version
pip show package_name

# Check package dependencies
pip show package_name | grep Requires

# Detailed dependency tree
pip install pipdeptree
pipdeptree -p package_name

3. Version Conflict Resolution

# If package_B installation fails:

# Check dependencies
pip show package_B

# Check installed versions
pip show package_A

# Try older version
pip install package_B==0.0.20

# Clean install in correct order
pip uninstall package_A package_B
pip install package_A==version
pip install package_B==version

4. Jupyter Notebook Version Checker

import pkg_resources

def get_package_version(package_name):
    try:
        return pkg_resources.get_distribution(package_name).version
    except pkg_resources.DistributionNotFound:
        return "Not installed"

# Check versions
packages = ['torch', 'transformers', 'xformers']
for pkg in packages:
    print(f"{pkg}: {get_package_version(pkg)}")

5. Practical Workflow Example

# Try requirements.txt if available
pip install -r requirements.txt

# Otherwise, systematic installation:

# 1. Check CUDA version
nvidia-smi

# 2. Install PyTorch for your CUDA version
pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu121

# 3. Install other packages
pip install transformers
pip install datasets
pip install xformers

# Debug installation
pip install package_name --verbose
pip install package_name==  # Show available versions

6. Troubleshooting Checklist

1. Check project's GitHub issues
2. Note notebook's last update date
3. Review major dependencies' release notes
4. Try fresh virtual environment
5. Consider using older framework versions
6. Check repository's:
   • requirements.txt
   • setup.py
   • Issue tracker
   • Release dates

Remember to document working combinations once found!