jupiter-env-preparation.md

Setting Up Jupyter Kernel with Conda

If you encounter ipykernel errors when running Jupyter notebooks, follow these steps to register your Conda environment as a Jupyter kernel.

1. Activate Your Conda Environment

conda activate [YOUR_ENV_NAME]

# Example
conda activate base

2. Install Required Packages

conda install ipykernel ipython numpy pandas pyarrow

3. Register Kernel with Jupyter

python -m ipykernel install --user --name=[YOUR_ENV_NAME] --display-name "[DISPLAY_NAME]"

# Example
python -m ipykernel install --user --name=base --display-name "Python (base)"

Note: Replace [YOUR_ENV_NAME] with your actual environment name and [DISPLAY_NAME] with how you want it to appear in Jupyter.

After completing these steps, restart Jupyter and select your kernel from the kernel dropdown menu.

If You Still Get Errors

Some code requires specific Python versions. If you get compatibility errors, check your Python version and create a new Conda environment with the required version.

Create a New Conda Environment with Specific Python Version

conda create -n [YOUR_ENV_NAME] python=[PYTHON_VERSION] [YOUR_PACKAGES] -y

# Example: Create environment with Python 3.11
conda create -n data-science python=3.11 pandas numpy pyarrow jupyter -y

Activate the New Environment

conda activate [YOUR_ENV_NAME]

# Example
conda activate data-science

Select the New Kernel in Jupyter/VS Code

1. In VS Code: Press Cmd+Shift+P → "Python: Select Kernel" → Choose your new environment
2. In Jupyter: Click the kernel selector in the top right → Choose your new environment

After switching kernels, run your code again to verify compatibility.

ArrowKeyError: Type Extension Error

Symptom

When saving a DataFrame to Parquet, you may get an error like:

ArrowKeyError: A type extension with name pandas.period already defined

or

ArrowKeyError: No type extension with name arrow.py_extension_type found

Root Cause

Your DataFrame contains pandas extension types (like StringDtype, PeriodDtype) that PyArrow cannot serialize properly. This often happens when:

1. You read CSV with dtype={"column": "string"} (creates StringDtype)
2. You perform operations like joins or groupby that create new extension types
3. Your Python version and pandas/pyarrow versions are incompatible

Solutions

Option 1: Upgrade PyArrow (Recommended)

conda activate [YOUR_ENV_NAME]
pip install --upgrade pyarrow

Option 2: Install Compatible Versions

conda activate [YOUR_ENV_NAME]
pip install pandas==2.2.0 pyarrow==14.0.0

Option 3: Use Python 3.11 Instead

If you're using Python 3.14.x, switch to Python 3.11:

# Create new environment with Python 3.11
conda create -n data-science python=3.11 pandas numpy pyarrow jupyter -y
conda activate data-science

Then select this kernel in VS Code or Jupyter.

Option 4: Convert Extension Types Before Saving

Convert all pandas extension types to regular NumPy types:

# Convert string columns to object type
final_clean = final.copy()
for col in final_clean.columns:
    if str(final_clean[col].dtype) == "string":
        final_clean[col] = final_clean[col].astype("object")

final_clean.to_parquet(OUT / "sales_processed.parquet", index=False)

Try Option 1 first. If the error persists, use Option 3 (Python 3.11).