Skip to content

Instantly share code, notes, and snippets.

@yatsu

yatsu/pyenv-python38-macos.sh

Created January 2, 2021 11:35
Show Gist options
  • Save yatsu/9ae2dc633511d6d97efccd4fcfa8c8b1 to your computer and use it in GitHub Desktop.
Save yatsu/9ae2dc633511d6d97efccd4fcfa8c8b1 to your computer and use it in GitHub Desktop.
Download ZIP
Install Python 3.8.6 with pyenv on macOS Big Sur (11)
Raw
pyenv-python38-macos.sh
#!/bin/sh
# 1) Install Xcode 12
# 2) Install command line tools: `xcode-select --install`
# 3) Install Homebrew
# 4) brew install zlib bzip2 xz
CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix xz)/include" \
LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib -L$(brew --prefix xz)/lib" \
pyenv install 3.8.6
@yatsu
Copy link
Author

yatsu commented Jan 2, 2021

If your Python has an error about module _bz2 as below, try this gist to reinstall Python.

$ python -c "import numpy; import pandas"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/__init__.py", line 51, in <module>
    from pandas.core.api import (
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/core/api.py", line 31, in <module>
    from pandas.core.groupby import Grouper, NamedAgg
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/core/groupby/generic.py", line 65, in <module>
    from pandas.core.frame import DataFrame
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/core/frame.py", line 119, in <module>
    from pandas.core import algorithms, common as com, generic, nanops, ops
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/core/generic.py", line 112, in <module>
    from pandas.io.formats import format as fmt
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/io/formats/format.py", line 77, in <module>
    from pandas.io.common import stringify_path
  File "/Users/yatsu/.pyenv/versions/py38/lib/python3.8/site-packages/pandas/io/common.py", line 3, in <module>
    import bz2
  File "/Users/yatsu/.pyenv/versions/3.8.6/lib/python3.8/bz2.py", line 19, in <module>
    from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'

@kailichou
Copy link

Thanks for your quick reply.

I still run into a WARNING after installing 3.8.6

bash-3.2$ pyenv install 3.8.6
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.6.tar.xz...
-> https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tar.xz
Installing Python-3.8.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
**WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?**
Installed Python-3.8.6 to /Users/kailichou/.pyenv/versions/3.8.6

What I did,

  1. xcode-select installed and up-to-date
  2. brew and dependencies (bzip2, zlib, readline, xz) have installed
  3. uninstall pyenv and reinstall pyenv
  4. install 3.8.6 and got a WARNING

Many recommendations are to install bzip2-devel, libbz2-dev... for Linux system, which brew doesn't have those formulae for MacOS.

What happened during installing pyenv
git issue, does this cause my bzip2 compiler problem?
Many thanks.

bash-3.2$ brew install pyenv
Updating Homebrew...
fatal: Unable to create '/usr/local/Homebrew/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
fatal: Unable to create '/usr/local/Homebrew/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/bottles/pyenv-1.2.2
Already downloaded: /Users/kailichou/Library/Caches/Homebrew/downloads/d824644ecebd72fcb903b1e8a2fecf162c94a7ce960f0977a782176dd387b559--pyenv-1.2.21.big_sur.bottle.tar.gz
==> Pouring pyenv-1.2.21.big_sur.bottle.tar.gz
🍺  /usr/local/Cellar/pyenv/1.2.21: 716 files, 2.6MB

@yatsu
Copy link
Author

yatsu commented Jan 4, 2021

Check bzip2 is correctly installed.

$ ls $(brew --prefix bzip2)/lib
libbz2.a

libbz2.a is the library Python should link to. -devel package is not required in macOS/Homebrew.

Next, make sure the arguments -I$(brew --prefix bzip2)/include and -L$(brew --prefix bzip2)/lib (don't mix up -I and -L, include and lib).

pyenv install -v shows the compile log. You can check if bz2 module was successfully installed.

$ LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install -v 3.8.5 2>&1 | grep bz2
building '_bz2' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I./Include -I. -I/opt/brew/opt/readline/include -I/opt/brew/opt/zlib/include -I/usr/local/include -I/private/var/folders/nb/dw5p3r0j17q4y5r5wkhpnr0r0000gn/T/python-build.20210104122251.50610/Python-3.8.5/Include -I/private/var/folders/nb/dw5p3r0j17q4y5r5wkhpnr0r0000gn/T/python-build.20210104122251.50610/Python-3.8.5 -c /private/var/folders/nb/dw5p3r0j17q4y5r5wkhpnr0r0000gn/T/python-build.20210104122251.50610/Python-3.8.5/Modules/_bz2module.c -o build/temp.macosx-11.1-x86_64-3.8/private/var/folders/nb/dw5p3r0j17q4y5r5wkhpnr0r0000gn/T/python-build.20210104122251.50610/Python-3.8.5/Modules/_bz2module.o
clang -bundle -undefined dynamic_lookup -L/opt/brew/opt/readline/lib -L/opt/brew/opt/readline/lib -L/Users/yatsu/.pyenv/versions/3.8.5/lib -L/opt/brew/opt/zlib/lib -L/opt/brew/opt/bzip2/lib -L/opt/brew/opt/readline/lib -L/opt/brew/opt/readline/lib -L/Users/yatsu/.pyenv/versions/3.8.5/lib -L/opt/brew/opt/zlib/lib -L/opt/brew/opt/bzip2/lib -L/opt/brew/opt/readline/lib -L/opt/brew/opt/readline/lib -L/Users/yatsu/.pyenv/versions/3.8.5/lib -L/opt/brew/opt/zlib/lib -L/opt/brew/opt/bzip2/lib -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/opt/brew/opt/readline/include -I/opt/brew/opt/readline/include -I/Users/yatsu/.pyenv/versions/3.8.5/include -I/opt/brew/opt/zlib/include build/temp.macosx-11.1-x86_64-3.8/private/var/folders/nb/dw5p3r0j17q4y5r5wkhpnr0r0000gn/T/python-build.20210104122251.50610/Python-3.8.5/Modules/_bz2module.o -L/opt/brew/opt/readline/lib -L/Users/yatsu/.pyenv/versions/3.8.5/lib -L/opt/brew/opt/zlib/lib -L/opt/brew/opt/bzip2/lib -L/usr/local/lib -lbz2 -o build/lib.macosx-11.1-x86_64-3.8/_bz2.cpython-38-darwin.so -Wl,-search_paths_first
/opt/brew/bin/ginstall -c -m 644 ./Lib/bz2.py /Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8
/opt/brew/bin/ginstall -c -m 644 ./Lib/test/test_bz2.py /Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/test
/opt/brew/bin/ginstall -c -m 644 ./Lib/encodings/bz2_codec.py /Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/encodings
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/test/test_bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/encodings/bz2_codec.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/test/test_bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/encodings/bz2_codec.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/test/test_bz2.py'...
Compiling '/Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/encodings/bz2_codec.py'...
copying build/lib.macosx-11.1-x86_64-3.8/_bz2.cpython-38-darwin.so -> /Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/lib-dynload
changing mode of /Users/yatsu/.pyenv/versions/3.8.5/lib/python3.8/lib-dynload/_bz2.cpython-38-darwin.so to 755

@kailichou
Copy link

kailichou commented Jan 6, 2021 •
edited
Loading

I took another way.

OS: Big Sur 11.1

What I tried

  • Remove brew and its packages
  • and reinstall brew, pyenv, its dependencies python3.8.5
  • without brew install bzip2 or brew install openblas

Result

  • no more Polyfit issue with numpy
  • or no more _bz2 module issue with pandas

I have no idea what's changed, but it worked.

Thanks for your help anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment