vidakDK/pycurl.md

Last active November 2, 2023 03:06

Star (24) You must be signed in to star a gist
Fork (1) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/vidakDK/de86d751751b355ed3b26d69ecdbdb99.js"></script>
Save vidakDK/de86d751751b355ed3b26d69ecdbdb99 to your computer and use it in GitHub Desktop.

Download ZIP

Install `pycurl` on MacOS Big Sur 11.0.1

Raw

pycurl.md

Introduction

Installing pycurl on MacOS with Python 3.6+ and newer proved to be tricky, especially since a lot of the available resources seem to be outdated (1, 2, 3).

This gist was last updated at 2020-12-31.

Steps:

Remove previously installed Homebrew versions of curl:

    brew uninstall curl
    brew uninstall openssl
    brew uninstall curl-openssl

Install openssl and a curl version that uses openssl rather than SecureTransport, which MacOS curl uses by default. Note that Homebrew no longer allows options while installing, so it's not possible to do brew install curl --with-openssl. This was replaced with a different brew formula, curl-openssl, and again replaced on 2020-01-20 with the current version of just curl.:

    brew install openssl
    brew install curl  # this is the latest version with openssl

Now we need to use the new curl when installing pycurl. We can also export the path to the new curl to make it used by default:

    echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.zshrc  # or ~/.bash_profile
    echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc
    source ~/.zshrc
    source ~/.virtualenvs/foo_venv/bin/activate  # or whichever venv you're using
    export PYCURL_SSL_LIBRARY=openssl
    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"
    pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/[email protected]" pycurl

Test with:

    python
    >>> import pycurl

There should now be two different versions of curl:

The original OS X one in /usr/bin/curl:

    $ /usr/bin/curl --version
curl 7.64.1 (x86_64-apple-darwin20.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.41.0
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

The Homebrew one in /usr/local/opt/curl/bin/curl:

    $ /usr/local/opt/curl/bin/curl --version
curl 7.74.0 (x86_64-apple-darwin20.2.0) libcurl/7.74.0 (SecureTransport) OpenSSL/1.1.1i zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.42.0 librtmp/2.3
Release-Date: 2020-12-09
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

And our pycurl should use the second one, both at linker level and at compile level.

8bitter commented Jan 22, 2020 •

edited

Loading

As of 2020-01-20, brew install openssl installs [email protected]. In the pip install command, the install-option for the openssl-dir should be updated to:
--install-option="--openssl-dir=/usr/local/opt/[email protected]"

DeoLeung commented Feb 24, 2020

I follow the example and got ld: library not found for -lcares, anyone knows how to add the missing cares?

DeoLeung commented Feb 24, 2020

I follow the example and got ld: library not found for -lcares, anyone knows how to add the missing cares?

found that it's called c-ares

in my case there still quite some missing library, I need to manually do
LDFLAGS="-L/usr/local/opt/curl-openssl/lib -L/usr/local/opt/c-ares/lib"

usually the missing dynamic library could be found under /usr/local/opt with a similar name

neoclust commented Aug 31, 2020 •

edited

Loading

In catalina, it wants to install in /System/Library/Framework/ and fails

idavidmcdonald commented Dec 31, 2020 •

edited

Loading

I got this working by slightly tweaking your step 3 to change openssl to [email protected] as already mentioned above but I also needed to change curl-openssl to curl based on the directories that exist in my /usr/local/opt. You'll see on https://formulae.brew.sh/formula/curl that the curl-openssl formulea is now known as curl.

echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.zshrc  # or ~/.bash_profile
    echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.zshrc
    source ~/.zshrc
    source ~/.virtualenvs/pm-sem/bin/activate  # or whichever venv you're using
    export PYCURL_SSL_LIBRARY=openssl
    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"
    pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/[email protected]" pycurl

Author

vidakDK commented Dec 31, 2020

@idavidmcdonald @8bitter I updated the gist with your comments and confirm they work :) thanks for the effort!

geosteffanov commented Mar 9, 2021 •

edited

Loading

@vidakDK I keep hitting the issue, even after following the instructions. Perhaps there has been an update to homebrew or BigSur?

For reference, my homebrew curl version

curl 7.75.0 (x86_64-apple-darwin20.2.0) libcurl/7.75.0 (SecureTransport) OpenSSL/1.1.1j zlib/1.2.11 brotli/1.0.9 zstd/1.4.9 libidn2/2.3.0 libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2021-02-03
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

When trying to run my code, I still get this error

    import pycurl  # type: ignore
ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Has anyone hit this issue again?

EDIT: I am hitting this on Python 2, sorry if this comment is completely unrelated seeing as you have posted this gist for Python 3. I still think if anyone finds a fix to a similar problem then, this will help me.

asifkhan69 commented Sep 29, 2021

I have recently upgrade my MacOS High serria to Catalina 10.15.7 and start facing this problem for virtual environment for pycurl. I did all your steps as it is to resolve my pycurl problem but am receving this error while executing this command
pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/[email protected]" pycurl

Error:
WARNING: Discarding https://files.pythonhosted.org/packages/fe/8f/711d436c76938093dd10dae465ccee316c6241cfdada06c5cc3b5c1f74b8/pycurl-7.18.2.tar.gz#sha256=cda1085b7911ca802746d816f37d541165ad1e20c241d22e4c304ddb2bc81ffe (from https://pypi.org/simple/pycurl/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pycurl (from versions: 7.18.2, 7.19.0, 7.19.0.1, 7.19.0.2, 7.19.0.3, 7.19.3, 7.19.3.1, 7.19.5, 7.19.5.1, 7.19.5.2, 7.19.5.3, 7.21.5, 7.43.0, 7.43.0.1, 7.43.0.2, 7.43.0.3, 7.43.0.4, 7.43.0.5, 7.43.0.6, 7.44.0, 7.44.1)
ERROR: No matching distribution found for pycurl

ZhaoGuan commented Oct 31, 2021

I have a error when i import pycurl
pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.77.0)

Byusa commented Nov 19, 2021

I have the same error when I import pycurl

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

fffffrolov commented Nov 21, 2021

Same for me. Can't find solution.

p-i- commented Nov 23, 2021 •

edited

Loading

Same:

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

I noticed:

    export LDFLAGS="-L/usr/local/opt/curl/lib"
    export CPPFLAGS="-I/usr/local/opt/curl/include"

are probably referencying my system curl.

brew list curl shows: usr/local/Cellar/curl/7.80.0/

So instead I do:

export LDFLAGS="-L/usr/local/Cellar/curl/7.80.0/lib"
export CPPFLAGS="-I/usr/local/Cellar/curl/7.80.0/include"

But fail:

(.venv)
🧢 pi@pPro18-4 ~/curlfail
> pip install --no-cache-dir --compile --ignore-installed --install-option="--with-openssl" --install-option="--openssl-dir=/usr/local/opt/[email protected]" pycurl
/Users/pi/curlfail/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py:229: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Collecting pycurl
  Downloading pycurl-7.44.1.tar.gz (227 kB)
     |████████████████████████████████| 227 kB 1.4 MB/s
Skipping wheel build for pycurl, due to binaries being disabled for it.
Installing collected packages: pycurl
    Running setup.py install for pycurl ... done
Successfully installed pycurl-7.44.1
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/pi/curlfail/.venv/bin/python -m pip install --upgrade pip' command.
(.venv)
🧢 pi@pPro18-4 ~/curlfail
> python
Python 3.10.0 (default, Oct 23 2021, 18:32:23) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

idavidmcdonald commented Nov 23, 2021

So I ran into new problems on my new laptop trying to get this to work. It might not be applicable for everyone else but thought I'd share just in case.

Context

pyenv --version

pyenv 2.0.7

brew list

...
[email protected]
openssl@3
...

curl --version

curl 7.80.0 ......

/usr/bin/curl --version                                                                                               
curl 7.64.1

where curl                                                                                                              

/usr/local/opt/curl/bin/curl
/usr/bin/curl

cat ~/.zshrc

...
export PATH="/usr/local/opt/[email protected]/bin:$PATH"
export PATH="/usr/local/opt/curl/bin:$PATH"
...

I tried the following with a few different versions of python and a few different versions of pycurl

PYCURL_SSL_LIBRARY="openssl" \
LDFLAGS="-L/usr/local/opt/curl/lib" \
CPPFLAGS="-I/usr/local/opt/curl/include" \
pip install --ignore-installed pycurl==7.44.1 --global-option="--with-openssl" --global-option="--openssl-dir=/usr/local/opt/[email protected]"

but then always reached an error when doing

python -c "import pycurl; print(pycurl.version)"                                                                        

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

Thing that solved it for me

We spotted that on my team mates machines that worked they had:

gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

whereas I had

gcc --version                        [2:30:22]
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

By deleting the Xcode Apple app from my machine, this changed my gcc --version output to match and it fixed my problems.

p-i- commented Nov 24, 2021

@idavidmcdonald Thankyou for taking the trouble to research scientifically.
Presumably the macOS system holds both gcc versions.
I wonder if it is possible to force the pycurl installation to use the non-xcode gcc without deleting the XCode app...
What is the where command? I don't have it, it seems.

p-i- commented Dec 1, 2021

Answer has been provided to how to change InstalledDir here:
https://stackoverflow.com/questions/70119147/how-to-install-gcc-on-macos-such-that-gcc-version-reports-installeddir-li/70146246#70146246

sudo xcode-select -switch /Library/Developer/CommandLineTools

Byusa commented Dec 1, 2021

For this error:

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.80.0)

2 Solutions:

Check your bash profile and see if you didn't mess up any python path.
Reinstall macOS (last Resort )
I figured that I had messed up with the Python of my system, so I decided to erase my MacOS and reinstall it and it worked.
It might not be the most optimum way but it fixed my issue.

geyang commented Jan 15, 2022

This worked for me on a brand new M1 max MBP

brew install openssl curl
10127  pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

then you can test with

python -c "import pycurl"

utxos commented Jan 21, 2022 •

edited

Loading

None of the above solutions worked for me on Mac 12.1 and I was getting error ImportError: pycurl: libcurl link-time version (7.77.0) is older than compile-time version (7.81.0)

I managed to work around it building the exact curl binary I needed (7.77.0) from the libcurl repo and wiring that into pycurl using install_name_tool. Locations of these will depend on your system but these were my steps. I already had xcode cli tools installed.

Idea is:
install_name_tool -change old.dylib new.dylib pycurl.so

For example:
install_name_tool -change /usr/lib/libcurl.4.dylib /usr/local/opt/curl/lib/libcurl.4.dylib /Users/user/Library/Python/2.7/lib/python/site-packages/pycurl.so

Another example on a different mac with python 3.9 and pyenv I ran this:
install_name_tool -change /usr/local/Cellar/curl/7.81.0/lib/libcurl.4.dylib /opt/vagrant/embedded/lib/libcurl.4.dylib /Users/username/.local/share/virtualenvs/website-TeaFLnO_/lib/python3.9/site-packages/pycurl.cpython-39-darwin.so

brew uninstall curl (might not be necessary)
Use install_name_tool to link the dynamic library you need. Use find \ -name if to locate the dylib's if needed.
Build the exact version of curl you need. Run autoreconf -fi, ./configure --prefix=/usr/local/opt/curl --without-ssl, make
pip uninstall pycurl && pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

Sanity is restored and everything just works now. Hope this helps someone!

eugen1j commented May 2, 2022 •

edited

Loading

Faced the same problem with Mac 11.6.5 and python 3.10

~ > pip install pycurl
~ > python -c 'import pycurl'
ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (7.77.0)

I didn't manage to change link-time version (7.64.1), so I download exact version of linked curl-7.64.1 and use it for pycurl compilation.

# Remove installed version
pip uninstall -y pycurl
brew uninstall curl

# Install openssl for pycurl compiling
brew install openssl || brew update openssl

# Downloading sources
wget https://curl.haxx.se/download/curl-7.64.1.tar.bz2
tar -xf curl-7.64.1.tar.bz2

# Setting up compiler flags and PATH
export PATH="$(pwd)/.cache/curl-${CURL_VERSION}/bin:$PATH"
export LDFLAGS="-L$(pwd)/curl-7.64.1/lib -L/usr/local/opt/openssl@3/lib"
export CPPFLAGS="-I$(pwd)/curl-7.64.1/include -I/usr/local/opt/openssl@3/include"

# Installing
pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl

# Checking up 
python -c "import pycurl" && echo "Success!"

nffdiogosilva commented Jul 10, 2022 •

edited

Loading

@eugen1j ,
This results to me in a:
ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Any suggestion?

eugen1j commented Jul 11, 2022 •

edited

Loading

@eugen1j , This results to me in a: ImportError: pycurl: libcurl link-time ssl backend (none/other) is different from compile-time ssl backend (openssl)

Any suggestion?

@nffdiogosilva Seems like you have no ssl backend in link-time libcurl. Try to set export PYCURL_SSL_LIBRARY=openssl before running python -c "import pycurl" && echo "Success!"

Also, you can try to compile pycurl without --install-option="--with-openssl" flag

zollak commented Jul 20, 2022

Thanks @eugen1j ! Based on your solution I was able to implement it for Mojave and Python2:

MacBook-Pro:v1.0 zollak$ source .env2.7/bin/activate
(.env2.7) MacBook-Pro:v1.0 zollak$ pip freeze
pycurl==7.43.0.5
(.env2.7) MacBook-Pro:v1.0 zollak$ python -c 'import pycurl'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: pycurl: libcurl link-time version (7.54.0) is older than compile-time version (7.64.1)

I didn't manage to change link-time version (7.54.0), so I download exact version of linked curl-7.54.0 and use it for pycurl compilation.

# Remove installed version
(.env2.7) MacBook-Pro:v1.0 zollak$ pip uninstall -y pycurl

# Downloading sources
(.env2.7) MacBook-Pro:v1.0 zollak$ mkdir curl
(.env2.7) MacBook-Pro:v1.0 zollak$ cd curl/
(.env2.7) MacBook-Pro:curl zollak$ wget https://curl.haxx.se/download/curl-7.54.0.tar.bz2
(.env2.7) MacBook-Pro:curl zollak$ tar -xf curl-7.54.0.tar.bz2

# Setting up compiler flags
(.env2.7) MacBook-Pro:curl zollak$ export LDFLAGS="-L$(pwd)/curl-7.54.0/lib -L/usr/local/opt/[email protected]/lib"
(.env2.7) MacBook-Pro:curl zollak$ export CPPFLAGS="-I$(pwd)/curl-7.54.0/include -I/usr/local/opt/[email protected]/include"

# Installing
(.env2.7) MacBook-Pro:curl zollak$ pip install --no-cache-dir --compile --install-option="--with-openssl" pycurl
Successfully installed pycurl-7.43.0.5

# Checking up 
(.env2.7) MacBook-Pro:curl zollak$ python -c 'import pycurl' && echo "Success"
Success

papagala commented Dec 29, 2022

I fell into this deep rabbit hole too many times, and finally this is probably the ultimate way of fixing this. Thanks @zollak. This works too for

Python 3
macOS Monterey 12.6.
Python installed using Pyenv

kumarvimal commented May 19, 2023 •

edited

Loading

--install-option is deprecated in the latest version of pip (e.g. pip 23.1.2).

Use --config-setting instead, e.g.

pip install --no-cache-dir --compile --ignore-installed  --config-setting="--build-option=--with-openssl" pycurl

Edit: Thanks to @jivens for pointing out, removed extra hyphen

jivens commented Jul 13, 2023

The above command is absolutely wonderful except that it contains an extra hyphen, so use this:

pip install --no-cache-dir --compile --ignore-installed --config-setting="--build-option=--with-openssl" pycurl

keshari-nandan commented Jul 28, 2023

Below command worked form me
MacOS: Apple M1 Pro (Ventura v13.4)

    pip uninstall pycurl
    PYCURL_SSL_LIBRARY=openssl LDFLAGS="-L/opt/homebrew/opt/openssl/lib" CPPFLAGS="-I/opt/homebrew/opt/openssl/include" pip install --no-cache-dir pycurl

Fabiankop commented Aug 28, 2023

Thanks @keshari-nandan , perfect job for me

noufalvlpr commented Oct 18, 2023

@keshari-nandan Thanks

vidakDK/pycurl.md

Introduction

Steps:

There should now be two different versions of curl:

8bitter commented Jan 22, 2020 • edited Loading

DeoLeung commented Feb 24, 2020

DeoLeung commented Feb 24, 2020

neoclust commented Aug 31, 2020 • edited Loading

idavidmcdonald commented Dec 31, 2020 • edited Loading

vidakDK commented Dec 31, 2020

geosteffanov commented Mar 9, 2021 • edited Loading

asifkhan69 commented Sep 29, 2021

ZhaoGuan commented Oct 31, 2021

Byusa commented Nov 19, 2021

fffffrolov commented Nov 21, 2021

p-i- commented Nov 23, 2021 • edited Loading

idavidmcdonald commented Nov 23, 2021

Context

Thing that solved it for me

p-i- commented Nov 24, 2021

p-i- commented Dec 1, 2021

Byusa commented Dec 1, 2021

geyang commented Jan 15, 2022

utxos commented Jan 21, 2022 • edited Loading

eugen1j commented May 2, 2022 • edited Loading

nffdiogosilva commented Jul 10, 2022 • edited Loading

eugen1j commented Jul 11, 2022 • edited Loading

zollak commented Jul 20, 2022

papagala commented Dec 29, 2022

kumarvimal commented May 19, 2023 • edited Loading

jivens commented Jul 13, 2023

keshari-nandan commented Jul 28, 2023

Fabiankop commented Aug 28, 2023

noufalvlpr commented Oct 18, 2023

8bitter commented Jan 22, 2020 •

edited

Loading

neoclust commented Aug 31, 2020 •

edited

Loading

idavidmcdonald commented Dec 31, 2020 •

edited

Loading

geosteffanov commented Mar 9, 2021 •

edited

Loading

p-i- commented Nov 23, 2021 •

edited

Loading

utxos commented Jan 21, 2022 •

edited

Loading

eugen1j commented May 2, 2022 •

edited

Loading

nffdiogosilva commented Jul 10, 2022 •

edited

Loading

eugen1j commented Jul 11, 2022 •

edited

Loading

kumarvimal commented May 19, 2023 •

edited

Loading