Importing configparser

The backported configparser is a P.I.T.A..

I tripped across this when trying to use py-jupyter-notebook with my Spack PR that adds dependencies via site.addsitedir(). It fails immediately. That package appears to work if you use "normal" Spack and PYTHONPATH, but it turns out that the only bit that uses configparser is jupyter-nbconvert and that does fail.

TL;DR is that it fails quickly in PR #8436 because configparser's .pth file is processed and breaks the backports namespace for every other package that uses it. In normal Spack w/ PYTHONPATH configparser's .pth file is never processed so the other backport packages work but when the code actually gets around to trying to use configparser if fails.

You can't import configparser via PYTHONPATH

In a standard Spack, with lmod modules...

Simply importing doesn't work

Adding the directory to PYTHONPATH and trying to import it doesn't work:

(alice)[15:27:28]spack-plain>>module purge
(alice)[15:27:40]spack-plain>>module load py-configparser
(alice)[15:27:47]spack-plain>>python
Python 2.7.15 (default, Jun 19 2018, 12:32:41)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import configparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages/configparser.py", line 12, in <module>
    from backports.configparser import (
ImportError: No module named backports.configparser
>>> from backports import configparser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named backports
>>>

pylint falls back to six.moves => works with PYTHONPATH

pylint works in a Spack environment that's tying things together with PYTHONPATH because when import configparser fails it just imports from six.moves (which aliases it to the old-school ConfigParser)

Here's the relevant bit from pylint:

 try:
     import configparser
 except ImportError:
     from six.moves import configparser

Here's the relevant bit from six.moves.

For the record:

ConfigParser is the Python 2 thing.
configparser is the Python 3 thing, in addition to the name change the functionality was improved (in 3.2?).
the configparser package (which should be "backports-configparser"...) makes the updated version available to Python 2.

configparser works when its dir is added via site.addsitedir()

site.addsitedir() provides an alternative to using PYTHONPATH to add directories to sys.path. Its documentation says:

Add a directory to sys.path and process its .pth files. Typically used in sitecustomize or usercustomize (see above).

configparser installs a .pth file.

.../opt/spack/.../py-configparser-3.5.0-.../lib/python2.7/site-packages/configparser-3.5.0-py2.7-nspkg.pth

If configparser's directory is added to sys.path via site.addsitedir() then it can be imported.

Here's an example of loading it:

(alice)[15:49:42]spack-plain>>module purge
(alice)[15:49:44]spack-plain>>module load py-configparser
(alice)[15:49:48]spack-plain>>printenv PYTHONPATH
/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages
(alice)[15:49:49]spack-plain>>module purge
(alice)[15:49:51]spack-plain>>python
Python 2.7.15 (default, May  1 2018, 16:44:08)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> site.addsitedir("/Users/hartzell/tmp/spack-plain/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt/lib/python2.7/site-packages")
>>> import configparser
>>>

You can't use addsitedir() reliably

I have a Spack PR that writes a package's dependencies to a file and provides a sitecustomize.py that reads that file when a script starts up and loads the dirs via site.addsitedir(). It allows e.g. flake8 to work without any environment variables.

Sadly, other packages don't work. E.g. py-jupyter-notebook.

It turns out that there are two different ways to create namespace packages in Python2 (plus a different one in Python3): pkgutil-style and pkg_resources-style.

Details here

You cannot mix packages that use the different techniques.

There's a backports package that does nothing other than claim the namespace. It's pkgutil-style.
Many (most?) "backports" packages, e.g. (backports-shutil-get-terminal-size) are pkgutil-style.
configparser is pkg_resources-style.

SIGH.

In an addsitedir() system...

Here are some demos, using the system in my PR.

If you're not familiar with the PR, scripts figure out the directory that they started in and look for a file named ../.spack/python-sitedirs. If it exists, they pass each line in it to site.addsitedir.

When configparser's directory has been added via addsitedir, the backports modules appears to be "built-in":

(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:17:20]bin>>cat foo
#!/bin/bash /Users/hartzell/tmp/spack-rpaths/bin/sbang
#!/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/python-2.7.15-fueghvuvuid6cw43s2eqovgyauoc73ii/bin/python2.7

import backports

print(backports)
(alice)[16:17:20]bin>>$(pwd)/foo
<module 'backports' (built-in)>

But if you don't add the configparser dir, then the backports module points to the directory that contains it.

(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:15:44]bin>>$(pwd)/foo
<module 'backports' from '/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o/lib/python2.7/site-packages/backports/__init__.pyc'>

Here's how that plays out with something that's actually trying to import a real piece of code.

(alice)[16:13:13]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-configparser-3.5.0-hcll7pv4hkl46st6vzswyrcedpq6r7vt
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:13:22]bin>>$(pwd)/demo
Traceback (most recent call last):
  File "/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-jupyter-notebook-4.2.3-l4hlvjv7ce64xantfcnfiga5jnr546wr/bin/demo", line 5, in <module>
    from backports.shutil_get_terminal_size import get_terminal_size as _get_terminal_size
ImportError: No module named shutil_get_terminal_size
(alice)[16:13:27]bin>># remove configparser line from ../.spack/python-sitedirs
(alice)[16:13:39]bin>>cat ../.spack/python-sitedirs
/Users/hartzell/tmp/spack-rpaths/opt/spack/darwin-highsierra-x86_64/clang-9.1.0-apple/py-backports-shutil-get-terminal-size-1.0.0-smyswe3gg4khpsinkiflyrhz3ra4zs5o
(alice)[16:13:43]bin>>$(pwd)/demo
(alice)[16:13:45]bin>>

hartzell/configparser-is-broken.md

Select an option

No results found