0-readme.rst

Namespace package pth files

The generated-pth.py is what is created when installing a namespace package. These files are loaded on the auto-import of the site module to make sure Python knows where to find everything. There's a slight bug in this, however, that causes a problem. What happens if you have an __init__.py file in one of the directories along the way?

We ran into this in Armstrong creating our armstrong.apps.audio.backends.* packages. The code that's checks to see if there's an __init__.py in the directory, and if there is it doesn't pre-populate it with a fake module. The problem is that those modules might not actually be imported and loaded yet. See where this is going yet?

This code creates the following in sys.modules:

sys.modules["armstrong"] = types.ModuleType("armstrong")
sys.modules["armstrong"].apps = types.ModuleType("armstrong.apps")
sys.modules["armstrong.apps"] = sys.module["armstrong"].apps
# skip armstrong.apps.audio, we know how to load it
sys.modules["armstrong.apps.audio"].backends = types.ModuleType("armstrong.apps.audio.backends")
# NOW PANIC AND FREAK OUT!!

The hacky fix? Add an empty armstrong/apps/audios/backends/__init__.py, but that should not be necessary. The namespace_packages code is supposed to handle that type of thing for you. Instead, I think the file what-it-should-be.py is what should be generated.

So tell me, why not? I feel like I've missed something here. Well, other than Python shouldn't import things by default, but... it's already sorta doing that by pre-populating the sys.modules.

Note

This is a preliminary version of a blog post for TravisSwicegood.com. It's created under the CC by-at-nc.

generated-pth.py

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong',types.ModuleType('armstrong')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps',types.ModuleType('armstrong.apps')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr(sys.modules['armstrong'], 'apps', m)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps', 'audio')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps.audio',types.ModuleType('armstrong.apps.audio')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr(sys.modules['armstrong.apps'], 'audio', m)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps', 'audio', 'backends')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps.audio.backends',types.ModuleType('armstrong.apps.audio.backends')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr(sys.modules['armstrong.apps.audio'], 'backends', m)

what-it-should-be.py

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong',types.ModuleType('armstrong')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps',types.ModuleType('armstrong.apps')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr('armstrong' in sys.modules and sys.modules['armstrong'] or __import__('armstrong'), 'apps', m)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps', 'audio')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps.audio',types.ModuleType('armstrong.apps.audio')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr('armstrong.apps' in sys.modules and sys.modules['armstrong.apps'] or __import__('armstrong.apps'), 'audio', m)
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('armstrong', 'apps', 'audio', 'backends')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('armstrong.apps.audio.backends',types.ModuleType('armstrong.apps.audio.backends')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); m and setattr("armstrong.apps.audio" in sys.modules and sys.modules['armstrong.apps.audio'] or __import__("armstrong.apps.audio"), 'backends', m)

zzz.diff

diff --git setuptools/command/install_egg_info.py setuptools/command/install_egg_info.py
index dd95552..e0efe3b 100755
--- setuptools/command/install_egg_info.py
+++ setuptools/command/install_egg_info.py
@@ -92,9 +92,10 @@ class install_egg_info(Command):
                 pth = tuple(pkg.split('.'))
                 trailer = '\n'
                 if '.' in pkg:
+                    parent_name = '.'.join(pth[:-1])
                     trailer = (
-                        "; m and setattr(sys.modules[%r], %r, m)\n"
-                        % ('.'.join(pth[:-1]), pth[-1])
+                        "; m and setattr(%r in sys.modules and sys.modules[%r] or __import__(%r), %r, m)\n"
+                        % (parent_name, parent_name, parent_name, pth[-1])
                     )
                 f.write(
                     "import sys,types,os; "

3 years ago, this commit caused me to lose my afternoon.