New post to publish

dyanmic_module_loading.rst

Loading python modules in run-time

Date:	2014-04-07 12:30
slug:	dynamic-module-loading
tags:	python, modules, import, programming
Author:	copyninja
summary:	Post describes about loading arbitrary python files or modules during runtime.

Some times it is desired to load arbitrary python files or pre installed python modules during application run time.I had encountered 2 such usecases, one is in SILPA application and other is dictionary-bot which I was refactoring recently.

Case 1: Loading installed python module

In case of SILPA I need to load pre-installed modules and here is the old code <https://github.com/Project-SILPA/Silpa-Flask/blob/master/core/modulehelper.py#L24>_ , that is a bit hacky code I copied from Active State Python recipies. I found a bit better way to do it using importlib module as shown below.

from __future__ import print_function
import sys
import importlib

def load_module(modulename):
    mod = None
    try:
        mod = importlib.import_module(modulename)
    except ImportError:
        print("Failed to load {module}".format(module=modulename),
                     file=sys.stderr)
    return mod

Here importlib itself takes care of checking if modulename is already loaded by checking sys.modules[modulename], if loaded it returns that value, otherwise it loads the module and sets it to sys.modules[modulename] before returning module itself.

Case 2: Loading python files from arbitrary location

In case of dictionary bot my requirement was bit different, I had some python files lying around in a directory, which I wanted to plug into the bot during run time and use them depending on some conditions. So basic structure which I was looking was as follows.

pluginloader.py plugins

|__ aplugin.py | |__ bplugin.py

pluginloader.py is the file which needs to load python files under plugins directory. This was again done using importlib module as shown below.

import os
import sys
import re
import importlib

def load_plugins():
    pysearchre = re.compile('.py$', re.IGNORECASE)
    pluginfiles = filter(pysearchre.search,
                           os.listdir(os.path.join(os.path.dirname(__file__),
                                                 'plugins')))
    form_module = lambda fp: '.' + os.path.splitext(fp)[0]
    plugins = map(form_module, pluginfiles)
    # import parent module / namespace
    importlib.import_module('plugins')
    modules = []
    for plugin in plugins:
             if not plugin.startswith('__'):
                 modules.append(importlib.import_module(plugin, package="plugins"))

    return modules

Above code first searches for all python file under specified directory and creates a relative module name from it. For eg. file aplugin.py will become .aplugin. Before loading modules itself we will load the parent module in our case plugins, this is because relative imports in python expects parent module to be already loaded. Finally for relative imports to work with importlib.import_module we need to specify parent module name in package argument. Note that we ignore files begining with __, or specifically we don't want to import __init__.py, this will be done when we import parent module.

The above code was inspired from a answer on StackOverflow, which uses imp module, I avoided imp because its been deprecated from Python 3.4 in favor of importlib module.