Python: Implement basic plugin architecture with Python and importlib

plugin_architecture.md

Implementing a basic plugin architecture shouldn't be a complicated task. The solution described here is working but you still have to import every plugin (inheriting from the base class).

This is my solution:

Basic project structure

$ tree
.
├── main.py
└── plugins
    ├── __init__.py
    ├── plugin_a.py
    ├── plugin_b.py

The base plugin

$ cat plugins/__init__.py

import os
import traceback
from importlib import util


class Base:
    """Basic resource class. Concrete resources will inherit from this one
    """
    plugins = []

    # For every class that inherits from the current,
    # the class name will be added to plugins
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.plugins.append(cls)


# Small utility to automatically load modules
def load_module(path):
    name = os.path.split(path)[-1]
    spec = util.spec_from_file_location(name, path)
    module = util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module


# Get current path
path = os.path.abspath(__file__)
dirpath = os.path.dirname(path)

for fname in os.listdir(dirpath):
    # Load only "real modules"
    if not fname.startswith('.') and \
       not fname.startswith('__') and fname.endswith('.py'):
        try:
            load_module(os.path.join(dirpath, fname))
        except Exception:
            traceback.print_exc()

A sample plugin

$ cat plugins/plugin_a.py
import plugins


class PluginA(plugins.Base):

    def __init__(self):
        pass

    def start(self):
        print("Plugin A")

How to use it

$ cat main.py
from plugins import Base

if __name__ == '__main__':
    for p in Base.plugins:
        inst = p()
        inst.start()

Sample run

$ python main.py
Plugin B
Plugin A

This is absolutely fantastic. I've spent the last few hours reading various guides on Python plugin architecture, but this is by far the most straightforward, uncomplicated code I've seen that implements automatic, dynamic loading at runtime. Thanks for putting this up, really helped me in my own project.

I'm glad this helped somehow. However, also make sure you check out/read Eli's post on more fundamental concepts of plugin infrastructures. He has some really interesting thoughts there.

Haha, same for me! I've been trying to understand the plugin system for hours on end and find a good method to discover plugins. This is it.

thank you for code

Whoa, this is awesome!

An improvement / simplification from me:

    # Find the plugins folder depending on whether we're packed up using pyinstaller or not
    if getattr(sys, "frozen", False) and hasattr(sys, "_MEIPASS"):
        plugins_dir = Path(sys._MEIPASS) / "plugins"
    else:
        plugins_dir = Path("plugins")

    for pluginp in plugins_dir.glob("*.py"):
        pstem = pluginp.stem
        if pstem.startswith(".") or pstem.startswith("__"):
            continue
        spec = util.spec_from_file_location(pstem, pluginp)
        module = util.module_from_spec(spec)
        spec.loader.exec_module(module)

    for plugin_cls in PluginBase.plugins:
        plugin = plugin_cls()
        plugin.run()

Is there any way for a plugin to import a module or method in main project?

I'd like to expose some function in main application to plugin codes, but have no idea how to do it.

The main application may be compiled and distributed to user, then user can write their own plugin and call the function exposed by main application to add their specific feature.