Download the plugin repository & search for usages with ripgrep

instructions.md

Script to download the entire plugin repository :)

1. Create a new folder/directory and download the below script to it
2. Create two new folders, plugins and extracted (I couldn't be bothered to make the script check & create them itself)
3. Run the script (Python 3). It may not be the most efficient process, as it downloads & extracts one by one, but it only took around 5 mins for me. That's 5 mins waiting or the 40 mins I could have spent trying to make it a fast script...

Using ripgrep to search every plugin for a string

On Linux, I used ripgrep which is a very fast searching tool - it matches practically instantly against the whole repo.

https://github.com/BurntSushi/ripgrep

Installed with sudo apt install ripgrep (Debian/Ubuntu)

Switch to the 'extracted' folder

Ran with rg <pattern> -g <filename pattern>

eg. rg '.*get_json.*' -g '*.py'

And that will search through every Python file in every plugin :)

repository_download.py

import time

import requests
import zipfile

PLUGIN_REPO_URL = "https://plugins.octoprint.org/plugins.json"


def get_plugin_list():
    res = requests.get(PLUGIN_REPO_URL)
    return res.json()


def download_file(url, filename):
    # NOTE the stream=True parameter below
    with requests.get(url, stream=True) as r:
        r.raise_for_status()
        with open(f"plugins/{filename}", 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192):  # Why 8192? Just seen on the internet
                f.write(chunk)


def main():
    repo = get_plugin_list()

    total = len(repo)
    progress = 0
    errors = 0

    start = time.monotonic()

    for plugin in repo:
        if errors > 5:
            print("Too many errors, exiting")
            break

        progress += 1
        filename = plugin["id"] + ".zip"

        try:
            download_file(plugin["archive"], filename)
        except Exception as e:
            errors += 1
            print(f"Error downloading {filename}")
            print(e)

        print(f"Downloaded {filename} ({progress}/{total}, {round(progress/total*100, ndigits=2)}%)")

        try:

            with zipfile.ZipFile(f"plugins/{filename}", 'r') as zipref:
                zipref.extractall(f"extracted/{plugin['id']}")

        except Exception as e:
            print(f"Failed to extract plugin {plugin['id']}")
            print(e)
            continue

        print(f"Extracted {filename} ({progress}/{total}, {progress / total * 100}%)")

    end = time.monotonic()

    print(f"Downloaded {total} plugins in {round(end - start, ndigits=2)}s")


if __name__ == '__main__':
    main()