Proposed conventions for IDA configurations

Plugin-Configuration-Standards.md

The Problem

When writing and using IDA plugins, configurations tend to be quite a mess. With each plugin having it's own:

1. Color scheme
2. Hotkeys
3. Configuration file format
4. Configuration location

(And that's when you have a seprtate configuration, and not some variables in the plugin itself).

In the current situation, each developer works as he sees fit, and the users usually end up with the unmodified defaults, as well as a large number of .some-plugin-confuguration files.

The Solution

To solve this, we need to create standards for plugin configuration, as well as C and Python libraries that follow them.

Key Features

1. Configurations should be easily exportable and modifiable (a YAML file might be a good solution)
2. Should allow user/project based heirarcy:
3. Global configuration, stored in the IDA directory
4. User configuration, in the user directory
5. Per-directory config, stored in the same directory as the .idb itself
6. Per-IDB config, stored as net-nodes in the IDB itself.
7. Should allow global-defaults, and per-plugin modifications (like default color schemes for highlighting).

C&C Welcome

If you feel this is relevant to you, please comment so that we can improve on those ideas before going ahead an implementing them.

I propose using the Qt QSettings API to store global-, user-, and project-level configurations. Its been supported by Qt for quite a while now, and I like the fact that it was developed essentially for this purpose. It supports all the major platforms, using the appropriate mechanisms (on Windows the Registry, on Linux config files, I think). Global- and user-level configurations should be straightforward. Project-level configurations (especially via QSettings) I'll have to consider further. Seems like a useful idea.

Using netnodes for IDB-level configuration is appropriate (and under utilized!). I have a general purpose netnode library that may be useful here. The native API is a bit obtuse.

As a motivating example, hooking up the themeing of IDASkins to instruction highlighting (such as in ret-sync) would be pretty neat.

Is there a standard way of sharing or exporting QSettings configurations?

Not to my knowledge, though it would be trivial to implement using the allKeys() accessor (src).

I've seen some code snippets that have QSettings explicitly reading from .ini files. Perhaps these could be the export format, and at least the means to implement per-directory settings.

Well then, I'll take a look at QSettings myself as well. Never used it before.

I started sketching the implementation, and before I knew it, had an initial module. You can check it out here: https://github.com/williballenthin/ida-settings. Supports global, user, directory, and IDB settings; import and export; and an intuitive dict-like interface.

I'd be interested to hear what you think about it. I'm still happy to discuss other techniques, this is simply a proof of concept.

Finally had time to play with it for a bit. It looks really good. The API is intuitive, and the use QSettings does seem to make sense, making to code simpler.

I do have some issues with it:

1. Exposing the organization/application hierarchy of QSettings to the plugins prompts the generation of many registry keys. Personally, I'd prefer having all plugin-settings in one place.
   If we do wish to allow this hierarchy, we can use groups (organization/plugin). They can be made seamless in the API, yet provide hierarchy in both the registry and the .ini files.
2. I don't really like the fact that each plugin gets its own .config.ini file. This, too, can be solved by using the settings groups, but I am not sure if I prefer one file with all the settings in it, or multiple files.
3. It should be possible to list/export/remove the settings of all plugins, even without knowing which settings exist.
4. Plugins settings should allow for non-string values.

Other than those issues, I think the only requirement not addressed here is the global-defaults.

Thanks for the feedback. I think you make valid arguments about organization of the configuration hierarchy. From a cursory study of the Qt documentation, I think groups can be used to solve issues 1-3. If all IDAPython configuration settings fall underneath one organization/application instance, then the QSettings API will allow enumeration and access of all keys and values. This should be easy to implement.

If we take this route, we should agree on how much each script's settings are namespaced. Providing arbitrary group-tree traversal (for a given plugin, settings can have children groups, grandchildren groups, etc.) sounds pretty confusing, with limited benefit. Perhaps each plugin is expected to have a unique name, and then gets a flat key-value store to work with? A separate initiative might define common settings locations for shared items, such as interface color schemes.

For #2, how would you imagine the multiple files split up?

Finally, #4. The simple solutions would be using msgpack, json, or pickle. Pickle sounds dangerous, since configurations may be shared without trust, and you can pickle code. msgpack is fast and small, but requires an external library. json is ubiquitous, and would probably work. dumps() on the way in, and loads() on the way out. This would support strings, numbers, arrays, dicts. Thoughts?

as for the hierarchy, I think that a flat key-value store for each plugin is the best solution. Pseudo-groups can always be created by a plugin writer.

For #2, I think that splitting files up is only useful when exporting the settings, and you want to only export the settings for a given plugin. Other than that, I don't see a reason for multiple files.

Concerning #4, it seems that QSettings already provide an API to store different types of values (see QSettings::setValue).

Ok, #2 makes sense to me. Sounds like keeping all the settings under one organization/application QSettings instance with multiple subgroups should work. Export-by-plugin will be an API, with the .INI file format as the intermediate representation.
#4: Although the C++ QSettings class supports multiple types for value data, the Python layer transparently encodes/decodes QVariants. More importantly, the netnode API is picky about datatypes: you can only store bytes objects as values. So, we'll have to do some encoding/decoding anyways. Therefore, I propose we pick a format that always encodes to string/bytes, and we'll have the same behavior with every backend.

So, #4. I see no advantage to using msgpack, "fast-and-small" is not significant when talking about small and mostly-static data. So I think JSON would be preferable, adding no dependencies to the project.

I've updated the module to address issues #1-4.

You must now instantiate IDASettings instances with a single parameter that is the name of the plugin whose settings you'll fetch. All settings are placed in the same QSettings instance, under separate groups. This reduces the number of registry keys and config files created.

You can now fetch the list of names of plugins that have settings using the class properties IDASettings.system_plugin_names, IDASettings.user_plugin_names, etc. I don't think this is the most beautifully-named API, but it works. It allows you to enumerate all plugin settings instances, and from there, you can enumerate the settings key-value pairs themselves.

Values are now transparently JSON encoded and decoded as they are stored and fetched from the backends. This allows for values of types like: int, float, list, dict, etc. The netnode backend does not support values larger than 1024 bytes, so users are encouraged to store small values rather than large complex items.

I recommend bringing up additional issues specific to the module as github issues, so we both receive notifications. This gist remains a great place to discuss the general concept, however.

Regarding the 1024 bytes limit. I find it a bit problematic. The first plugin I wanted to use ida-settings is my plugin-loader. The loader currently holds a list of paths for all the plugins it loads. Paths are naturally long, so the data is easily over 1024 bytes. Any solution that resolves that on the plugin side is bound to be messy.

Any thoughts on that? I am leaning towards letting a JSON dump spread over several nodes.

Yes, I think this is a pretty annoying limitation.

Spreading the JSON over multiple netnodes is the same solution I came to a while back for another project. I have an existing library for netnode access that supports arbitrarily sized values, and handles the allocation of nodes. It would be a good fit for this project, and should solve the issue.

I'll spawn off another project to hold this code. You've provided some great feedback on ida-settings; would you mind giving the new project a review as well? I've enjoyed having a second pair of eyes note issues, suggest changes, and provide logical arguments.