This document will report on everything that was found while researching how I see thunderkit's role in Bug Fables modding moving forward. The short answer is it will become essential if at the very least due to the assets management, but it most likely will be for everything in general. This document isn't a guide, it's a technical report of everything that I observed while researching tk which is why it doesn't appear in my docs repos: it's reports of findings, but not documentation because there's more R&D to do. Think of them as "notes", but shared so people don't refind everything I spent so much to find.
https://github.com/PassivePicasso/ThunderKit
Thunderkit is best described as a "meta" IDE for unity modding. It's a unity package that provides specialised editor scripts and assets for the sole purpose of simplifying the process of developping, building and distributing a mod. It does not do anything you can't do with other methods, but the major selling point is it either simplifies, trivialise or at least offer a way to do so p much any day to day task when it comes to modding. This is why I call it a "meta" IDE: it turns the Unity editor into a modding IDE with tools you would typically use for game dev, but specialised for modding.
The advantages are numerous:
- greatly simplifies the build of asset bundles
- reduces the hassles of copying files around in specific places to test mods
- the mod is a unity project: it can be version controlled a lot better including both assets and code instead of just the code
- since you are using the same tool the game was originally built on, it allows to have the closest sets of capabilities as the game reducing the risks of incompatibility
- etc...
So it's no wonder why I have been interested since a while.
My goals was trying to get the following setup:
- build a mod with one click containing a simple code patch and another involving an asset bundle load
- the runtime should be upgraded with doorstop 4 and a custom version of bepinex 5 to support it as doorstop 4 is by far the best way to debug the game and it reduces the amount of files to overwrite when upgrading the runtime
- everything must be done in one click: I should be able to rebuild the entire mod from scratch after performing a single change
- The setup should be possible on both windows and linux
- document the process to get there
I can say this was a success for all of these, but there's several caveats I got into along the way, but fortunately, none of them are showstoppers.
Unity 2018.4.12f1 MUST be used as the unity version. This is the same version the game ships with and version upgrades isn't really commonplace in unity modding for many reasons (mainly, lots of incompatibilities, there's a reason there's version checks everywhere in the assets).
Other than that, that's actually all that's needed here other than your favorite coding IDE (visual studio and rider are the best with the latter being paid, but works on linux).
I initially tried using the vanilla version linked above, but that changed for a reason I will explain later. For now, just remember that the readme file mentions a couple ways to install, but because unity 2018.4 doesn't support installing froma git URL using the gui, it MUST be installed using a git URL. To note, relative paths are NOT supported, but specifying a tag or branch is. This is going to be important later. Use the package.json method from the readme to install it.
The docs doesn't mention this, but specifically for this unity version, it's not enough. You will get 2 errors about missing packages. You resolve them by installing the addressable package and the performance testing api package (note: the latter is a preview meaning the option to view preview package MUST be enabled to see it). From there, tk should be installed.
This packaging thing is annoying because it means for some odd reasons, tk isn't pulling the dependencies. I should try to make it pull or at least just make it clear this is required to proceed.
During testing, I found a very problematic issue exclusively affecting the linux version of the editor: if you press ctrl following by any letter while on an editor text field, it will type a control character whose value depends on the letter you pressed. Because they aren't printable YOU WILL NOT BE ABLE TO SEE THEM IN THE EDITOR OR WHILE DEBUGGING THUNDERKIT! The only way to see them is by opening the asset file and seeing characters such as \x00. It is possible to delete them: if you press backspace and nothing happens, it means you deleted one. There are some exceptions to this (I can ctrl a without a control char), but generally, right clicking is safer.
I ran into an issue where the "get bitness" importer had to be turned off because for some reasons, thunderkit thinks it's not supported. The weird part is it might be mistaken on this, but it only affects the dev build conversion option which isn't required so if you are on linux, this needs to be turned off fow now (you can set the boolean at a later time anyway).
From there however, this is the usual process: in the settings, click browse and select your bug fables.exe file. Then click import and let it do its thing (accept every restart prompt if needed, this should take 2 unity restarts).
What does this process do? 2 things:
- It creates a thunderkit settings folder that has the settings you set and some others thunderkit determined. You can change these values, but for the most part, their default are fine except if you are one linux, you may need to set the is 64 bit thing correctly if you disabled the get bitness
- It installs a hidden package called Bug Fables which is actually just a made up package thunderkit made in your project by scanning the games files. It contains all plugins (the steam dll here) and all game assemblies (the game's code and steamworks.NET here). This will allow unity to see the Assembly-CSharp.dll which is the only thing we care about. NOTE: THIS MEANS YOU NEED TO GITIGNORE THIS TO NEVER COMMIT THESE FILES!. They're just in your packages folder so git WILL see them unless you gitignore them.
From there, that's the simple part, let's talk about how you actually would configure a mod project.
The entirety of thunderkit boils down to 3 custom assets type: manifests, pipelines and pathReferences. Everything surrounding tk involves one or more of the three.
This one's p simple, but important: it's a smart way to refers to filesystem paths. Why is that needed? Well sometimes, it's not clear the exact path of things. For example: thunderkit comes with some path references that talks about the game's directory, but that in theory could have come from anywhere and it has to account for games having different names and such. While they could just store it (which they do), that alone isn't modular: what if you need to do stuff in some bepinex config or you need to do special stuff?
That's where this comes in: you define a path as a series of parts. It could be something as simple as a constant directory name to append or as complex as the game executable, thunderkit offers lots of options ootb here. Most of the ones thunderkit ships with covers most cases. For example, it comes with a manifest staging path which is just a folder at the root of the project used specifically to hold artifacts.
A big thing with them is they are shared accross the project and are indexable directly in the editor. This means you don't drag and drop a path refference asset: you put its name surrounded by angled brackets . The reason it does that is because it allows you to specify a hardcoded part and a dynamic part like /somethingFun. It does mean that typos and mistakes are possible, but thunderkit will break if you do so it's not a huge deal.
Some paths are unfortunately not useful, but we'll cover them later. The important point here is ANY PATH WHATSOEVER IS A PATH REFERENCE! This means that you have flexibility: you can hardcode an absolute path or involve some components thunderkit has or even create your own resolver!
A manifest is sort of like the csproj of thunderkit: it declares basic informations about the project and how pipelines needs to deal with it. The former is simple: it's just name, author, description, version and potential dependencies. We'll talk about how this is supposed to be used, but as far as actual modding build goes, only the name matters (but it's a good practice to set all of them just in case ig).
The latter is where it gets a bit complicated. Basically, a manifest is composed of so called "datum" (this is supposed to be the singular form of data...but it's not used well here, whatever). A datum is a piece of data that dictate how the correspoinding pipeline job related to the datum deals with the mod.
The identity and each datum all take an array of so called staging paths. It's basically "where to put the artifacts?" so the pipelines knows where things are after building.
Let's talk about the different types of datum:
- Assembly definitions: this tells thunderkit to build assemblies. You provide a list of assembly defs (basically a file that tells unity to build a dll with some options) and the path to place them once done. This is paired to the stage assemblies pipeline job.
- Assets bundle definitions: this tells thunderkit to build asset bundles. You provide a list of bundles to build and to place them where. A bundle is composed of a name and a list of assets. NOTE: THE NAME MUST BE UNIQUE ACCROSS ALL BUNDLES EVERY MODS WILL LOAD! This means it might be advantageous to make a custom version that prefixes the name with the manifest name. Also to note: while thunderkit says it needs a list of "assets", there's nothing stopping you from just putting a folder and it will be smart enough to take the assets inside it.
- Files: It just copies files, very similar to the other 2 datum with the diffences that you copy assets files and you have the option to include the .meta one. Could be useful.
- Thunderstore data: It just tells to make a manifest.json compatible with the thunderstore packaging system. More on that later.
- Unitypackage: It's possible to not build a mod, but a redistributable thunderstore package others can install as dependencies. This allows to specify a list of assets packages to include. More on thunderstore later.
That covers everything about manifest.
Pipelines are composed of jobs and they are where manifest and paths comes together. They define the actual work to perform and it's basically an orchestration system. You specify the list of jobs (or what you can call tasks) and once the pipelines executes, tk will execute the jobs in order.
It's a good time to mention that pipelines and manifests have a special toolbar area dedicated to quickly executing them to the right of the play, pause and step buttons. They can only be selected once the quick access checkbox is checked on them. This allows you to quickly see the logs and execute your common pipelines and manifests.
So if the pipelines contains the jobs, what are jobs? They define what to do and if needed, it will use the information you specified in a manifest datum to do so.
There's a lot that comes by default, but it's possible to create your own. Here's a non exhaustive list to give an idea:
- Copy: copy files from source to dest
- Delete: delte files
- stage assemblies: build the assemblies specified by the corresponding datum and place them at the staging path also specified in the datum
- stage asset bundles: same deal, but with the asset bundles definitions
- stage manifest files: same, but with the manifest
- execute process: what it sounds like, you can execute whatever with custom arguments
- execute pipeline: this orchestration system supports nesting: you can call another pipeline as a job
- etc...
All in all, you can see how much this allows: with one click, it's possible to build entire modding infrastructure and setup to launch the game automatically with configs to use your mods. This is where the main value of thunderkit lies.
This is where it's going to get intense because I ran into couples of issues and kinks relating to what thunderkit comes with and is useful vs what is broken.
The package comes with so called "templates" that is just packing a bunch of manifest, path refs and piepelines. They are of varying usefulness concerning the bepinex template, but some are worth to mention:
- The default manifest is a p good template for most simple mods
- The jobs are mixed: the ones that are useful are stage, deploy, bepinex launch and rebuild + launch, but they only work with everything vanilla which we will deviate from (pls note the steam launch one is designed to only work on windows because it checks the registry).
- The path refferences while good in theory, are only useful in a vanilla bepinex setup which we will deviate from and this causes some problems, particularly with the bepinex pack source path
It's possible to setup a basic project using ONLY assets from the tk package, but there's problems with it:
- It ONLY supports vanilla bepinex. This means no doorstop 4, no clean runtime upgrade and stuff. More on why later.
- if you use harmony (vanilla or harmonyx that comes with bepinex), it's straight up broken when trying to build.
Because of this, I mostly based off works from the vanilla pipelines, but ended up editing them.
That second issue however is the worst one I encountered so let's talk about it.
The stage assemblies pipeline has an issue with unity 2018.4: it will include every assemblies it can detect, even the ones you didn't specified in any assembly definitions. The problem is bepinex ships with 2 harmony verions with similar types and namespaces so unity won't compile ANY vanilla harmony plugins because it will type conflicts. You can use monomod instead, but I consider this a problem and it's not something I should make an extension for because it's straight up borked.
Solving this involves setting the exclusion lists to all the assemblies we never said to want and that no assemblies decided to want. I made a branch on my fork of thunderkit with the fix and installed that one instead (reminder: you can install a git repos of anyone if it's a unity package of any branches). I did tried to upstream it here: PassivePicasso/ThunderKit#98 but it's not likely to go through because it takes extensive testings to make sure this works accross versions. I could do such test, but idk if it's that important vs just hosting a fork for now with the fix: you can't change the unity version anyway.
After this fix, I was able to build. I didn't spotted any issues when building a simple asset bundle.
The deploy part however...this takes lots of explanations
Thunderstore is a large modding platform that hosts a sort of custom nuget like things that work well with thunderkit (website: https://thunderstore.io/ ). It hosts many communities and they are open to request creating communities if one can prove there's at least one modder that will bring everything together.
The problem is I am looking into the future and I don't think we are close to get here: I would like to establish some foundations before I can request them to create a community. This means at least at the start, I don't think things will be on thunderstore, but nothing prevents me to move my stuff to it later.
Why do I mention this with thunderkit? Because from my understandings, the primary usecase of tk is to use and make thunderstore packages. While it doesn't force you, it's VERY strongly integrated to the point that not using it means I have to come up with some creative ways to make things work (with asset packages and extensions mainly).
It also goes into how a vanilla bepinex setup is supposed to work in thunderkit: there's a bepinex package (it's missnamed to be for Kerbal space program 2, that is not the case, it's universal) that you can install from a default setup. It includes bepinex 5...and that's all there is to it!
The thing is, thunderkit ASSUMES you'll setup bepinex this way, but that locks you into using vanilla bepinex. Since we need to use a custom version for Doorstop 4...this is a problem: how can we use the vanilla pipelines with this in mind?
Well...it's possible, but it's ugly. See adding a thunderkit package does 2 things:
- Adds the files in your unity package folder making them read only, but copyable
- allows assembly defintions to see the dlls within
That second part is important: it allows thunderkit to see the bepinex dlls, but it keeps the package readonly which is very safe. The issue is it also means it's not possible to put your custom version easily...and because the only way to properly supply one is to publish a package to thunderstore which causes problems...well we're going to have to do it in more creative ways.
First off, the actual core dlls needs to be in the assets folder so that you can at least see them when making an assembly definitions. Since we need to put them there, we might as well just put the entire install, but make jobs to copy those to staging during the deploy. It could just be a simple asset package I host and have people download, there's lots of ways to go about it, but it needs to be revisited at a later time. For now, I just chose to NOT install the package, but keep the paths the same (remember that bepinex pack source? that assumes you have a bepinex folder under package). This allowed me to have a custom doorstop and bepinex version to deploy.
Finally, let's talk about launching
Surprisingly, if you have wine installed and are using vanilla bepinex, the pipeline should just work! I did had a lot of misshaps before realising this, but it was due to some unrelated issues. As long as you can type an exe name and it automatically opens it with wine, that's all you need for vanilla bepinex.
For our custom one though, it's only a matter of changing the doorstop arguments. This takes a bit of explanation because I learned a lot about how doorstop is configured.
Doorstop actually preserves the working directory which you can configure in the launch process job. This is handy because it means you can launch the game with doorstop enabled, but resolve files like bepinex and plugins in a completely different directory than the game! This is amazing because it leaves your game copy clean with the exception of winhttp.dll and the remnants of a runtime upgrade.
One thing that took me a while to learn though: you can either supply command lines arguments OR use the ini file, BUT ONLY ONE OF THE 2 IS ACTIVE AT A TIME AND THE CLI ARGS TAKE PRIORITY! This means if you use the launch process pipeline with arguments, you MUST specify ALL of them, even if you define them in the ini, doesn't matter, ALL arguments you want defined must be defined.
To get an upgraded runtime setup with the debugger, I ended up with this as list of arguments:
--doorstop-enabled true
--doorstop-target-assembly "<PWD>/<BepInEx.Preloader.dll>"
--doorstop-mono-dll-search-path-override "<GamePath>/MonoBleedingEdgeLibs"
--doorstop-mono-debug-enabled true
--doorstop-mono-debug-address "0.0.0.0:10000"
And it takes one more to tell it to suspend on boot until a debugger is attached!
This works, it's just cumbersome and it would be better to just offer an extension with a better UI than this like a doorstop launch job or something, it's messy.
Oh btw, the bepinex thunderkit package isn't COMPLETELY vanilla: it toggles the console and prevents its closing in the config. This means it also needs to be done for the custom version as having the console pop is really handy for checking logs.
AND WITH EVERYTHING ABOVE, I WAS FINALLY ABLE TO HAVE EVERYTHING WORKING!
But hold on, I kept mentioning the need to use doorstop 4 and upgrade the runtime despite the fact I tested both that and vanilla, why?
I saw this coming a while away, but it's time for me to explain why. This supplements the existing findings on upgrading the runtime for research here: https://github.com/aldelaro5/Bug-Fables-Internal-Docs/wiki/Upgrading-the-mono-runtime
It's best to explain what happens if I DO NOT upgrade the runtime, I keep bepinex vanilla and therefore, stay on doorstop 3. This already looses 2 BIG things:
- It's no longer possible to use thunderkit to build assemblies. Thunderkit uses unity's own runtime to build which make sense, it gives assemblies built by the same tooling, but the problem is thunderkit does not and will not support the older mono. This means it will build using the default net standard 2.0 profile on the newer mono and while the compile will suceed, it will throw at runtime because it's linking core libs that aren't present or incompatible with the game! This means it MANDATES the code to be built externally to unity and make pipelines jobs to deal with it. Doable, but the second point makes it not worth
- You loose the ability to debug the game comfortably because this limits your options to 2: dnspyex and converting the release build into a dev build. The former straight up is horrible ESPECIALLY if you debug SetText, DoAction or EventControl. It is straight up not usable. The later...works...with a ton of caveats. This is detailed in my wiki article I linked above, but it boils down to it's very messy (overwrites a lot of files), alters behaviors (the game can tell it's a dev build) and you are limited to visual studio on windows for debugging due to using an outdated symbols format for debugging the mod and debugging the game is even worse. Doorstop 4 remains by far the cleanest option to debug the game while upgrading the runtime.
All in all, it gives 2 choices:
- Don't upgrade, setup pipelines to build externally, but you have the inconvenience of having to manage 2 projects of different structures, you don't have as much capabilities (things in my experience can break such as LINQ) and you can't debug the game well
- Upgrade the runtime, setup pipelines that does this for you in a clean way, benefit from best debugging on the game or mods and have access to the fully capabilities of a proper unity 2018.4 tooling with predictable outcomes in .net standard 2.0
It seems p obvious what choice to make at this point. To be clear, IT HAS BEEN POSSIBLE TO DO MODS WITHOUT UPGRADING! The problem is it's not sustainable: it limits research, it limits debugging, it limits what you can do and honestly, thunderkit not working is the nail in the coffin for me: it simply doesn't make sense to keep using the old runtime. This is on top of the setup only overwritting a single word in a text file: it's very easy to revert back and when you do, you can entirely disable doorstop to run purely vanilla again with minimal steps.
Therefore, this is the path I will go moving forward. The actual process is simple: just copy files to staging and on shipping the mod, the runtime could be upgraded easily. The one complication is boot.config, but there's ways to handle it, it's not a huge issue.
With that in mind, that's p much everything there is to say with the exception of some stuff:
- I don't like how the manifest has its own version number when you have to specify it in the bepinex plugin thing. I might look into not having this double versioning system cause I hate it.
- Since I am going to fork thunderkit (unless the building issue gets fixed upstream which I have low hopes for), it allows me to do some interesting stuff to suit bug fables needs, but hopefully kept to a minimum
- I will likely provide additional extensions via an asset package or put everything in my fork, we'll see.
- There is a feature that I was told to be broken about converting the game into a dev build. While I don't think its messiness is worth for debugging, it is HEAVILLY beneficial for profiling because it is the only way to get the profilers of unity, rider and visual studio to work right. This has allowed me to uncover memory management issues and assets use issues so I would like to investigate ways to fix it upstream, we will see, I might bring it back in my fork even.
One last thing: if there's one thing I learned from this, it's Bug Fables is an incredebly unique game when it comes to modding. Thanks to the runtime issue, it's a unity 5.5.4 game disguised to be on 2018.4 which has not only caused tons of confusion when seeing the litteratures available with unity modding (because this is exceedling rare for that to happen), but the author of thunderkit themselves wrote this when I shared how I was debugging the game with recompiled game assemblies:
Tbh the fact that you have debugging at all for the main game assemblies is a step above everyone else afaik
They didn't even know you can setup mods without upgrading the game assemblies, just the runtime upgrade is enough. It's fair to say Bug Fables is unprecedented as a game when it comes to unity modding. In other words, I am on my own for this stuff: there's no litterature other than my own docs for Bug Fables that explains how this works.
With that in mind, that's all I had to say. As you can see, it's complex, but this R&D session makes me hopeful: I can build mods MUCH simpler than I could before and the only reason it takes a while is because it's a lot of kinks to iron out, but the beauty about tooling is you fix it once, you saved hours forever in the future so it's worth in the end :)