Ability to implement complex syntactic constructs that otherwise might not be possible in an arbitrary language.
- Ruby
Allowing runtime modification of the AST imposes several issues with optimization and reasoning. A non-exhaustive list includes:
- Being unable to satisfy that a specific bit of code is correct without having to inspect the whole program, including all libraries (including those dependent libraries of other libraries) and compilers used in its construction.
- Invaliding internal caches (monomorphic, polymorphic inline caches)
- Invalidating compiled JIT blocks of code
- Others...
The proposed solution is to instruct the compiler, at compile time—ex post facto, AST generation—allowing a set of transforms to be loaded into the compiler at this time, and to be applied to any subsequently loaded code, during its compile step. Languages like Ruby compile code not all ahead of time, but incrementally as code is loaded. As part of this invarient, those considerations are important for performance, amongst other concerns.
I don't believe that instructing the compiler at it's invocation time of a path to transforms is a convenient, or desirable solution, as this decouples the transforms from the places they will be applied, adding to an already large spike in cognitive load. However, this is an option. Instead, it's proposed that the transform be described in a metalanguage which might be normal ruby code, augmented with some API to receive AST nodes and act on them, as well as any helpers needed to make that transform make sense. Additionally, the transform must list any dependencies (external requires) so the compiler is able to bring these things into scope wherever this transform is applied.
As a consequence of these transforms being in separate files, we now have to, at our call site, instruct the compiler that it's desirable to use these modifications. This loading step requires a syntactic construct in the language that can be ignored by runtime. Its only use is at compile time. It also works in two parts:
- Search a table of transforms for the requested transform
- If found, apply the transform to this file, ex post facto, AST generation. If not found, go to Part 2.
- Freeze the syntax tree.
- Continue execution normally, with the modified syntax tree.
- Load the transform from disk
- Compile it as normal to some internal format which is capable of specifying the transform in the future
- Apply that internal representation to the now mutable syntax tree
- Freeze the syntax tree
- Continue execution as normal.
The transform table acts as a cache, and any cache optimization strategies that might be applied to caches, may be applied to this table.
It is expected that transforms will generate well formed ruby code.
Compile time errors must contain two pieces of information: The error in the transform, and the error as it relates to some arbitrary point in runtime ruby code. Both of these pieces of information are important for resolving any error or making sense of it.
Required errors:
- Invalid AST generated by transform
- Transform cannot be loaded from disk
This list is not meant to be exhausted, there may be other special case errors that are required.
All other errors, for instance, NameError that a transform might generate as a result of its AST manipulation, should also include the place in the transform as well as information about where the transform was applied. These errors are expected to be raised at runtime however, if the transform is correct, its output is correct, and the error occurs only when applying the transform to some arbitrary chunk of code.