To understand Tree-sitter using Zig, we start with the absolute atom of parsing: the Lexer.
In a traditional compiler, the Lexer (or Scanner) turns a string of characters into a stream of Tokens. In an incremental system like Tree-sitter, the lexer must be able to start at any byte offset, but for our first step, we will build a linear Lexer in Zig.
First, we define what our "Tokens" look like. Using a Zig enum is perfect for this because we can use tagged unions later for more complex metadata.