Björn Höhrmann hoehrmann

Frameworks like llama.cpp support context-free grammars to restrict the output of a large language model to a specific format.

The specification for the xml2rfc format comes with a RELAX NG schema that describes this particular format.

The RELAX NG specification defines its semantics based on a simpler format called the simple syntax. Some more advanced constructs are basically just syntactic sugar in this sense.

There are tools that convert the full format into the simple syntax.

The simple syntax is very easy to work for for all kinds of purposes.

Missing:

A few elements allow infinite nesting, that is not yet represented
Some TODOs are mentioned in the YAML files directly.
Where XML paths have more variables than the JSON paths, variables have to be reviewed; Usually that means the first [n] in the JSON path would become [n,o] (SDMX-ML 3.0.0 distributes artefacts in a way that requires aggregation in SDMX-JSON 2.0)

Note:

SDMX 2.0 and SDMX 2.1 are converted to SDMX-JSON 1.0
SDMX 3.0.0 is converted to SDMX-JSON 2.0.0.

	#!/usr/bin/env python
	from llama_index.core.ingestion import IngestionPipeline, IngestionCache
	from llama_index.core.query_pipeline import QueryPipeline
	from llama_index.core.storage.docstore import SimpleDocumentStore
	from llama_index.cli.rag import RagCLI
	from llama_index.core.node_parser import SentenceSplitter
	from llama_index.embeddings.huggingface import HuggingFaceEmbedding
	from llama_index_client import CloudChromaVectorStore
	from llama_index.vector_stores.chroma import ChromaVectorStore
	from llama_index.llms.openai import OpenAI

	#
	# In a JSON document, there is a unique normalized JSON Path for everything,
	# so you can write out a JSON document as a combination of paths and values:
	#
	# $.meta.id = "ID123"
	#
	# Given some JSON schema document, you can write out all possible paths. To
	# handle infinite nesting, part of the path could be described using a regular
	# expression.
	#

	import click
	import ijson
	import ijson.common
	import json
	import pathlib
	import functools

	def escape(s: str):
	esc = {
	"\u0000": "\\u0000",

	function process_command() {
	if [ "$$" -eq "$BASHPID" ]; then
	if [[ "$HANDLER_INSTALLED" -ne "1" ]]; then
	exec > >(
	trap "" INT TERM;
	awk '{ print strftime("%Y-%m-%d %H:%M:%S ", systime()) $0; fflush(stdout) }'
	)
	exec 2> >(
	trap "" INT TERM;
	awk '{ print strftime("%Y-%m-%d %H:%M:%S ", systime()) $0; fflush(stdout) }' >&2

	; FIXME: The grammar has been transformed so that `w` appears after a
	; token, but there is no way in ABNF to define it as token-separator
	; that can optionally contain a mix of comments and white-space. Take
	; `;;` as an example, for that to match `sql-stmt-list` `w` would
	; have to match the empty string. But if `w` matches the empty string
	; then `ISNOT` is the same as `IS NOT`.

	sql-stmt-list = [ sql-stmt ] *( ";" w [ sql-stmt ] )
	sql-stmt = [ "EXPLAIN" w [ "QUERY" w "PLAN" w ] ] ( alter-table-stmt / analyze-stmt / attach-stmt / begin-stmt / commit-stmt / create-index-stmt / create-table-stmt / create-trigger-stmt / create-view-stmt / create-virtual-table-stmt / delete-stmt / delete-stmt-limited / detach-stmt / drop-index-stmt / drop-table-stmt / drop-trigger-stmt / drop-view-stmt / insert-stmt / pragma-stmt / reindex-stmt / release-stmt / rollback-stmt / savepoint-stmt / select-stmt / update-stmt / update-stmt-limited / vacuum-stmt )
	alter-table-stmt = "ALTER" w "TABLE" w [ schema-name w "." w ] table-na

	#!/usr/bin/perl
	use strict;
	use warnings;
	use DBI;
	use JSON;
	use YAML::XS;

	my $dbh = DBI->connect('dbi:SQLite:dbname=:memory:');

	our $Arg;

	WITH
	bytes AS (
	SELECT 0x00 AS byte
	UNION ALL
	SELECT byte+1 FROM bytes WHERE byte < 0xFF
	),
	base AS (
	SELECT 0x0000 AS cp
	UNION ALL
	SELECT cp+1 FROM base WHERE cp < 0x10FFFF