PRD: Language-Agnostic Schema Navigation Service for Elastic Client Generators

PRD.md

PRD: Language-Agnostic Schema Navigation Service

Date: 2026-05-29
Status: Proposal
Author: Generated from research on all 8 Elastic client generators

---

1. Problem Statement

The Elasticsearch specification (schema.json) is consumed by 8 official code generators across 7 programming languages (TypeScript, Python, Ruby, PHP, Go, Java, C#, Rust). Every generator independently reimplements the same core infrastructure:

• Type lookup by name/namespace
• Inheritance and behavior traversal
• Availability/visibility filtering
• Variant and union classification
• Relationship navigation (endpoint → request → inherited properties → property types)

The duplication ranges from naïve O(n) linear scans (JS, Ruby, PHP) to full-scale graph engines (Java's bidirectional RDF RelationGraph, .NET's Typez type-system framework). This represents significant wasted effort and inconsistent behavior across generators.

Goal: Provide a single, language-agnostic tool that all generators can use to navigate the schema, eliminating duplicated infrastructure and making new generator development trivial.

---

2. Protocol Evaluation

2.1 GraphQL (Recommended)

Fit: The schema is a typed, cyclic graph with named nodes and typed edges. GraphQL is purpose-built for selective traversal of typed graphs.

Strength	Explanation
Selective field retrieval	A Ruby generator needing only endpoint names and HTTP methods fetches exactly that — no parsing of 300K lines
Typed relationship traversal	endpoint → request → allProperties → type → enumMembers in a single query
Union type dispatch	... on Request { path { name } } maps directly to the spec's tagged union of type kinds
Self-documenting	SDL + introspection = every generator author can explore the schema in GraphiQL without reading metamodel source
Mature ecosystem	Production-quality client libraries exist for all 7 target languages
No client-side graph logic	Computed fields (inheritance traversal, availability filtering) live in resolvers, not in each generator

Weakness	Mitigation
HTTP overhead for build-time CLI tools	All generators already fetch schema.json over HTTP; this replaces that fetch
Recursive types (ValueOf is self-referential)	Named union types in SDL handle cycles between named types cleanly
SDL maintenance	Generate SDL from TypeScript metamodel types; drift is caught at build time
Over-fetching impossible to prevent	Not relevant — this is a read-only, low-QPS development tool

2.2 Alternatives Considered

Alternative	Verdict	Reasoning
REST API	Rejected	Multiple round-trips for relationship traversal; no standard for selective fields; no union type dispatch; would reinvent GraphQL poorly
gRPC / Protocol Buffers	Rejected	Strong for high-throughput services, weak for graph traversal. No selective field fetching at nested depth. Requires generated client stubs in each language — more infrastructure, not less. Binary protocol makes debugging harder for a dev tool.
JSON-RPC	Rejected	No schema introspection, no type-safe selective queries, no union dispatch. Would require inventing a query language.
Flat JSON file with a helper library per language	Rejected	This is the status quo. 8 libraries is 8× the maintenance. The Java team already concluded this wasn't sufficient and built a graph engine.
Language Server Protocol (LSP)	Rejected	Designed for IDE interactions (completions, hover, go-to-definition), not bulk data traversal. Wrong paradigm.
OData	Rejected	Heavier than GraphQL for this use case. Weaker ecosystem. Filter syntax is less natural for graph traversal.
Shared SQLite database	Interesting but rejected	Could work for indexed lookups, but relational model is awkward for recursive type traversal, union dispatch, and computed inheritance chains. No self-describing schema for exploration. Requires SQL in every generator.

2.3 Delivery Mode: HTTP Service with Embedded CLI

The tool should be deliverable as:

1. HTTP service (primary) — for cross-language consumption during builds
2. CLI mode — execute a .graphql query file against a local schema.json and print JSON to stdout, with no network required
3. In-process library — for TypeScript generators in this repo, bypassing HTTP entirely

This satisfies all deployment contexts: CI pipelines (HTTP service or CLI), local development (CLI), and same-repo generators (library import).

---

3. Recommended Architecture

┌─────────────────────────────────────────────────────┐
│                   spec-api/                          │
│                                                     │
│  ┌───────────┐    ┌──────────────┐    ┌──────────┐ │
│  │ SchemaIndex│───▶│ GraphQL      │───▶│ GraphQL  │ │
│  │ (in-memory │    │ Resolvers    │    │ Yoga     │ │
│  │  indexed   │    │ (Pothos      │    │ HTTP     │ │
│  │  model)    │    │  schema)     │    │ Server   │ │
│  └───────────┘    └──────────────┘    └──────────┘ │
│        │                  │                         │
│        │                  ▼                         │
│        │          ┌──────────────┐                  │
│        └─────────▶│ In-process   │                  │
│                   │ executor     │                  │
│                   │ (graphql-js) │                  │
│                   └──────────────┘                  │
│                          │                          │
│                          ▼                          │
│                   ┌──────────────┐                  │
│                   │ CLI runner   │                  │
│                   │ (stdin/file  │                  │
│                   │  → stdout)   │                  │
│                   └──────────────┘                  │
└─────────────────────────────────────────────────────┘

3.1 Layer 1: SchemaIndex

A typed, indexed in-memory representation of the schema. Equivalent to what Java's TypeRegistry, Python's Schema.types dict, and Rust's IndexedModel each provide independently.

Responsibilities:

• O(1) type lookup by {namespace, name}
• O(1) endpoint lookup by name
• Precomputed inheritance chains (all properties including ancestors and behaviors)
• Availability/visibility filtering
• Topological ordering of types
• Bidirectional relationship index (inspired by Java's RelationGraph): given a type, find all endpoints that reference it; given an endpoint, resolve the full request/response chain

Key design decision: Like the Java generator's RelationGraph, the index is built once and locked. Immutability guarantees correctness across concurrent queries.

3.2 Layer 2: GraphQL Schema (Pothos)

A code-first GraphQL schema built with Pothos that exposes SchemaIndex through typed resolvers.

Why Pothos over SDL-first (e.g., graphql-tools makeExecutableSchema):

• TypeScript types and GraphQL types are defined in a single location — no drift between SDL and resolver signatures
• Plugin system (relay, dataloader, scope-auth) available if needed later
• Resolver code co-located with type definition — easier to navigate than separate SDL + resolver files
• No code generation step required (unlike graphql-codegen with SDL-first)

Key computed fields exposed by resolvers:

• Interface.allProperties — resolves full inheritance chain including behaviors
• Request.allQuery — includes CommonQueryParameters if attached
• Endpoint.request / Endpoint.response — resolves TypeName pointers to full TypeDefinition objects
• InstanceOf.resolvedType — resolves the TypeName reference to the actual TypeDefinition
• endpoints(visibility, stability, namespace, deprecated) — centralized filtering

3.3 Layer 3: HTTP Server (GraphQL Yoga)

A minimal HTTP server wrapping the schema for cross-language consumption.

Why GraphQL Yoga:

• Built-in GraphiQL explorer (zero-config interactive UI for generator authors)
• Framework-agnostic (works with Node.js http, no Express/Koa/Fastify dependency)
• Built-in response caching plugin (useful since schema is static)
• Minimal footprint (~50KB)
• Same maintainers as the broader Guild GraphQL ecosystem (Envelop, Codegen, etc.)
• Supports @stream and @defer directives if incremental delivery ever becomes relevant

3.4 Layer 4: CLI Mode

A CLI entry point that:

1. Loads a local schema.json file
2. Reads a .graphql query from a file path or stdin
3. Executes it in-process via graphql-js's graphql() function
4. Prints the JSON result to stdout

This enables generators that cannot or prefer not to depend on an HTTP service (air-gapped builds, latency-sensitive CI) to still benefit from the computed fields and filtering logic.

# Example usage
spec-api query --schema ./schema.json --query ./my-endpoints.graphql
spec-api serve --schema ./schema.json --port 4000

---

4. GraphQL Schema Design (Sketch)

type Query {
  endpoint(name: String!): Endpoint
  endpoints(
    visibility: Visibility
    stability: Stability
    namespace: String
    deprecated: Boolean
  ): [Endpoint!]!

  type(namespace: String!, name: String!): TypeDefinition
  types(kind: TypeKind, namespace: String): [TypeDefinition!]!

  # Utility queries
  namespaces: [String!]!
  topologicallySortedTypes(namespace: String): [TypeDefinition!]!
}

type Endpoint {
  name: String!
  namespace: String!
  description: String
  urls: [UrlEntry!]!
  request: Request
  response: Response
  availability: Availabilities
  requestBodyRequired: Boolean!
  deprecation: Deprecation
}

union TypeDefinition = Interface | Request | Response | Enum | TypeAlias

type Interface {
  name: TypeName!
  description: String
  properties: [Property!]!
  allProperties: [Property!]!       # Computed: full inheritance chain + behaviors
  inherits: Inherits
  behaviors: [Inherits!]
  generics: [TypeParameterDefinition!]
  specLocation: String!
  # Reverse lookups (inspired by Java's RelationGraph reverse queries)
  referencedBy: [TypeDefinition!]!  # Types that reference this interface
}

type Request {
  name: TypeName!
  description: String
  path: [Property!]!
  query: [Property!]!
  allQuery: [Property!]!            # Computed: includes CommonQueryParameters
  body: Body
  inherits: Inherits
  attachedBehaviors: [String!]
  generics: [TypeParameterDefinition!]
}

type Response {
  name: TypeName!
  description: String
  body: Body
}

type Enum {
  name: TypeName!
  description: String
  members: [EnumMember!]!
  isOpen: Boolean!
}

type TypeAlias {
  name: TypeName!
  description: String
  type: ValueOf!
  generics: [TypeParameterDefinition!]
}

union ValueOf = InstanceOf | ArrayOf | UnionOf | DictionaryOf | UserDefinedValue | LiteralValue

type InstanceOf {
  type: TypeName!
  resolvedType: TypeDefinition     # Computed: the actual TypeDefinition this points to
  generics: [ValueOf!]
}

type ArrayOf {
  value: ValueOf!
}

type UnionOf {
  items: [ValueOf!]!
}

type DictionaryOf {
  key: ValueOf!
  value: ValueOf!
  singleKey: Boolean!
}

union Body = ValueBody | PropertiesBody | NoBody

type ValueBody {
  value: ValueOf!
  codegenName: String
}

type PropertiesBody {
  properties: [Property!]!
}

type NoBody {
  _empty: Boolean  # GraphQL requires at least one field
}

type Property {
  name: String!
  codegenName: String
  type: ValueOf!
  required: Boolean!
  description: String
  deprecation: Deprecation
  availability: Availabilities
  serverDefault: JSON
  docUrl: String
  docId: String
}

# Enums for filtering
enum TypeKind { INTERFACE REQUEST RESPONSE ENUM TYPE_ALIAS }
enum Visibility { PUBLIC PRIVATE FEATURE_FLAG }
enum Stability { STABLE BETA EXPERIMENTAL }

# Scalars
scalar JSON

Key design decisions:

1. InstanceOf.resolvedType — this is the critical GraphQL win. One query traverses endpoint → request → property → type → resolvedType → ... → enumMembers without multiple round trips. This is what Java's RelationGraph provides in-process.

2. Interface.allProperties — a computed field that traverses the inheritance chain and merges behavior properties. Every generator currently reimplements this traversal.

3. Interface.referencedBy — reverse-edge query inspired by Java's findSources(). Answers "which types use this interface?" without full-scan.

4. Filtering at the Query level — endpoints(visibility: PUBLIC, stability: STABLE) centralizes logic that currently exists in 7+ separate implementations.

---

5. Technology Choices

Component	Library	Version	Justification
GraphQL schema builder	Pothos	^4.x	Code-first TypeScript; types and resolvers co-located; no SDL drift; excellent plugin system. Used by Shopify, Airbnb. GitHub
HTTP server	GraphQL Yoga	^5.x	Minimal, built-in GraphiQL, framework-agnostic, response caching plugin. GitHub
GraphQL execution	graphql-js	^16.x	Reference implementation; used by Pothos and Yoga internally; enables in-process execution without HTTP
CLI argument parsing	citty or built-in parseArgs	—	Minimal CLI framework; Node.js 18.3+ has util.parseArgs built-in, no dependency needed
TypeScript runtime	tsx	^4.x	Already used in this repo; zero-config TypeScript execution
Testing	Node.js built-in test runner	—	Already used in this repo (node --test); no additional dependency
Container	Docker (Alpine + Node.js)	—	For hosted deployment; ~50MB image for a stateless service loading a 4MB JSON file

Libraries explicitly NOT recommended:

Library	Reason for exclusion
Apollo Server	Heavier than Yoga; brings unnecessary middleware abstractions; telemetry/tracing overhead irrelevant for a dev tool
type-graphql	Decorator-based; requires reflect-metadata; less flexible than Pothos for complex union/interface schemas
Nexus	Effectively unmaintained since 2023; Pothos is the successor in the code-first space
graphql-tools (SDL-first)	Requires maintaining separate SDL files and resolver maps; drift risk; less type-safe
Express / Fastify	Unnecessary framework layer; Yoga runs on raw http.createServer()

---

6. Implementation Plan

Phase 1: SchemaIndex (Foundation) — ~2–3 days

Build the indexed in-memory model that all subsequent layers depend on.

Deliverables:

• spec-api/src/schema-index.ts — loads schema.json, builds:
  • Map<string, TypeDefinition> keyed by "namespace.name"
  • Map<string, Endpoint> keyed by endpoint name
  • Precomputed allProperties for every Interface and Request (inheritance + behaviors traversed once at build time)
  • Bidirectional relation index: typeToEndpoints, endpointToTypes
  • Availability filter: getVisibleEndpoints(visibility, stability)
  • Topological sort utility
• Tests validating O(1) lookup, correct inheritance traversal, filtering behavior

Verification: Unit tests pass; all existing types.find() patterns in the repo could be replaced by index calls.

Phase 2: GraphQL Schema + In-Process Executor — ~3–5 days

Build the Pothos schema with resolvers delegating to SchemaIndex, plus an in-process execution function.

Deliverables:

• spec-api/src/schema/ — Pothos type definitions and resolvers:
  • query.ts — root Query type with endpoint, endpoints, type, types
  • types/endpoint.ts, types/interface.ts, types/request.ts, etc.
  • types/value-of.ts — union type with InstanceOf, ArrayOf, etc.
  • types/body.ts — union type
• spec-api/src/execute.ts — function: (query: string, variables?: object) => Promise<ExecutionResult>
• Tests: snapshot tests of key queries (fetch all stable endpoints, resolve a type's full property chain, navigate endpoint → request → body → property types)

Verification: Can execute the following query in-process and get correct results:

query {
  endpoints(visibility: PUBLIC, stability: STABLE) {
    name
    request {
      allQuery { name type { ... on InstanceOf { resolvedType { ... on Enum { members { name } } } } } }
      body { ... on PropertiesBody { properties { name required } } }
    }
  }
}

Phase 3: HTTP Server + CLI — ~2–3 days

Wrap the schema in Yoga for HTTP serving, and add a CLI entry point.

Deliverables:

• spec-api/src/server.ts — Yoga HTTP server with GraphiQL at /graphql
• spec-api/src/cli.ts — CLI entry point:
  • spec-api serve --schema <path> [--port 4000]
  • spec-api query --schema <path> --query <file.graphql> [--variables <file.json>]
• Dockerfile — Alpine + Node.js, copies built JS, exposes port
• spec-api/package.json — bin entry for CLI

Verification: curl against running server returns correct data; CLI mode produces identical output to HTTP mode for the same query; Docker image builds and serves.

Phase 4: Documentation + Adoption Guide — ~1–2 days

Deliverables:

• spec-api/README.md — setup, usage, example queries for common generator patterns
• spec-api/examples/ — .graphql query files demonstrating:
  • "Get all endpoint names and HTTP methods" (simplest possible generator)
  • "Get full property trees for a namespace" (typical generator need)
  • "Find all types that inherit from a given interface" (reverse query)
  • "Get topologically sorted types for a namespace" (forward-reference ordering)
• Migration guide for each existing generator showing before/after

Verification: A generator author unfamiliar with the tool can run it locally and execute example queries within 5 minutes.

---

7. Total Effort Estimate

Phase	Effort	Cumulative	Value unlocked
1. SchemaIndex	2–3 days	2–3 days	Fixes O(n) scans in JS generators immediately
2. GraphQL Schema	3–5 days	5–8 days	In-process queries; SDL published for all teams
3. HTTP + CLI	2–3 days	7–11 days	Full cross-language service operational
4. Docs + Adoption	1–2 days	8–13 days	Other teams can start adopting

POC (Phases 1–2): ~1–1.5 weeks for one engineer. Delivers a working in-process GraphQL executor with computed fields.

MVP (Phases 1–4): ~2–3 weeks for one engineer. Delivers a Docker-deployable HTTP service + CLI + documentation that any of the 8 generators can adopt.

---

8. Estimated Code Volume

Component	Lines of TypeScript (approx.)
SchemaIndex	~200–300
Pothos schema definitions + resolvers	~400–600
HTTP server (Yoga wrapper)	~30–50
CLI entry point	~50–80
Tests	~300–500
Total	~1,000–1,500

This is deliberately small. The complexity lives in SchemaIndex (correct inheritance traversal, bidirectional index construction); the GraphQL layer is a thin typed wrapper.

---

9. How Each Generator Would Consume This

Generator	Language	Recommended client	Migration path
JS	TypeScript	In-process execute() (no HTTP)	Replace types.find() calls with GraphQL queries or direct SchemaIndex import
Python	Python	gql or httpx	Replace schema.types[name] dict with queries; gain computed allProperties
Ruby	Ruby	graphql-client	Replace O(n) .find calls entirely; no local schema parsing needed
PHP	PHP	webonyx/graphql-php client or plain HTTP	Eliminates dual rest-api-spec/schema.json source split
Go	Go	hasura/go-graphql-client	Replace grouping logic with namespace-filtered queries
Java	Java	graphql-java or HTTP client	Could supplement or replace RelationGraph; team may prefer existing infra
.NET	C#	GraphQL.Client	Could supplement Typez; most benefit in validation/exploration
Rust	Rust	cynic or graphql_client	Replace IndexedModel with queries; gain computed fields not yet implemented

Java and .NET note: These generators have the most sophisticated existing infrastructure (RelationGraph and Typez respectively). They may not have immediate motivation to migrate. The service is most immediately valuable to the 6 generators without graph engines — and to any future generators, which could start generating code in hours rather than weeks.

---

10. Risks and Mitigations

Risk	Likelihood	Impact	Mitigation
Non-JS teams don't adopt	Medium	Reduces value by ~50%	CLI mode requires zero client library; plain HTTP + JSON works with curl/fetch. Make adoption frictionless.
Schema metamodel evolves	High	SDL must track	Generate SDL validation tests from metamodel TypeScript types. CI breaks if they diverge.
Recursive ValueOf causes infinite query depth	Low	Performance/stack overflow	Set max query depth (e.g., 10) in Yoga config. Spec's actual max nesting is ~5 levels.
Service availability becomes a build blocker	Medium	CI failures if service is down	CLI mode as fallback; publish Docker image for local/self-hosted use; cache responses in CI.
Performance with full schema (~3,757 types, 588 endpoints)	Low	Slow queries	Schema fits in memory (~4MB parsed). All lookups are hash-map O(1). Startup is <1s.
GraphQL client library friction in some languages	Low	Slows adoption	Every language with an HTTP client can use plain POST with a JSON body — no GraphQL-specific library required.

---

11. Success Criteria

The tool is successful if:

1. At least 3 generators adopt it within 6 months of MVP
2. New generator bootstrap time drops from "weeks of infrastructure" to "hours of query writing"
3. O(n) scan patterns are eliminated from JS, Ruby, and PHP generators
4. Computed fields (allProperties, allQuery, availability filtering) are consistent across all consumers — no more behavioral drift between generators
5. Generator authors can explore the spec interactively via GraphiQL without reading metamodel TypeScript source

---

12. Future Extensions (Post-MVP, Not In Scope)

• Schema diff queries — compare two spec versions, surface added/removed/changed endpoints and types
• Code generation hints — language-specific annotations (e.g., "this property should be Optional in Python, nullable in C#")
• Persisted queries — pre-register common query patterns for even faster CI execution
• WebSocket subscriptions — notify generators when a new spec version is published (relevant only for a hosted service)
• Mermaid diagram export — generate type-relationship diagrams (already exists in .NET's Typez.Extensions.Mermaid)

---

13. References

• Pothos GraphQL — code-first GraphQL schema builder for TypeScript
• GraphQL Yoga — lightweight GraphQL HTTP server
• graphql-js — reference GraphQL implementation for JavaScript
• W3C RDF — the graph model that inspired Java's RelationGraph
• GraphiQL — interactive GraphQL IDE (bundled with Yoga)
• elasticsearch-specification — source schema
• Java generator RelationGraph: elastic-client-generator-java/generator/src/main/java/.../graph/
• .NET generator Typez: elastic-client-generator-net/src/Elastic.ClientGenerator.Core/