ndng architecture.md

architecture.md

$ cat docs/architecture.md

Big Picture

We're building a next-gen network topology tool. It is a successor to Netdisco (https://github.com/netdisco) and inherits a lot of the concepts and features. Main Features:

• discover routers, switches and other networking gear (we call these entities Device)
• discover ports (entity Port)
• discover links between these components (entity Link)
• discover leaf elements like regular PCs, printers (entity Node)
• discover their connection to ports (entity CAMEntry)
• discover IP to MAC mappings (entity ARPEntry)
• discover hardware componentes inside Device, loosely based on SNMP Entity MIB
• the sum of the above entities is called Observations

All of this is to be stored in a database.

Significant required features that Netdisco does not have:

• the discovery layer is technology agnostic and pluggable. We provide both an API to store Observations and a real discovery framework that can use different methods and transports per device class.
• all Observations are stored with a timestamp, so it's possible to look at the network state at a certain time in the past
• Observations need a natural key in addition to the timestamp, so it can be observed on the time axis (e.g. router ip, node mac address)
• modern network virtualization should be considered, e.g. VRF, VLAN, VDOM, VXLAN etc.
• development is spec-driven and test-driven

Test-driven Method

Use containerlab with small, simple images to build a test harness network simulator. Verify all features and changes with this. Create a small discovery simulator that SSHs into a container lab device, performs "show arp" or similar, and post the result to the API.

Specification Driven Design

We use these phases:

• Specify: analyse or expand this document
• Gap Analysis: create .md files in docs/gap for each gap, remove them if something was implemented
• Develop: work on one gap at a time and implement it

Tech Stack

Postgres

• do not use relational integrity constraints because the observations can arrive out of order
• our database for everything

Python API

• evaluate best framework, should be something simple like Flask
• envision a Kubernetes style API: each observation is a type-checked YAML document
• allow for optional fields and default values
• provide an API explorer with Swagger or similar

Discovery

• build a real discovery framework, starting with SSH and containerlab-backed fixtures
• discovery methods should use hierarchical fallback, but be selected by an explicit resolver instead of hardcoding behavior into a dotted class tree
• a central netdisco3.yml config file maps addresses or ranges to vendor/product and credentials

Discovery Transports

Supported transport types:

• snmp - use PySNMP for polling standard MIB objects and vendor-specific OIDs
• rest - use requests
• ssh - use an interactive network automation library instead of one-shot ssh <host> <command>. The preferred choice is scrapli, because we need terminal interaction, prompt handling, and the ability to move through device-specific shells or privilege modes
• custom - the user supplies the code themselves

For parsing command output:

• use ntc-templates where coverage exists
• allow local TextFSM templates or custom parsers where ntc-templates does not cover a platform yet

The framework should assume that many network operating systems do not behave like a normal POSIX shell. Discovery code must support:

• interactive prompts
• paging disable commands such as terminal length 0
• entering CLI subsystems or enable/configuration modes when needed
• transport-specific error handling and retries

Test Harness Devices

The containerlab harness should include at least one real but lightweight NOS, not only mock Linux containers.

The first default lab device should be FRRouting:

• lightweight enough for contributors to run locally
• exposes a real routing CLI
• can be discovered via both SSH and SNMP
• good enough to validate the handler registry, parser flow, and normalization pipeline

Initial default implementations should include:

• generic Device discovery over SNMP using standard system OIDs
• FRRouting ARPEntry discovery over SSH using vtysh and show ip neigh

This gives us a mixed-transport discovery path in the default lab.

Hierarchical Discovery

Discovering entities should use hierarchical fallback. The important idea is correct: start generic, then become more specific when device identity and transport capabilities are known.

However, this should not be modeled as a rigid naming tree alone. Instead, we should build a discovery resolver that evaluates:

• entity type, such as ARPEntry or CAMEntry
• vendor identity, such as Cisco
• platform or operating system identity, such as NXOS
• product or family identity when useful
• supported transport, such as SSH or SNMP
• configured hints from netdisco3.yml
• discovered identity learned from the device itself

This is still a hierarchy, but transport is treated as a separate selector, not as a mandatory part of the primary identity path.

Conceptual examples:

• generic ARPEntry handler
• Cisco-specific ARPEntry handler
• Cisco NX-OS-specific ARPEntry handler
• SSH transport variant for a Cisco NX-OS ARPEntry handler
• SNMP transport variant for a generic Device handler

Examples of behavior:

• a Cisco device may use a Cisco-specific ARP command over SSH
• an NX-OS device may use a more specific ARP command such as show arp vrf all
• an unknown device may fall back to a generic handler or skip unsupported entities cleanly

Discovery Resolver

The discovery runtime should use a central resolver to choose the best handler.

Resolution precedence:

• use configured overrides from netdisco3.yml when present
• merge these with any identity discovered from the target itself
• choose the most specific compatible handler
• choose the best supported transport for that handler
• fall back toward more generic handlers when a specific one does not exist

This is preferable to treating the whole problem as a single dotted path such as <Entity>.<Vendor>.<Product>.<Transport>, because:

• vendor, platform, and product are related but not identical
• transport is a capability dimension, not purely an identity dimension
• resolution remains understandable even when some identity fields are unknown
• we can support both configured hints and discovered facts without forcing either to be the only source of truth

Every handler should declare:

• which entity it discovers
• which vendor, platform, or product identities it matches
• which transports it supports
• which credentials or access method it expects
• which normalized observations it emits

Every handler must normalize its output into timestamped Observations before persistence.

This lets us implement broad coverage early while still supporting targeted optimizations for specific platforms.

Contributor Extensibility

It should be easy for contributors to add their own Python classes.

The framework should provide:

• a small base handler class or interface
• a central registry or resolver
• a predictable module layout for built-in handlers
• support for loading custom handler classes supplied by users

Contributors should be able to add a handler by:

• writing a Python class
• declaring the entity, identity match, and transport it supports
• implementing normalization into Observation documents
• registering it with the resolver

Discovery Configuration

We use a central config file, netdisco3.yml.

This file should allow:

• specifying vendor, platform, or product hints for a single address or an IP range
• selecting credentials per device, following the ideas of Netdisco's deployment.yml device_auth configuration
• defining transport preferences such as SSH or SNMP
• providing discovery defaults that apply when a device-specific override is not present

The discovery runtime should use this config to answer:

• what device is this likely to be
• which credentials should be tried
• which transport should be attempted first
• which discovery handler should be selected

The config should provide hints, not immutable truth. If the device later identifies itself more accurately, the resolver should prefer discovered facts while still allowing explicit operator overrides where needed.