Moose CLI code notes

moose-cli-notes.md

Main site

https://www.moosejs.com/ Docs: https://docs.moosejs.com/

create-moose-app

The create-moose-app is a NodeJS script that creates (seeds) the a moosejs application.

Issuing the following command creates an initial project folder.

npx create-moose-app my-moose-app

$ tree
├── app
│   ├── datamodels
│   ├── flows
│   └── insights
│       ├── charts
│       └── metrics
└── package.json

7 directories, 1 file

The package.json contains a key development dependency:

  "devDependencies": {
    "@514labs/moose-cli": "latest"
  }

When the user runs npm i or npm install the devDependency for moose-cli is pulled from NPM.

That results in the creation of two additional modules under the node_modules/@514labs directory branch: moose-cli and moose-cli-darwin-arm64.

$ tree
├── app
│   ├── datamodels
│   ├── flows
│   └── insights
│       ├── charts
│       └── metrics
├── node_modules
│   └── @514labs
│       ├── moose-cli
│       │   ├── LICENSE
│       │   ├── README.md
│       │   ├── dist
│       │   │   └── index.js
│       │   └── package.json
│       └── moose-cli-darwin-arm64
│           ├── bin
│           │   └── moose-cli
│           └── package.json
├── package-lock.json
└── package.json

The first dependency is @514labs/moose-cli which is executed by npx and runs the dist/index.js script. That script in-turn runs (spawns) the @514labs/moose-cli-darwin-arm64/moose-cli application.

It wasn't clear to me is how the @514labs/moose-cli-darwin-arm64 folder is created. After speaking with callicles I learned that the correct version is chosen from the list below using this line in the create-moose-app/src/index.ts script

require.resolve(
      `@514labs/moose-cli-${os}-${arch}/bin/moose-cli${extension}`,
    )

from a list of these optional dependencies defined in the package.json file.

  "optionalDependencies": {
    "@514labs/moose-cli-darwin-arm64": "0.3.83",
    "@514labs/moose-cli-darwin-x64": "0.3.83",
    "@514labs/moose-cli-linux-arm64": "0.3.83",
    "@514labs/moose-cli-linux-x64": "0.3.83"
  },

moose-cli dev

The moose-cli is a command-line executable with an entry point at main.rs. In main.rs a config is checked to determine whether the executable is built a development or production executable.

The Tokio crate is used to initialize a new multi-threaded runtime builder - which I suspect is a container for async functions. Per the source documentation it's necessary to avoid having main() be an async function because the initialization of sentry instrumentation needs to happen before Tokio takes over the main process thread.

Tokio blocks the current thread until the future returned by cli::cli_run() completes. This prevents the process main from terminating until other async calls are completed.

The cli_run function is defined in the cli.rs create and performs the following tasks upon start:

• setup_user_directory()
  • defined in settings.rs
  • obtains the user directory and then uses std::fs::create_dir_all() to create the parent folders leading to the full path provided.
• init_config_file()
  • defined in settings.rs
  • checks whether there is a config file path exists and if so creates a TOML file with initial hardcoded content.
• read_settings()
  • defined in settings.rs
  • calls config_path() to retrieve the path to the config file.
    • calls user_directory() to retrieve the path to the user's directory
      • calls home_dir() from home crate
        • https://crates.io/crates/home
        • Required because: "The definition of home_dir provided by the standard library is incorrect because it considers the HOME environment variable on Windows."
  • Uses the cargo config crate to support config related concerns
    • https://crates.io/crates/config
• setup_logging()
  • defined in settings.rs
  • Configures logging file and logging format
    • Defines a closure for handling incoming log messages
• Cli::parse()
  • Parses the command line arguments
• top_command_handler()
  • Top-level command dispatcher

cli_run()

In the cli_run function in cli.rs the above functions are used to:

• Determine the path location of the user's home directory
• Create an initial configuration file if one doesn't exists
• Use that path location to load the (config.toml) configuration file
• Setup logging using the path specified in config.logger
  • The Cargo log crate is used.
    • https://crates.io/crates/log
  • LoggerSettings are created in the logger.rs file.
  • A cli.log file is used to hold log messages
  • Uses the cargo fern crate for log entry formatting.
    • https://docs.rs/fern/latest/fern/
    • Uses a closure with the fern.format function to handle the output display / logging format for incoming log messages.
• The process command line arguments are parsed using Cli::parse()
  • The struct Cli in cli.rs is derived from clap::Parser which comes from the clap crate.
    • https://docs.rs/clap/latest/clap/
    • Clap is a command line argument parser
  • So Cli::Parse() invoked the derived Parse function from clap.
  • After parsing a cli is returned which contains a .command value.
• Both the config and the cli.command is passed to thetop_command_handler()
  • Execution is awaited until top_command_handler completes.
• top_command_handler()
  • First checks whether the settings (config) features.coming_soon_wall is not set to true.
    • If true then a message is displayed to the user saying coming soon and offers an invitation to join the MooseJS community.
  • The cli.rs crate references a Commands crate which maintains an enum with command variants: Init, Dev, Update, Stop, Clean
    • The Commands enum is derived from the clips::Subcommand trait.

The top_command_handler() functions as a command dispatcher and is inspired by the Rust command design pattern The implementation uses RoutineController defined in routines.rs to hold a Vec collection of routines. The RoutineController's impl defines an add_routine method for pushing routines to the routines collection and a run_routines method for iterating through the collection of routines an calling the routine.run() method on each.

The routines.rs module defines a Routine trait which can be used by command handlers to implement run_xxxx() methods for execution. This is where the meat of a command's operations are triggered.

Each command handler in top_command_handler() creates a RoutineController and called one or more controller.add_routine() methods before calling the controller.run_routines() method.

• Commands::Init:

  • Determines the path for a new project
  • If path is in user's home directory then an error message is displayed saying "You cannot create a project in your home directory" and the process exists.
  • A new InitalizeProject struct is created and passed into the controller.add_routine() and later executed via the controller.run_routines() method.
    • A new project struct is created using from_dir defined in the project.rs crate
  • The code operations required to implement a project initialization is stored in initialize.rs there the Route trait is implemented for the InitalizeProject struct.
  • Afterwards, the project file is initialized with default parameters from:
    • RedpandaConfig
    • ClickhouseConfig
    • LocalWebserverConfig
    • ConsoleConfig
  • The serde crate is used to serialize and deserialize Rust data structures.
    • https://docs.rs/serde/latest/serde/
    • As far as I can tell it seems that the JSON format and TOML format is used with serde serialization.
  • The Init method concludes by validating that the project repo path is indeed a Git repository.
    • This is accomplished using the git crate inside of the utilities folder.
      • is_git_repo
    • The repo's first commit is accomplished using the crate::utilities::git::create_init_commit function
    • Git functionality is made possible with the use of the git2 crate.
      • https://docs.rs/git2/latest/git2/

• Commands.Dev:

  • Attempts to load a project file that should have been created during the Commands.Init phase.
  • Panics if "No project found, please run moose init to create a project"
  • Otherwise proceeds to create a RoutineController and adds two routines for execution:
    • RunLocalInfrastructure
    • ValidateRedPandaCluster
  • The RunLocalInfrastructure struct is defined in start.rs
    • A Routine is implemented for it with a run_silent function.
    • A long list of steps are defined:
      • Create Internal Temp Directory Tree
      • Validate Mount Volumes
      • Create Docker Network
      • Validate Panda House Network
      • Run RedPanda Container
      • Validate RedPanda Run
      • Run Clickhouse Container
      • Validate Clickhouse Run
      • Run Console Container
      • Validate Console Run
    • If nothing panics then a success message is displayed the user stating that "Successfully ran local infrastructure"
    • Each of the above steps are implemented / marked with the Routine trait and implements a run_slient() handler.
  • The above routines are executed and lastly, the routines::start_development_mode method is invoked with a reference to the loaded project.
    • One of the first things that happens is that the initialize_project_state function is invoked with the directory of the project's schemas, a project struct and a mutable reference to a route_table.
    • The process_schemas_in_dir function recursively traverses the schema directory and uses the process_schema_file method to process actual files.
      • The process_schema_file function is defined in the schema.rs file.
      • process_objects() is utilized to create a topic in RedPanda and creates or replaces tables and kafka triggers which matches a table name.
      • The create_language_objects function is called to create a TypeScript interface.
        • This creates an interface that web developers can use to avoid having to create an interface for their data models.
        • The process_schema_file function continues to generate a TypeScript SDK. The SDK is then built using npm install, the package is then run, then the npm link --global command is executed to ensure the package is available for local dev use.
          • The process creates a symbolic link between a local package and a project. This is especially useful during development when you want to test changes to a package without having to publish it to npm first.
      • The information is used to create a route entry which will later be passed to and used by the webserver.

• Commands.Update:

  • TBD

• Commands.Stop:

  • TBD

• Commands.Clean:

  • TBD

Data flow from the api to clickhouse / RedPanda

{in progress}