CSS ops

ops.md

CSS operations

CSS = Conjunction Streaming Service, which consumes orbital data (currently TLEs) and emits conjunction reports for objects on orbit (or crossing orbits).

Contact

Email contact: [email protected]. Please include "CSS" in the subject line.

Overview

Design

The service consists of a streaming index and a message bus. The message bus sends input to the streaming index, and the streaming index sends output to the message bus, which is an MQTT broker. The (demo) Web UI is served by an HTTP server in one (or more) of the streaming index processes (for now), and Javascript in the Web UI receives streaming index output direct from the message bus. Publishers can use either HTTP or MQTT to send input to the service.

This release is not organized for horizontal scaling, so the story here is pretty simple. A single css process runs the streaming index as well as a Web server, on port 8000, for the Web UI. That process uses the MQTT broker EMQX on TCP port 1883 on localhost (by default), and the bus streams the reports to the Web UI via MQTT over WebSockets on port 8083.

Communications

1. css runs an HTTP server on 8000. That port will receive traffic from the Internet but usually via an iptables redirect of port 80 (see below).
2. EMQX listens on several ports. The important ones are TCP 8083 (MQTT over WebSockets) and TCP 1883 (plain MQTT). The Web UI needs direct access to 8083, and the css process needs access to 1883. It's okay (and even preferable) for the Internet to have access to 1883 (in addition to 8083).
3. css talks to EMQX at 1883.
4. The Web UI talks to css at 8000 (probably via a redirect of 80).
5. The Web UI also talks to EMQX at 8083.

In summary, on a single-machine deployment, the following ports should be open to in-bound traffic from the Internet:

1. TCP 80 (assuming iptables redirect of 80 to 8000)
2. TCP 1883 (not currently used will be soon)
3. TCP 8083

The Web UI is at /ui/min.html.

Miscellaneous

The following installation instructions are basic. No process supervision, log rotation, metrics, etc.

css is written in Go, and it uses goroutines extensively. It's a single executable, but the files to serve the Web UI are separate. (We don't embed them in the executable now in order to facilitate quick, in-place updates of those files.)

Depending on EMQX's configuration in emqx_retainer.conf, disk use is either minimal or non-existent (except for nohup.out!). CSS is mostly an in-memory thing.

Installation and start-up

Summary

1. Redirect 80 to 8000.
2. Install EMQX.
3. Configure EMQX.
4. Start EMQX.
5. Install CSS.
6. Run CSS.
7. Load example data into CSS.

Each of these steps is covered below.

Preliminaries

Forward port 80 to 8000 with

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000

Install killall:

sudo yum install psmisc

EMQX

1. Install EMQX.

2. Configure EMQX:

   1. Replace /etc/emqx/acl.conf with etc/acl.conf from the CSS release.
   2. Replace /etc/emqx/plungins/emqx_retainer.conf with etc/emqx_retainer.conf from the CSS release.

3. Start EMQX with emqx start.

The command sudo emqx_ctl retainer clean will clear EMQX's retained messages state.

CSS without docker

1. Extract the release.
2. (cd build && bin/restart.sh)

CSS with docker

1. Extract the saved Docker image image.tgz from the CSS release.
2. Load the image: cat image.tgz | gzip -d | sudo docker load.
3. Run a container: docker run --name css --network=host css:1.0

CSS is a single (Go) executable, so Docker isn't really that helpful here.

Loading data

The release comes with some TLE data in the files data/r*.tle. You can load one of those files with

curl -X POST --data-binary "@data/r2000.tle" http://localhost:8000/tlesIn

Management

Restarting

The script bin/restart.sh will clear retained messages, restart css, and load some example TLEs.

Health, version, metrics

HTTP APIs:

1. /ui/version.txt reports git commit has and build time.

   This file is at httpfiles/version.txt.

2. /ui/changes.txt possibly outdated summary of recent changes.

3. /metrics returns JSON representing some metrics. Useful as a health check.

Web UI

It's at ui/min.html.

CSS configuration and capacity planning

CSS has a few important knobs:

1. -window, which is a positive integer, specifies the prediction range in seconds. This value is the main driver of resource consumption (both CPE and RAM).

   When new input arrives, it's propagated across the entire window (and subsequently at each tick of the clock). This initial propagation for a block of input will consume all CPU cores until the window has been processed. For example, if a block of 1,000 TLEs is published to the system, then 1,000 TLEs are propagated across the window immediately. This burst per block of (streaming) input is a phase of maximum CPU use, which is proportional to the number of inputs in a block multipled by the size of the window.

   At steady state, RAM use is proportional to the window size multiplied by the number of distinct objects (e.g., TLEs) tracted.

   As an example, a demo deployment with 1,000 TLEs and a 1200-second window has an RSS of about 1GB and has acceptable performance with two decent CPU cores.

2. -dist, which is a floating point number, specifies the maximum distance between to objects for a conjunction report. Reasonable values are in the range [2,5]. If lots of data is running through the system, then a smaller -dist will reduce output and declutter the current UI.

3. -speed, which is a floating point number, specifies the lower limit for the speed in a conjunction report. Reasonable values are in the range [0,5].

Complete css command-line options:

  -allowed-host string
    	For Letsencrypt (default "astriacss.morphism.com")
  -block int
    	pprof blocking sample rate
  -broker string
    	MQTT broker (default "localhost:1883")
  -cert
    	With Letsencrypt cert
  -client-id string
    	MQTT client id (default "service")
  -dist float
    	maximum distance in km (default 10)
  -http-files string
    	directory contain static HTTP files (default "httpfiles")
  -http-port string
    	HTTP service port (default ":8000")
  -in-prob float
    	the default probability for in-bound TLEs (default 1)
  -max-alt float
    	A TLE that props to higher than this altitude (km) is ignored (default 2000)
  -password string
    	MQTT password
  -pn int
    	partition count
  -prob float
    	minimum probability (default 0.5)
  -read-db-tles
    	read previous TLEs from database (default true)
  -rep-prop duration
    	GUI support: interval to propagate a report (default 5m0s)
  -speed float
    	minimum speed in km/s (default 1)
  -tles string
    	optional TLE file
  -user string
    	MQTT user (default "service")
  -v	show reports
  -window int
    	window in seconds (default 600)