Skip to content

Instantly share code, notes, and snippets.

@gregberns
Last active December 28, 2019 06:53
Show Gist options
  • Save gregberns/4a3f4d28080f94d41739853f50e3f216 to your computer and use it in GitHub Desktop.
Save gregberns/4a3f4d28080f94d41739853f50e3f216 to your computer and use it in GitHub Desktop.
UniVerse Change Data Capture File Format

UniVerse Change Data Capture File Format

Below is a proposed data format to send UniVerse database changes via file. It uses standard UNIX delimiters for new lines, and standard UniVerse characters to deliniate fields (ASCII code 254 (þ)) and data (ASCII code 253 (ý)).

Example:

src=cssþns=CONTRACTSþts_ms=1550549470126þuid=12345678þop=uþdata=ý_id=1234ýname=Bobýþ
src=cssþns=CONTRACTSþts_ms=1550549470126þuid=12345678þop=uþdata=ý_id=1234ýname=Simonýþ

Technical Notes:

  • All lines are ended in ASCII code 10 (LF), including the last line
  • The file is ended with an EOF (handled by the file system)

Change Set

A 'change set' is zero or more 'row events', where each row event describes , which describe a series of row changes to one or more 'files' in UniVerse.

Delimiter: ASCII code 10 (LF)

Row Event

Metadata and data representing the change that has occured.

Delimiter: ASCII code 254 (þ)

  • Source (src) - indicator that the data is coming from UniVerse (Similar to a Database name)
  • Namespace (ns) - location from within the source where data is coming from (UniVerse File name)
  • Timestamp (ts_ms) - time when row changed (from system clock) (Milliseconds since Epoch)
  • Incrementing Id (uid) - unique, incrementing Id that uniquely represents the row trigger event
  • Operation (op) - representation of the type of change to the row (Read, Create, Update, Delete)
  • Data (data) - set of key/values representing the data that has changed
src=cssþns=CONTRACTSþts_ms=1550549470126þuid=12345678þop=uþdata=ý_id=1234ýname=Bobýþ

Operation

Format: "op={operation_type}"

Examples: op=r, op=c, op=u, op=d

  • Read (op=r) - An initial
  • Create (op=c) - An insert into the table. A new entity/row has been created.
  • Update (op=u) -
  • Delete (op=d)

Data

Contains the data that has changed. Should generally contain the row's Primary Key with _id

The key/values should be surrounded, both start and end, with the delimiter.

Format: "ýkey1=value1ýkey2=value2ý"

Condition: key must not contain '='

Delimiter: ASCII code 253 (ý)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment