Data Orchestration Decisions: Entities

data-orchestration-decisions.md

Data Orchestration Decisions: Entities

Problems we tried to solve with Entities

We need to migrate fragmented parts of code that handle business logic into a centralised, predictable place

Our reasoning behind that is that, at the time, we are writing logic based on our data structures in UI components. Thus no component can act as a smart or dumb component, since every component when needed interprets the data in its own way.

For example the XView filters and changes the shifts data as needed, and the shifts container does the same, etc.

We need to guarantee the consistency of our internal data structure and have an overview over it

The problem is, that our pure data structure, that is only determined by the backend, is being changed on the fly on the backend. So there is no source of truth, regarding what a structure might look like. Dependencies are resolved in different ways and as needed inside connectStore transformer-functions. The only central place where data structures are described is inside the serializers, but the structures often and easily change at other parts of the code, e.g. Actions, Containers, Components.

First Attempt at a Solution

Data Orchestration and Dependency Resolution

We will have a central API that handles dependencies and resolve them in a sensible and consitent way.

This ended up being tackled with Fluorine Orchestra. https://github.com/philpl/fluorine-orchestra

Data Structure consistency and Business logic

We will have entity-classes that:

• split up our data structures in logical units,
• contain filter and getter methods,
• are type-checked,
• are immutable,
• check dependencies

This pattern ended up including filter-logic and starting to look more like an OO-approach.

Problems we are now facing with Entities

We need to migrate fragmented parts of code that handle business logic into a centralised, predictable place

1. They break every part of the app that is relying on Immutable.js. Every part of our business logic -- which is almost the entire thing -- would have to be rewritten and ported. Thus they leave us with the task of refactoring 10,000+ lines of code, that already works.

2. They need us to repeat ourselves, or write our own methods, that Immutable.js already implements. It's not hard to rewrite these, but it is ultimately a lot of work, that furthermore has to be unit tested.

We need to guarantee the consistency of our internal data structure and have an overview over it

1. While they are type-checking, this ends up being very repetitive code, as it is included in their parser

2. It turns out that parsing the entities wraps the dependencies again, thus repeating part of the dependency resolvment Furthermore they describe the dependencies in what is type-checked, which is already guaranteed by Orchestra anyway.

3. It is very imperative code, that prevents us from writing entities quickly.

4. At the end of the day, not using Immutable.js doesn't prevent us from doing something stupid, as we can mutate internal data and are even encouraged to "break out" of the entities and write a temporary data structure without them.

The gains are low, while the cost is high

The cost is actually even higher, since the data is not being cached. That means that it stays hard to keep React from doing unnecessary rerenders. This is not referring to recreating / reparsing data, as that is part of any immutable structure anyway, but this refers to our state management.

Our state management emits a certain data structure and tracks changes. Due to the of how it emits changes and how Orchestra resolves the dependencies, we only emit a new state (from @connectStore) when the data actually changes. If we introduce Entities (f.e. on the shifts view) then further down the Component tree, it gets harder to reason about why we're rerendering. Entities are breaking the original reference and if we're doing operations on a per-render basis, the further we go down in the Component tree, the harder it gets to detect unnecessary rerenders.

This is something that we get for free without Entities. And not an irrelevant once, since rerendering an entire container is quite expensive.

Advantages of an alternative Approach with Models

• Models extend Immutable.js Maps, which allows us to use all of their helper methods Rewriting any business logic for methods won't be necessary. Models allow fine-grained, incremental migration.

• Methods type-check data, that is coming from the backend through AttributeTypes. These checks are only done on creation. Not when we're resolving dependencies, serializing more data, doing operations.

• Instead of always type-checking we guarantee their structure by "freezing" them. In the Orchestra post-hook (after dependency resolvement) we call .freeze(). A frozen Ancestor-Model can't create a new Child-Model in any way. This makes sure that we won't have a Model with an improper structure.

• We can still add type-checking to every mutation, if we want, but the current checks allow us to have dependencies only partially resolved, thus being able to display data, that is still loading. Meanwhile Entities need all dependencies to be present and resolved.

• They support our FRP-like coding style

• They bring a lot of caching for free. We don't break references, thus it is more likely that we don't have to worry about unnecessary rerenders.