Some notes about the challenge of performing large scale simulations with DTs.
HIGH UTILIZATION
The guiding principle is that you best solve compute problems by running resources just below their saturation level, saving both money and time. Money and time are both sucked up by under utilized resources and I/O processing.
| Utilization | Overall performance and costs align with high-utilization. |
| Liveliness | Liveliness sufferes from high utilization, if you are not careful, so auto-scale tuned to maintain resources just below their saturation level is required |
| Blocking I/O | Blocking I/O is bad - non-starter |
| Lazy I/O | Lazy I/O is bad but exceptions to this rule can unlock a design - don't categorically discount (ie: event sourcing and rewind are implemented via lazy I/O) |
| Eager I/O | Eager I/O is free and good |
| Sharding | All use cases must be compatable with sharding - no exceptions. A simulation should be able to plan the shards to avoid reshuffling but a live system may be challenged because it will have to support reshuffling that works in concert with auto-scaling. Leaves cannot be assumed to talk to leaves outside a shard - exceptional routing must be enabled for these cases and these cases must be exceptional. to break free of this limitation, have a DT in a shard compute an aggregate to be shared outside the shard (this is how cloud billing and monitoring scales) |
| Copying | Moving data towards the code is the enemy! Ideally, the data a DT needs for a computation is present in the DT state naturally w/o any additional I/O required before invoking the function. Using a popular thirdparty lib for a calculation may involve marshalling a large ammount of data to a single point to be passed as input to the function - this is where all the time goes and low utilization comes from. This cost can be defeated by having aggregate DTs that compute from a hierarchy of aggregates - no recalculating anything (but this may not be supported in a given algorythm) |
| Numbers | All internal data must be native numerical - no envelopes. metadata must be normalized and resolved lazily for presentation or setting up context - but not parsed in the actual procesing |
| Tuning Containers | If the above are not respected then container tuning runs into Amdahl's Law :) If the above are respected, container performace can be achieved by limiting the copying of data between userland and kernel space, IE: reduce context switching, eliminate bad paging / memory thrashing, etc.. |
The above suggests an implementation of:
Streaming observerations sharded accross a cluster of containers. Kafka co-located with the containers is ideal for this. Conversely, Kafka as a PAAS is death to scaling this.
| Cloud Provider | Parts |
|---|---|
| Azure | AKS, Kafka in K8s, k8s or managed Cassandra |
| AWS | EKS, Kafka in K8s, k8s or managed postgress (maybe) |
| GKE, Kafka in K8s, k8s or managed postgress (maybe) |
observations via Inet
|
V
cloud ingress (w/sharding)
|
V
managed k8s (w/moresharding)
|
V
container(s) (where actors within each have free reign to talk to each other but cross container comm is via aggregate)
|
V
output to streams (PAAS Kafka ok here, webhooks ok, anything that looks like a log ok)