coda-implementation.md

Key points on implementation:

1. There are two parts in coflow client: aggregation and enforcement.

2. Aggregation passes ground truth coflow information to the scheduler, which requires application specific modification.

3. Enforcement gives out concrete scheduling orders, and also collect other low-level information that is application-agnostic, such as flow bytes sent, starting time, TCP port and IP, etc..

4. Aalo requires both parts to operate, which modify applications in a application-specific way.

5. Aggregation is difficult to implement for two reasons.

5.1 Coflow API requires centralized coflow registration, and the result of registration, i.e. coflow ID, needs to be distributed to all the participants before each coflow starts. This is not efficient for small jobs, as at least one round of extra coordination is required.

5.2 While new application could be designed in a coflow compliant way from ground up, it would be difficult to modify existing applications, which usually designed in a way that job level information is strictly separated from networking socket level information. The latter is usually implemented in a form of library for both abstraction and efficiency reasons, provide specific interfaces to above, e.g. RPC or message passing.

5.3 For example, in Apache Spark, each flow has its own "stream Id" that referring to a TCP stream, which has no direct relations to the "shuffle Id" that has specific meanings to Spark jobs.

5.4 Another example is the Akka library that implements the actor model for communication in distributed application. Such library is designed specifically for programmers to focus on message passing and be obilivious to the underlying communication mechanism.

5.5 Overall speaking, it requires a considerable amount of programming effort to add coflow API on existing applications, which usually leave little space, usually in their design, to pass extra information around.

5.6 While aggregation is implemented for evaluaton, it is implemented using ad-hoc instrumentation to private API that runnable only for specific version of Hadoop/Spark, subjected to future changes.

6. Coda requires only enforcement, which requires application-agnostic modifications.

7. Coda includes aggregation part for evaluation against ground truth.

8. Enforcement is implemented as a shim layer between JVM and OS, using Java byte code instrumentation.

9. Enforcement could be implemented in kernel space as well, but less portable than Java Virtual Machine.

10. Kernel level implementation is not readily deployable for Coda, as it would be difficult to prevent intervention to background (irrelevant) flows, of which the pattern is not extensively investigated and, potentially, varies with different data center environment.