Bairen Yi byronyi

Testing Gluster RDMA with Soft-iWARP

There’s a Gluster 3.4 RDMA test day right around the corner, and I want to join in on the fun. The trouble is, I don’t have any RDMA-capable hardware in my lab right now. Undaunted, I hit the Web in search of a software-based solution, one that would at least allow me to run through the tests.

I found a pair of promising-looking options:

The information out on the web about these projects is a bit thinner than I’d like, but I found a blog post howto on installing Soft-iWARP on Ubuntu 10.10 and another for Debian 6 and figured I’d try it out on Fedora 18.

#System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

##Basic Steps

Clarify and agree on the scope of the system

User cases (description of sequences of events that, taken together, lead to a system doing something useful)
- Who is going to use it?
- How are they going to use it?

Parameter split: number of parameter servers
Input Split: number of workers
Staleness: number of iterations

5M parameters, 5M instances, 30 iterations, 0 staleness, 2 PS, 5 worker

time: 4:14 appid: application_1453379488095_0319

50M parameters, 5M instances, 30 iterations, 0 staleness, 2 PS, 5 worker

Key points on implementation:

There are two parts in coflow client: aggregation and enforcement.
Aggregation passes ground truth coflow information to the scheduler, which requires application specific modification.
Enforcement gives out concrete scheduling orders, and also collect other low-level information that is application-agnostic, such as flow bytes sent, starting time, TCP port and IP, etc..
Aalo requires both parts to operate, which modify applications in a application-specific way.

	$ ibv_rc_pingpong -d rxe0 -g 0
	local address: LID 0x0000, QPN 0x000011, PSN 0xd0fbd7, GID fe80::ae8:4fff:fefb:abbe
	remote address: LID 0x0000, QPN 0x000011, PSN 0x691d06, GID fe80::ae8:4fff:fefb:ade6
	8192000 bytes in 0.19 seconds = 341.88 Mbit/sec
	1000 iters in 0.19 seconds = 191.70 usec/iter

	Server Version:76
	TWS Time at connection:20160420 01:38:14 CST
	ERROR 2016-04-20 01:38:13,231 1 byronyi.github.io.hft.IBClient [EReader] Market data farm connection is OK:cashfarm
	ERROR 2016-04-20 01:38:13,231 1 byronyi.github.io.hft.IBClient [EReader] Market data farm connection is OK:usfarm.us
	ERROR 2016-04-20 01:38:13,232 2 byronyi.github.io.hft.IBClient [EReader] Market data farm connection is OK:usfarm
	ERROR 2016-04-20 01:38:13,232 2 byronyi.github.io.hft.IBClient [EReader] HMDS data farm connection is OK:cashhmds
	ERROR 2016-04-20 01:38:13,232 2 byronyi.github.io.hft.IBClient [EReader] HMDS data farm connection is OK:ushmds
	INFO 2016-04-20 01:38:15,609 2379 byronyi.github.io.hft.IBClient [EReader] 0 bidPrice 1.13803
	INFO 2016-04-20 01:38:15,611 2381 byronyi.github.io.hft.IBClient [EReader] 0 askPrice 1.13805
	INFO 2016-04-20 01:38:15,611 2381 byronyi.github.io.hft.IBClient [EReader] 0 high 1.13845

	from __future__ import division, print_function, unicode_literals


	def counting_sheep(N):
	'''
	Code Jam 2016, Qualification Round, Problem A
	'''
	digits = set(range(10))
	for i in range(1, int(1e6)):
	number = i*N

	#include <stdlib.h>
	#include <stdio.h>
	#include <string.h>
	#include <libgen.h>
	#include <signal.h>

	#include <err.h>
	#include <assert.h>

	#include <zmq.h>

	def getSchedule(flows: Iterable[Flow], priorities: Iterable[Int], bandwidth: Long): Map[Flow, Long] = {

	val priorityQueue = priorities.zip(flows).groupBy(_._1).toArray.sortBy(_._1).map(_._2.map(_._2))

	val egressFree = flows.map(_.srcIp).map((_, bandwidth)).toMap
	val ingressFree = flows.map(_.dstIp).map((_, bandwidth)).toMap

	priorityQueue.flatMap(queue => {
	val egressNumFlows = queue.groupBy(_.srcIp).mapValues(_.size)
	val ingressNumFlows = queue.groupBy(_.dstIp).mapValues(_.size)