Skip to content

Instantly share code, notes, and snippets.

View byronyi's full-sized avatar
:octocat:
Just for fun

Bairen Yi byronyi

:octocat:
Just for fun
339 followers · 9 following
View GitHub Profile
@byronyi
byronyi / coda.c
Last active March 25, 2016 08:26
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/bug.h>
#include <linux/version.h>
#include <linux/kobject.h>
#include <linux/hashtable.h>
#include <linux/kern_levels.h>
#include <linux/skbuff.h>
@byronyi
byronyi / a.rb
Last active December 29, 2015 06:49
$ yarn jar target/ml-0.0.1-SNAPSHOT.jar com.tencent.ml.example.LDA.LDA job.xml
15/12/29 14:38:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/29 14:38:08 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
15/12/29 14:38:08 INFO utils.HdfsUtil: tmp output dir is hdfs:///tmp/byronyi/application_1451365016523_0018_e0394527-4994-4cc5-b549-83b9fed9a0fe
15/12/29 14:38:08 INFO yarn.AngelYarnClient: angel.tmp.output.path=hdfs:/tmp/byronyi/application_1451365016523_0018_e0394527-4994-4cc5-b549-83b9fed9a0fe
15/12/29 14:38:08 INFO yarn.AngelYarnClient: Configuring job application_1451365016523_0018 with /tmp/hadoop-yarn/byronyi/.staging/application_1451365016523_0018 as the submit dir
15/12/29 14:38:08 INFO yarn.AngelYarnClient: default FileSystem: hdfs://localhost:9000
15/12/29 14:38:08 INFO yarn.AngelYarnClient: libjarsDir=/tmp/hadoop-yarn/byronyi/.staging/application_1451365016523_0018/libjars
15/12/
@byronyi
byronyi / a.rb
Created December 29, 2015 06:48
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/Cellar/hadoop/2.7.1/libexec/logs/userlogs/application_1451365016523_0018/container_1451365016523_0018_01_000001 (Is a directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
at org.apache.hadoop.yarn.ContainerLogAppender.activateOptions(ContainerLogAppender.java:55)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
@byronyi
byronyi / job.xml
Created December 29, 2015 06:49
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000/</value>
</property>
<property>
<name>angel.job.libjars</name>
<value>file:///Users/byronyi/Develop/angel/target/ml-0.0.1-SNAPSHOT.jar,file:///Users/byronyi/Develop/angel/lib/SizeOf-1.0.jar</value>
</property>
<property>
@byronyi
byronyi / log4j.properties
Created December 29, 2015 06:55
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
@byronyi
byronyi / angel.log
Created December 29, 2015 08:39
$ yarn jar target/ml-0.0.1-SNAPSHOT.jar com.tencent.ml.example.LDA.LDA job.xml
15/12/29 16:38:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/29 16:38:32 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
Exception in thread "main" java.io.IOException: java.lang.IllegalArgumentException: Can not create a Path from a null string
at com.tencent.ml.client.yarn.AngelYarnClient.submitApplication(AngelYarnClient.java:186)
at com.tencent.ml.example.LDA.LDA.main(LDA.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
@byronyi
byronyi / coda-implementation.md
Last active January 6, 2016 12:15

Key points on implementation:

  1. There are two parts in coflow client: aggregation and enforcement.

  2. Aggregation passes ground truth coflow information to the scheduler, which requires application specific modification.

  3. Enforcement gives out concrete scheduling orders, and also collect other low-level information that is application-agnostic, such as flow bytes sent, starting time, TCP port and IP, etc..

  4. Aalo requires both parts to operate, which modify applications in a application-specific way.

@byronyi
byronyi / getSchedule.scala
Last active January 26, 2016 15:00
def getSchedule(flows: Iterable[Flow], priorities: Iterable[Int], bandwidth: Long): Map[Flow, Long] = {
val priorityQueue = priorities.zip(flows).groupBy(_._1).toArray.sortBy(_._1).map(_._2.map(_._2))
val egressFree = flows.map(_.srcIp).map((_, bandwidth)).toMap
val ingressFree = flows.map(_.dstIp).map((_, bandwidth)).toMap
priorityQueue.flatMap(queue => {
val egressNumFlows = queue.groupBy(_.srcIp).mapValues(_.size)
val ingressNumFlows = queue.groupBy(_.dstIp).mapValues(_.size)
@byronyi
byronyi / zmqstub.c
Created February 8, 2016 11:23 — forked from HarryR/zmqstub.c
zmq & libevent stub
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <libgen.h>
#include <signal.h>
#include <err.h>
#include <assert.h>
#include <zmq.h>
@byronyi
byronyi / experiments.md
Created February 24, 2016 06:49
  • Parameter split: number of parameter servers
  • Input Split: number of workers
  • Staleness: number of iterations

5M parameters, 5M instances, 30 iterations, 0 staleness, 2 PS, 5 worker

time: 4:14 appid: application_1453379488095_0319

50M parameters, 5M instances, 30 iterations, 0 staleness, 2 PS, 5 worker