tnachen

Up level your company's Big Data with Resource Management

Where is Big data at now?
------------------
Big data was once one of the biggest technology hype, where tons of presentations and posts talk about how the new systems and tools allows large and complex data to be processed that traditional tools wasn't able to. While Big data was being at the peak of its hype, most companies are still getting familir of the new data processing frameworks such as Hadoop, and new databases such as HBase and Cassandra. Fast foward to now where Big data is still a popular topic, lots of companies has already jumped into the Big data bandwagon and are already moving past the first generation Hadoop to evaluate newer tools such as Spark and newer databases such as Firebase, NuoDB or Memsql. But most companies also learn from running all of these tools, is that deploying, operating and planning capacity for these tools are very hard and complicated. Although over time lots of these tools has became more mature, all of these

tnachen

Up level your company's Big Data with Mesos:

Last post I talked about how utilizing a resource management platform can allow your Big Data workloads to be more efficient with less resources, in this post I want to continue the discussion with a specific resource management platform which is Mesos.

Introduction to Mesos
-----------------------------------

Mesos (http://mesos.apache.org/) is an Apache top-level project that provides an abstraction to your datacenter resources and an API to program aganist these resources to launch and manage your workloads. Mesos is able to manage your CPU, memory, disk, ports and other resources that the user can custom defines. Every application that wants to use resources in the datacenter to run tasks talks with Mesos is called a scheduler, and uses the scheduler API to receive resource offers and each scheduler can decide to use the offer, decline the offer to wait for future ones, or hold on the offer for a period of time to combine the resources. Mesos will ensure to 

tnachen

錫安兒女  歡喜快樂
Rejoice, rejoice, daughter of Zion

以色列  揚聲歡呼
Shout aloud, Israel

全心全意  歡欣歌頌
Sing, rejoice with all your heart

哦  耶路撒冷

tnachen

[==========] Running 1 test from 1 test case.                                                                                                                                             [85/1944]
[----------] Global test environment set-up.
[----------] 1 test from LinuxFilesystemIsolatorTest
[ RUN      ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost
Using temporary directory '/tmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHost_hvxwGY'
I1025 12:38:09.026170  1527 linux.cpp:80] Making '/tmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHost_azTxzS' a shared mount
Work dir mounting: /tmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHost_azTxzS
I1025 12:38:09.050469  1527 linux_launcher.cpp:103] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I1025 12:38:09.072546  1548 containerizer.cpp:615] Starting container '102d04fe-85e4-464a-a901-55ad6f1b6086' for executor 'test_executor' of framework ''
I1025 12:38:09.094017  1543 provisioner.cpp:289] Provisioning image rootfs '/tmp/LinuxFilesys

tnachen

[==========] Running 1 test from 1 test case.                                                                                                                                             [99/1807]
[----------] Global test environment set-up.
[----------] 1 test from LinuxFilesystemIsolatorTest
[ RUN      ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost
Using temporary directory '/tmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHost_4iFnnG'
I1026 19:37:36.392563  1399 linux.cpp:80] Making '/tmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHost_T57ktv' a shared mount
I1026 19:37:36.407768  1399 linux_launcher.cpp:103] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I1026 19:37:36.412539  1399 systemd.cpp:128] systemd version `208` detected
W1026 19:37:36.412575  1399 systemd.cpp:136] Required functionality `Delegate` was introduced in Version `218`. Your system may not function properly; however since some distributions have patche
d systemd packages, your system may still be functiona

tnachen

diff --git a/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp b/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
index a8edf13..2d486dc 100644
--- a/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
+++ b/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
@@ -625,18 +625,15 @@ Future<size_t> RegistryClientProcess::saveBlob(
 {
   return reader.read()
     .then([this, fd, reader, totalSize](
-        const string& data) mutable -> Future<size_t> {
+        const string& data) -> Future<size_t> {

tnachen

diff --git a/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp b/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
index 10c5e7a..29d4d4d 100644
--- a/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
+++ b/src/slave/containerizer/mesos/provisioner/docker/registry_client.cpp
@@ -672,12 +672,13 @@ Future<size_t> RegistryClientProcess::getBlob(
     return Failure("Invalid repository path: " + path);
   }
 
+  const string blobURLPath = "v2/" + path + "/blobs/" + digest.getOrElse("");
+

tnachen

# Prepare Steps:
# We are assuming we’re using CUDA 7.5
# Clone and Build latest Mesos with GPUs (skipping instructions here…)
# CUDA prepare script for ubuntu 14.04 with AWS G2 instance. 
# http://tleyden.github.io/blog/2015/11/22/cuda-7-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/
# Start mesos master and slave
# ./mesos-master --work_dir=/tmp/mesos
# sudo GLOG_v=1 ./mesos-slave --isolation=gpu/nvidia --master=`hostname -i`:5050 --nvidia_gpu_devices=0

# Add cuda bin to PATH

tnachen

import numpy as np
import tensorflow as tf
import os
from tensorflow.python.platform import gfile
import os.path
import re
import sys
import tarfile
from subprocess import Popen, PIPE, STDOUT
def run(cmd):

tnachen

我曾聽無數人在述說
你是怎樣的神
但夜裡我卻聽見你
溫柔慈愛聲音
對我說 你喜悅我
你永遠都不離棄
You're a Good, Good Father
那就是你 那就是你 那就是你
我是你所愛
那就是我 那就是我