hustcat

#!/bin/bash

if ! [ $# -eq 1 ]; then
    echo "Usage: $0 <node-name>"
    exit 1
fi

# Directory where everything happens
DIR=~/work/virt-install
# The image downloaded from the http://fedoraproject.org/en/get-fedora#clouds site

hustcat

Spark driver

Spark driver的初始化主要围绕SparkContext的初始化，SparkContext初始化完之后，才能向Spark集群提交任务。

• spark context初始化

    // Create and start the scheduler
    val (sched, ts) = SparkContext.createTaskScheduler(this, master, deployMode) ///创建任务调度器
    _schedulerBackend = sched

hustcat

• 运行exmaple中示例错误1:

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration

VM options启动参数增加:

-Dspark.master=local

hustcat

• Apache Spark: core concepts, architecture and internals
• https://spark-summit.org/2014/a-deeper-understanding-of-spark-internals/
• Resource Allocation in Mesos: Dominant Resource Fairness

scheduler

• Spark源码分析之-scheduler模块
• 深入理解Spark 2.1 Core （二）：DAG调度器的原理与源码分析

hustcat

Install

Install minikube:

# curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.18.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Install kubectl:

hustcat

hive

• Impala与Hive的比较