junegunn

# GIT heart FZF
# -------------

is_in_git_repo() {
  git rev-parse HEAD > /dev/null 2>&1
}

fzf-down() {
  fzf --height 50% --min-height 20 --border --bind ctrl-/:toggle-preview "$@"
}

smartnose

Spark internals through code

Nothing gives you more detail about spark internals than actually reading it source code. In addition, you get to learn many design techniques and improve your scala coding skills. These are the random notes I make while reading the spark code. The best way to comprehend the notes is to load spark code into an IDE, e.g. IntelliJ, and navigate the code on the side.

Genesis - creation of a spark cluster

The scripts for creating a spark cluster are: start-master.sh and start-slave.sh. Read them carefully, and you can see that both scripts are very similar except the values for $CLASS variable. For start-master.sh, the value is CLASS="org.apache.spark.deploy.master.Master", while the value for start-slave.sh is shown below with more context.

# NOTE: This exact class name is matched downstream by SparkSubmit.

bernardolopes8

package my.application.dao;

import java.util.List;

import my.application.entity.Book;
import my.application.restutil.RequestOptions;

public interface BookPersister {
    List<Book> getBooks(RequestOptions requestOptions);
}

sebsto

drop table store;

drop table sales;

drop materialized view city_sales;


CREATE TABLE "sales" (
  "id" int PRIMARY KEY,
  "item" varchar,

dramaticlly

import argparse
import logging
import os
import sys
from datetime import datetime, timedelta

# import google.cloud.logging
import requests

# import twitter