monbang

# The Best Medium-Hard Data Analyst SQL Interview Questions  

By Zachary Thomas ([[email protected]](mailto:[email protected]), [Twitter](https://twitter.com/zach_i_thomas), [LinkedIn](https://www.linkedin.com/in/thomaszi/)) 

**Tip: **See the Table of Contents (document outline) by hovering over the vertical line on the right side of the page 

## Background & Motivation

> The first 70% of SQL is pretty straightforward but the remaining 30% can be pretty tricky.

eduardorost

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
import org.slf4j.{Logger, LoggerFactory}

object Main {
  val logger: Logger = LoggerFactory.getLogger(this.getClass)

  private lazy val sparkConf: SparkConf = new SparkConf()
    .setMaster("local[*]")

rmoff

kafkacat -b localhost:9092 \
         -t _kafka-connect-group-01-status \
         -C \
         -o-3 \
         -c3 \
         -f 'Topic %t / Partition %p / Offset: %o / Timestamp: %T\nHeaders: %h\nKey (%K bytes): %k\nPayload (%S bytes): %s\n--\n'

yashk

1. Auto Scaling Systems With Elastic Spark Streaming: Spark Summit East talk by PhuDuc Nguyen

• https://www.youtube.com/watch?v=N10k7tRsTUA

2. Tool for launching spark cluster in aws

• https://github.com/nchammas/flintrock

3. Spark dynamic allocation

• https://www.slideshare.net/databricks/dynamic-allocation-in-spark
• https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-dynamic-allocation.html

afranzi

Setup Environmnet variables for Hadoop.

export HADOOP_VERSION=2.8.5
export HADOOP_HOME=${HOME}/hadoop-$HADOOP_VERSION
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=${HADOOP_HOME}/bin:$PATH

Download Hadoop files.

ericnormand

#!/bin/sh
#_(

   #_DEPS is same format as deps.edn. Multiline is okay.
   DEPS='
   {:deps {clj-time {:mvn/version "0.14.2"}}}
   '

   #_You can put other options here
   OPTS='

laughedelic

Some of these practices might be based on wrong assumptions and I'm not aware of it, so I would appreciate any feedback.

1. avoiding some dependency conflicts:

   • install sbt-explicit-dependencies globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
   • run undeclaredCompileDependencies and make the obvious missing dependencies explicit by adding them to libraryDependencies of each sub-project
   • (optionally) run unusedCompileDependencies and remove some obvious unused libraries. This has false positives, so ; reload; Test/compile after each change and ultimately run all tests to see that it didn't break anything
   • (optionally) add undeclaredCompileDependenciesTest to the CI pipeline, so that it will fail if you have some undeclared dependencies

2. keeping dependencies up to date and resolving conflicts:

   • install sbt-updates globally in your `~/.sbt/{0.13,1.0}/plugins/plugins.

mayankcpdixit

Install kafka in your local mac machine

run following commands:

brew install kafka
sudo mkdir -p /usr/local/var/run/zookeeper/data
sudo chmod 777 /usr/local/var/run/zookeeper/data
zkServer start

mkdir -p /usr/local/var/lib/kafka-logs

hansohn

#!/usr/bin/env bash

BREW_PREFIX=$(brew --prefix);

# abort if already installed, else continue
if ! grep -q hadoop <(brew list); then
  # unlink broken installation attempts
  brew unlink hadoop > /dev/null 2>&1;

  # install hadoop 2.7.3

squarism

# Examples of Different Styles and Syntax

# legal / positive case / you can do these stuffs
# ---------------------------------------------------------------------------

# single line, nothing special
Enum.map(0...2, fn i -> IO.puts i end)
"""
0
1