Laxman-SM

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-001
spec:
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteOnce
    - ReadWriteMany

Laxman-SM

resource "azurerm_virtual_machine" "vm" {
  name                  = "myex-vm"
  location              = "northeurope"
  resource_group_name   = "my-rg"
  vm_size               = "Standard_DS1_v2"
  network_interface_ids = ["${azurerm_network_interface.nic.id}"]

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"

Laxman-SM

#!/usr/bin/env bash

export JAVA_HOME=/usr/java/default
export M2_HOME=/opt/maven/apache-maven-3.0.5
export MAVEN_OPTS="-Xmx4096m -XX:MaxPermSize=512m"
export PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin

amb-install-dev() {
 yum clean all -y && yum update -y
 yum -y install vim wget rpm-build sudo which telnet tar openssh-server openssh-clients ntp git python-setuptools python-devel httpd

Laxman-SM

########################################################################
########################################################################
## Run this full block to set the environent for talking to solr via curl
##   - must be on an Ambari Infra host
########################################################################

## authenticate
sudo -u infra-solr -i
keytab=/etc/security/keytabs/ambari-infra-solr.service.keytab
kinit -kt ${keytab} $(klist -kt ${keytab}| awk '{print $NF}'|tail -1)

Laxman-SM

SSSD Configuration

What I use for Hortonworks HDP (Hadoop) systems, but should work for anyone.

Some configurations are tuned for Active Directory without relying on 'sssd-ad' such that the hosts don't need to join the domain.

Install requirements

sudo yum install sssd sssd-ldap sssd-krb5 sssd-tools authconfig \
  oddjob oddjob-mkhomedir openldap-clients cyrus-sasl-gssapi \

Laxman-SM

Compress files which are already on HDFS

This hacky method processes 1 file at a time:

1. copy to a local disk
2. compress
3. put back onto HDFS
4. delete original file from HDFS and compressed file from local disk.

BE CAREFUL: Before executing, inspect the size of each file!

• The risk is: a single large file could fill the local disk or you could leave the server compressing a single large file for hours.

Laxman-SM

Laxman-SM

#######################################################################
# This is Kyle's Python script for Data Quality Assessment
# Assume df = your dataframe
# Replace "df" with "[Your Dataframe]"
#######################################################################

#######################################################################
# Check summary statistics
#######################################################################
df.describe()

Laxman-SM

#!/usr/bin/env bash

set -xeuo pipefail

usage()
{
    echo "Usage: $0 -p <path of configs to be overriden>" >&2
    exit 1
}

Laxman-SM

#!/bin/bash
set -eux

VERSION=0.172
wget -O /tmp/HDInsightUtilities-v01.sh -q https://hdiconfigactions.blob.core.windows.net/linuxconfigactionmodulev01/HDInsightUtilities-v01.sh && source /tmp/HDInsightUtilities-v01.sh && rm -f /tmp/HDInsightUtilities-v01.sh

mkdir -p /var/lib/presto
chmod -R 777 /var/lib/presto/

if [[ `hostname -f` == `get_primary_headnode` ]]; then