Laxman Singh Laxman-SM

Jupyter + Pyspark how-to

Version 1.0 2016/11/14
Pierdomenico Fiadino | pierdomenico.fiadino@eurecat.org

Synopysis

Install Jupyter Notebook in a dedicated Python virtualenv and integrate with Spark terminal pyspark on a cluster client (for this example, we will use the tourism-lab node).

This how-to assumes that we have SSH access to the machine and pyspark already installed and configure (try executing it and see if the variables sc, HiveContext and sqlContext are already installed).

Azure DevOps CI/CD with Data Factory

Create a Data Factory pipeline in our Azure Portal

In this example, we have one simple copy activity.

Please note on the upper left hand corner, github is already set as the default repository
Currently, the branch is pointing at Master
Click the drop down on the New Branch or [Alt-N]
Enter a new feature Branch
Example "Thirdcopybranch Branch"

Notes

Search text is case sensitive
Try to force attribute and text in your search to be in the same case
Wildcard search use STARTSWITH and ENDSWITH
two different types of nulls:
missing elements
actual null values.
Use IS_NULL()and IS_DEFINED() functions, to detect missing attributes and ternary logic
Cosmos DB SQL currently does not have an equivalent of SQL LEFT() function. Instead, use SUBSTRING()
There is no Year() function in Cosmos DB SQL, use SUBSTRING() to the date string and isolate

	FROM python:3.7-slim-stretch

	LABEL maintainer="Yevhen Lebid <yevhen.lebid@loopme.com>"

	# Never prompts the user for choices on installation/configuration of packages
	ENV DEBIAN_FRONTEND=noninteractive \
	TERM=linux

	# Airflow
	ARG AIRFLOW_VERSION=1.10.7

	# TODO: write the help
	# TODO: support SSL self-signed certs
	# TODO: need to validate JSON, and/or add a new param set that accepts hashtable and
	# convert internally.
	# TODO: support RAW mode
	# TODO: refactor to use EC batch (concatenated events in one HTTP request) instead of
	# PowerShell pipelines which will do a request per object (event payload) on the pipeline
	# TODO: think about load balancing per Geoffrey Martins.

	#!/bin/bash

	MINICONDA_VERSION="4.3.21"
	PANDAS_VERSION="0.20.3"
	SCIKIT_VERSION="0.19.0"

	while [[ $# > 1 ]]; do
	key="$1"

	case $key in

	#!/usr/bin/env python
	import boto3
	import json
	import base64
	import requests
	import os

	def headers_to_go_style(headers):
	retval = {}
	for k, v in headers.items():

	#############################
	# ec2 instance
	#############################
	resource "aws_instance" "nt-test-ec2" {
	count = 1

	ami = "${lookup(var.ami, var.region)}"
	instance_type = "t2.medium"
	key_name = "${var.key_pair_name}"
	#ebs_optimized = true

	DECLARE @serverName VARCHAR(20)
	DECLARE @storageSecret VARCHAR(MAX)
	DECLARE @storageLocation VARCHAR(MAX)

	SET @serverName = convert(VARCHAR(20),(SELECT SERVERPROPERTY('ServerName')))

	IF(@serverName = 'peet-bi-dev')
	BEGIN
	SET @storageSecret = '{your_own_secret}'
	SET @storageLocation = 'wasbs://{container_name}@{storage_name}.blob.core.windows.net'

	#
	# ManipulateADF.ps1
	#

	# automate log-in
	$resourceGroupName = "{your_resource_group}"
	$azureAccountName = '{your_email}'
	$azurePassword = ConvertTo-SecureString '{your_password}' -AsPlainText -Force
	$azureRMCredential = New-Object System.Management.Automation.PSCredential($azureAccountName, $azurePassword)
	$account = Login-AzureRmAccount -Credential $azureRMCredential