Prateek Rungta prateek

Offline Python Package Install Notes

Two common cases which make the install of Python packages harder due to networking issues are: a) Install behind a Proxy b) Install without access to the internet

(a) Install behind a Proxy

In the case where the target machine connects to the internet over a network proxy, export the following environment vars, as appropriate - http_proxy and https_proxy. Eg:

Create a FlameGraph to visualize where your code is spending its time.

Requires plop and FlameGraph.

	#!/usr/bin/env python
	# encoding: utf-8

	# File Transfer for Pythonista
	# ============================
	# This script allows you to transfer Python files from
	# and to Pythonista via local Wifi.
	# It starts a basic HTTP server that you can access
	# as a web page from your browser.
	# When you upload a file that already exists, it is

	/**
	git clone https://github.com/twitter/scalding.git
	cd scalding
	./sbt scalding-repl/console
	*/

	import scala.io.Source
	val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLines
	// Add the line numbers, which we might want later
	val aliceLineNum = alice.zipWithIndex.toList

	#!/bin/bash

	# paranoia with shell scripts, always to be encouraged
	set -e
	LOG=/tmp/shell-impala-$USER-$(date +%s).log

	# echo-ing params to find logs on local fs in case of error
	echo "HOST: ${HOSTNAME}"
	echo "LOG: ${LOG}"

	-- 1. Place in ~/Library/Scripts and enable the Applescript menu via the Applescript Editor
	-- (Or export to .app to run from spotlight.)
	-- 2. Substitute "vpn.example.com" and "redacted" for your VPN server and password
	-- 3. Open Security & Privacy System Preferences, go to Privacy, Accessibility
	-- 4. Enable Applescript Editor and System UI Server (or for this .app if so exported)
	-- 5. Trigger script from the menu (or run from spotlight)
	-- 6. Enjoy being connected
	-- 7. Run script again to close connection

	/**
	* Created by prungta on 7/22/15.
	*/
	import org.apache.oozie.client.AuthOozieClient;
	import org.apache.oozie.client.OozieClient;
	import org.apache.oozie.client.OozieClientException;
	import org.apache.oozie.client.WorkflowJob;

	import javax.security.auth.Subject;
	import javax.security.auth.login.AppConfigurationEntry;

	// once upon a time, what did I see, but two RDDs
	val tableA = sc.parallelize(List((1,2), (3,4), (5,6)))
	val tableB = sc.parallelize(List((1,'A'), (3,'B'), (5,'C')))
	// one of them small enough to fit into map
	val mapTableB = tableB.collectAsMap
	// which everyone could read
	val broadcastB = sc.broadcast(mapTableB)
	// and join for all eternity
	val mapJoin = tableA.map({case (id, value) => (id, value, broadcastB.value.get(id))})
	// q.e.d.


	// function to get list of files in a directory, recursively
	getFileList := func(path string) map[string]struct{} {
	fileList := make(map[string]struct{})
	err = filepath.Walk(path, func(path string, f os.FileInfo, err error) error {
	fileList[path] = struct{}{}
	return nil
	})
	require.NoError(t, err)
	return fileList

	package main

	import (
	"fmt"
	"io"
	"io/ioutil"
	"net"
	"os"
	"strings"