Hassan Abedi habedi

On-CRAN packages for "Moving objects, trajectories"

This is for an update to the the SpatioTemporal Task View.

If you have any other suggestions, please leave a comment below - or raise an issue here: https://github.com/mdsumner/ctv-mdsumner/issues (or just email/tweet me).

(This is not the final presentation format).

Moving objects, trajectories

NiFi Example: Load CSV file into RDBMS Table using the traditional way and the new way using Record

Example Data

Ingested to the example flows by GenerateFlowFile:

ID, CITY_NAME, ZIP_CD, STATE_CD
001, CITY_A, 1111, AA
002, CITY_B, 2222, BB
003, CITY_C, 3333, CC

Proxy SSL Client Certificate through NGINX Load-Balancer

The frontend stream proxy_pass can be used for load-balancing without SSL off-loading. All SSL connections will be terminated on the backend and client certificate information can be properly authenticated.

This should be used in cases:

you have enough CPU to decrypt SSL on the backend servers
you require direct client AUTHENTICATION on the backend servers

Backend

Zeppelin, Spark, PySpark Setup on Windows (10)

I wish running Zeppelin on windows wasn't as hard as it is. Things go haiwire if you already have Spark installed on your computer. Zeppelin's embedded Spark interpreter does not work nicely with existing Spark and you may need to perform below steps (hacks!) to make it work. I am hoping that these will be fixed in newer Zeppelin versions.

If you try to run Zeppelin after extracting the package, you might encounter "The filename, directory name, or volume label syntax is incorrect."

Google search landed me to https://issues.apache.org/jira/browse/ZEPPELIN-1584, this link was helpful but wasn't enough to get Zeppelin working.

Below is what I had to do to make it work on my Windows 10 computer.

feature hashing in vw

base hash function

Murmur32 hash is implemented in uniform_hash in hash.cc. It takes a string and a seed and return uint64_t.

There are two hash modes specified by --hash option.

Strings mode (hashstring in parse_primitives.cc)

	from pyspark.sql import SparkSession

	def init_spark():
	spark = SparkSession.builder.appName("HelloWorld").getOrCreate()
	sc = spark.sparkContext
	return spark,sc

	def main():
	spark,sc = init_spark()
	nums = sc.parallelize([1,2,3,4])

	graph = {
	1: {2:1,5:1},
	2: {1:1,3:1,5:1},
	3: {2:1,4:1},
	4: {3:1,5:1,6:1},
	5: {1:1,2:1,4:1},
	6: {4:1}
	}

	time = 0

	#!/bin/sh
	# This script is from the tutorial of set docker in ubuntu server, further info https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#next-steps
	# This usually fail in aws ubuntu linux because is no extra packages in aws linux
	sudo apt-get install \
	linux-image-extra-$(uname -r) \
	linux-image-extra-virtual

	sudo apt-get update

	# Allow to use apt over https

	import keras.backend as K
	import multiprocessing
	import tensorflow as tf

	from gensim.models.word2vec import Word2Vec

	from keras.callbacks import EarlyStopping
	from keras.models import Sequential
	from keras.layers.core import Dense, Dropout, Flatten
	from keras.layers.convolutional import Conv1D


	import java.io.File;
	import java.io.FileInputStream;
	import java.io.StringWriter;
	import java.security.KeyStore;

	import javax.net.ssl.SSLContext;

	import org.apache.commons.io.IOUtils;
	import org.apache.http.HttpEntity;