Javier Arturo Porras Luraschi javierluraschi

Adapted for sparklyr from cloud.google.com/solutions/spark-on-kubernetes-engine:

First, configure the cluster using Google Cloud client and kubectl:

gcloud config set compute/zone us-central1-f
gcloud container clusters create spark-on-gke --machine-type n1-standard-2

Next we need to bind the cluster admin to your email (you can retrieve your account email with gcloud config get-value account if needed):

<html>
    <head>
        <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.0/jquery.min.js"></script>
        <script type="text/javascript">
            function message(msg) {
                // innerHTML
                // document.getElementById("mymessage").innerText = document.getElementById("mymessage").innerText + " <b>" + msg + "</b>";
                $("#mymessage").text($("#mymessage").text() + " " + msg);
 }

This script uses an Amazon EMR Spark cluster with 10 m3.xlarge instances.

Installing

install.packages(devtools)
install.packages(sparklyr)
devtools::install_github("tylermorganwall/rayrender")

system2("hadoop", args = c("fs", "-mkdir", "/rendering"))

brew update
brew install mesos

/usr/local/Cellar/mesos/1.6.1/sbin/mesos-master --registry=in_memory --ip=127.0.0.1

mkdir temp
cd temp

pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-boost
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-brotli
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-cmake
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-flatbuffers
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gcc
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gobject-introspection
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gtk-doc
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-lz4

https://kafka.apache.org/quickstart

wget http://apache.claz.org/kafka/2.1.0/kafka_2.12-2.1.0.tgz
tar -xzf kafka_2.12-2.1.0.tgz

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Building Arrow R Bindings in Windows

This doc captures results from investigating how to build the Arrow R bindings in Windows. The following options were explored:

Building Arrow in MSys and bindings in RTools.
Building Arrow and bindings in RTools.

The most promising long-term solution is to write a CMake generator that can be run from RTools; short term, we can continue making progress compiling Arrow from MSys.

This document explains other approaches considered and provides additional details.

Create EC2 Windows Machine and connect.
Download and install MSys2.
Launch c:\msys64\mingw64 and run:

pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-i686-toolchain
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-cmake

Create EC2 Windows Machine and connect.
Download and install MSys2.
Launch c:\msys64\mingw64 and run:

pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-i686-toolchain
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-cmake


	sudo pip install PyArrow

	./pyspark --master yarn --num-executors 2

	from pyspark.sql.functions import rand
	df = spark.range(1 << 22).toDF("id").withColumn("x", rand())

	from pyspark.sql.functions import udf
	@udf('double')