Javier Arturo Porras Luraschi javierluraschi

Automated Install

EMR Configuration, replace <a-github-pat> with a valid PAT:

[{
  "configurations":[{
    "classification":"export",
    "properties":{"GITHUB_PAT":"<a-github-pat>"}
 }],

Create EC2 Windows Machine and connect.
Download and install MSys2.
Launch c:\msys64\mingw64 and run:

pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-i686-toolchain
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-cmake

Create EC2 Windows Machine and connect.
Download and install MSys2.
Launch c:\msys64\mingw64 and run:

pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-i686-toolchain
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-cmake

Building Arrow R Bindings in Windows

This doc captures results from investigating how to build the Arrow R bindings in Windows. The following options were explored:

Building Arrow in MSys and bindings in RTools.
Building Arrow and bindings in RTools.

The most promising long-term solution is to write a CMake generator that can be run from RTools; short term, we can continue making progress compiling Arrow from MSys.

This document explains other approaches considered and provides additional details.

https://kafka.apache.org/quickstart

wget http://apache.claz.org/kafka/2.1.0/kafka_2.12-2.1.0.tgz
tar -xzf kafka_2.12-2.1.0.tgz

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-boost
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-brotli
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-cmake
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-flatbuffers
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gcc
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gobject-introspection
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-gtk-doc
pacman -S --noconfirm $MINGW_PACKAGE_PREFIX-lz4

brew update
brew install mesos

/usr/local/Cellar/mesos/1.6.1/sbin/mesos-master --registry=in_memory --ip=127.0.0.1

mkdir temp
cd temp

This script uses an Amazon EMR Spark cluster with 10 m3.xlarge instances.

Installing

install.packages(devtools)
install.packages(sparklyr)
devtools::install_github("tylermorganwall/rayrender")

system2("hadoop", args = c("fs", "-mkdir", "/rendering"))

<html>
    <head>
        <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.0/jquery.min.js"></script>
        <script type="text/javascript">
            function message(msg) {
                // innerHTML
                // document.getElementById("mymessage").innerText = document.getElementById("mymessage").innerText + " <b>" + msg + "</b>";
                $("#mymessage").text($("#mymessage").text() + " " + msg);
 }


	sudo pip install PyArrow

	./pyspark --master yarn --num-executors 2

	from pyspark.sql.functions import rand
	df = spark.range(1 << 22).toDF("id").withColumn("x", rand())

	from pyspark.sql.functions import udf
	@udf('double')