Jeroen Ooms jeroen

SUPERSEDED

A new version of Solaris VM for R is available from https://github.com/jeroenooms/solarisvm

jsonlite and mongolite

The jsonlite package provides a powerful JSON parser and generator that has become one of standard methods for getting data in and out of R. We discuss some recent additions to the package, in particular support streaming (large) data over http(s) connections. We then introduce the new mongolite package: a high-performance MongoDB client based on jsonlite. MongoDB (from "humongous") is a popular open-source document database for storing and manipulating very big JSON structures. It includes a JSON query language and an embedded V8 engine

Most intuitive to me would be if dat import reads jsonlines or csv:

curl https://jeroenooms.github.io/data/diamonds.json | dat import -d diamonds
curl https://demo.ocpu.io/MASS/data/cats/csv | dat import -d cats --csv
cat mydata.csv | dat import -d mydata --csv

And dat export does the opposite: stream out jsonlines/csv to be used in a pipe:

	# Creating closures in R is actually very fast:
	data("diamonds", package = "ggplot2")
	diamonds$get <- lapply(seq_len(nrow(diamonds)), function(i){
	k <- i
	function(what){
	diamonds[k,what]
	}
	})

	# Execute:

	# If you are running beta version, you might need to symlink 'dat-beta' to 'dat'

	# Dependencies
	library(jsonlite)
	data("diamonds", package = "ggplot2")

	# Initiate the dat repo
	setwd(tempdir())
	system2("dat", "init")

	# Dependencies
	library(jsonlite)

	# Initiate the dat repo
	setwd(tempdir())
	system2("dat", "init")

	# Add 10 lines of data
	stream_out(iris[1:10,], pipe("dat add -"))
	head1 <- readLines(pipe("dat heads"))

	# Use MongoDB map-reduce for in-database k-means clustering
	# Requires recent version of monoglite:
	# devtools::install_github("jeroenooms/mongolite")
	stopifnot(packageVersion("mongolite") >= "0.3.9001")
	library(mongolite)

	# Connect to 'diamonds' collection
	m <- mongo("diamonds")

	# Wipe old data if any

	# Profile only works after changing default shell to bash with: passwd -e
	# If you are root just store in /.profile

	# Set Terminal
	export TERM=xterm

	# To find solaris 'make'
	export PATH=$PATH:/usr/ccs/bin:/usr/sfw/bin

	# To find opencsw libraries

	/* Compile with:
	* cc -lcurl test.c -o test
	*/
	#include <curl/curl.h>

	int main (int argc, char *argv[]) {
	curl_global_init(CURL_GLOBAL_DEFAULT);
	CURL *handle = curl_easy_init();

	//comment out to fix problem

	# Script to list CRAN servers with https
	library(curl)
	h <- new_handle(timeout_ms = 30000, connecttimeout_ms = 5000)
	mirrors <- read.csv(curl("https://svn.r-project.org/R/trunk/doc/CRAN_mirrors.csv"))
	mirrors$SSL <- vapply(mirrors$URL, function(url){
	https_url <- paste0(sub("^http://", "https://", url), "src/contrib/PACKAGES")
	cat("Trying", https_url, "\n")
	identical(200L, try(curl_fetch_memory(https_url, handle = h)$status))
	}, logical(1))
	subset(mirrors, SSL == TRUE, select = c("Name","URL"))