Skip to content

Instantly share code, notes, and snippets.

View ShanSabri's full-sized avatar

Shan Sabri ShanSabri

View GitHub Profile
@ShanSabri
ShanSabri / parallelRDS.R
Created February 21, 2020 01:04 — forked from retrography/parallelRDS.R
Parallelize RDS compression/decompression to improve serialization performance in R
# The functions below use parallelized versions of gzip, xz, and bzip2 to
# improve compression/decompression performance of RDS serialization in R.
# Each function searches for the appropriate program (based on the required
# compression format) and if found, offloads the compression handling to the
# external program and therefore leaves R free to do the data import/export.
# The two main functions (saveRDS and readRDS) mask R's native read and write
# functions. The functions have been only tested on macOS, but they must work
# on any Linux/Unix.
#
# Requires the following packages: pxz, pbzip2, and pigz.