dereckmezquita/Makevars

dereckmezquita · 2021-11-17T18:36:01Z

Install multithreaded openMP `data.table` on an `M1` Mac

Installing data.table on an M1 Mac with multithreading support can be frustrating. Here is the process I used to accomplish this.

If you've come across an error something like, cannot link file wrong architecture it is likely you've experienced the same erroneous set up I had:

ld: warning: ignoring file /Library/Frameworks/R.framework/R, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

The solution, install an arm version of R (I had intel) the new version of RStudio which supports the arm version of R, and then setup your Makevars so everything can compile to arm. What was happening before is you had an intel version of R installed and wanted to compile for arm which is incompatible when linking.

Install `arm R`

Install R by using brew the package manager. The current version available is:

For more detailed information on installing data.table visit: data.table installation instructions

brew info R

Warning: Treating r as a formula. For the cask, use homebrew/cask/r
r: stable 4.1.2 (bottled)
Software environment for statistical computing
https://www.r-project.org/
/opt/homebrew/Cellar/r/4.1.2 (2,216 files, 64.6MB) *
  Poured from bottle on 2021-11-17 at 12:23:39
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/r.rb
License: GPL-2.0-or-later
==> Dependencies
Build: pkg-config ✔
Required: cairo ✔, gcc ✔, gettext ✔, jpeg ✔, libffi ✔, libpng ✔, openblas ✔, pcre2 ✔, readline ✔, tcl-tk ✔, xz ✔
==> Analytics
install: 15,733 (30 days), 35,010 (90 days), 137,251 (365 days)
install-on-request: 15,634 (30 days), 34,798 (90 days), 136,002 (365 days)
build-error: 25 (30 days)

brew install R

This will install in the following directory: /opt/homebrew/opt/r/

brew --prefix

/opt/homebrew

Note if you want to have a multi-user set up for your brew installations use this guide: github.com/dereckdemezquita/mac-setup

Install `llvm`

We need llvm to compile data.table:

brew install llvm

Make sure the path is exported in your .zshrc:

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

Makevars

Now that you have R installed (should be the Apple silicon version; brew installs this one for you). You can create the Makevars. This is simply a file with the name Makevars. This files holds the configurations necessary to compile packages on installation or with Rcpp. You don't have to have this configuration active at all times, just for compiling data.table.

The location of this file is: ~/.R/Makevars

Paste the content shown in the gist.

Install `data.table`

Install data.table through an external R console (terminal then type R). Note if R in the terminal window doesn't work this is because you haven't exported the path to R executable to your console.

install.packages("data.table", type = "source")

Installing package into ‘/opt/homebrew/lib/R/4.1/site-library’
(as ‘lib’ is unspecified)

trying URL 'https://cran.microsoft.com/src/contrib/data.table_1.14.2.tar.gz'
Content type 'application/octet-stream' length 5301817 bytes (5.1 MB)
==================================================
downloaded 5.1 MB

* installing *source* package ‘data.table’ ...
** package ‘data.table’ successfully unpacked and MD5 sums checked
** using staged installation
zlib 1.2.11 is available ok
R CMD SHLIB supports OpenMP without any extra hint
** libs
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c cj.c -o cj.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c coalesce.c -o coalesce.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fifelse.c -o fifelse.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c freadR.c -o freadR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c froll.c -o froll.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frollR.c -o frollR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frolladaptive.c -o frolladaptive.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwriteR.c -o fwriteR.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c nafill.c -o nafill.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c nqrecreateindices.c -o nqrecreateindices.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c snprintf.c -o snprintf.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c types.c -o types.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c utils.c -o utils.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -I"/opt/homebrew/Cellar/r/4.1.2/lib/R/include" -DNDEBUG   -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/llvm/include    -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
/opt/homebrew/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/opt/homebrew/Cellar/r/4.1.2/lib/R/lib -L/opt/homebrew/opt/gettext/lib -L/opt/homebrew/opt/llvm/lib -Wl,-rpath,/opt/homebrew/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o nafill.o nqrecreateindices.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -lz -L/opt/homebrew/Cellar/r/4.1.2/lib/R/lib -lR -lintl -Wl,-framework -Wl,CoreFoundation
PKG_CFLAGS =
PKG_LIBS = -lz
if [ "data.table.so" != "datatable.so" ]; then mv data.table.so datatable.so; fi
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
installing to /opt/homebrew/lib/R/4.1/site-library/00LOCK-data.table/00new/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (data.table)

The downloaded source packages are in
	‘/private/var/folders/hv/3571hdb159zggq6whkpdnq400000gq/T/RtmpdJ6Pfu/downloaded_packages’

library("data.table")

data.table 1.14.2 using 4 threads (see ?getDTthreads).  Latest news: r-datatable.com

dereckmezquita · 2021-11-17T19:01:11Z

Versions

R --version

R version 4.1.2 (2021-11-01) -- "Bird Hippie"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.

brew --version

Homebrew 3.3.4
Homebrew/homebrew-core (git revision 9ba60a6af62; last commit 2021-11-17)
Homebrew/homebrew-cask (git revision 8f52edfbb5; last commit 2021-11-17)

RStudio

RStudio
2021.09.1 Build 372
© 2009-2021 RStudio, PBC

dereckmezquita · 2021-11-18T15:28:35Z

RStudio troubleshooting

If you have trouble running RStudio, installed by one user launched from another user on the same machine. Your permissions and ownership might be broken. Try first to re-install RStudio. If this doesn't work the run:

sudo chown -R root RStudio.app
sudo chmod -R 0755 RStudio.app

gregmacfarlane · 2022-04-27T20:58:53Z

As a note, this worked for me after being failed by the data.table instructions.

	# if you downloaded llvm manually above, replace with your chosen NEW_PATH/clang
	LLVM_LOC = /opt/homebrew/opt/llvm
	CC=$(LLVM_LOC)/bin/clang -fopenmp
	CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
	# -O3 should be faster than -O2 (default) level optimisation ..
	CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
	CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
	LDFLAGS=-L/opt/homebrew/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib
	CPPFLAGS=-I/opt/homebrew/opt/gettext/include -I$(LLVM_LOC)/include

dereckmezquita/Makevars

dereckmezquita commented Nov 17, 2021 •

edited

Loading

dereckmezquita commented Nov 17, 2021 •

edited

Loading

dereckmezquita commented Nov 18, 2021

gregmacfarlane commented Apr 27, 2022

dereckmezquita/Makevars

dereckmezquita commented Nov 17, 2021 • edited Loading

Install multithreaded openMP data.table on an M1 Mac

Install arm R

Install llvm

Makevars

Install data.table

dereckmezquita commented Nov 17, 2021 • edited Loading

Versions

dereckmezquita commented Nov 18, 2021

RStudio troubleshooting

gregmacfarlane commented Apr 27, 2022

dereckmezquita commented Nov 17, 2021 •

edited

Loading

Install multithreaded openMP `data.table` on an `M1` Mac

Install `arm R`

Install `llvm`

Install `data.table`

dereckmezquita commented Nov 17, 2021 •

edited

Loading