test_mclapply.R

library(parallel)

## small/large obviously depend on the OS and memory available
n_small <- 1e2
n_large <- 1e4

## example from ?svd
hilbert <- function(n) { i <- 1:n; 1 / outer(i - 1, i, "+") }
my_svd <- function(..., n){
  X <- hilbert(n)[, 1:6]
  s <- svd(X)
  return(TRUE)
}

mclapply(1:2, my_svd, n = n_small, mc.cores = 1) # works fine
mclapply(1:2, my_svd, n = n_small, mc.cores = 2) # works fine
mclapply(1:2, my_svd, n = n_large, mc.cores = 1) # works fine
mclapply(1:2, my_svd, n = n_large, mc.cores = 2) # returns a list of NULL

I'm getting here from you R-devel post. FYI, the future framework (disclaimer: I'm the author) detects when forked processes terminate. By using:

library(future.apply)
plan(multicore, workers = 2)
y <- future_lapply(1:2, my_svd, n = n_large)

you more or less will run the same as mclapply() but if one of the "workers" dies, you'll get an informative error instead of NULL, e.g.

Error: Unexpected result (of class 'NULL' != 'FutureResult') retrieved for MulticoreFuture future [...]

BTW, try also with plan(multisession, workers = 2), which will use a PSOCK cluster instead - it might use less memory. See futureverse/future#198 (comment) for a recent discussion on this.

Silly me - I only now see that you're the same person. Oh well.

The multisession version definitely works, but I wondered about forked workers on linux/mac.

The multisession version definitely works, but I wondered about forked workers on linux/mac.

multicore (!= multisession) workers (e.g. plan("multicore")) use forked processes utilizing the same framework as parallel::mclapply() does internally.