Exclude node_modules in timemachine

command.sh

find `pwd` -type d -maxdepth 3 -name 'node_modules' | xargs -n 1 tmutil addexclusion

You may use launchd to scan your project folder on a regular basis to exclude new node_modules directories, see https://gist.github.com/osopolar/9d8e8188091f7cbbfcee924ccfbb32dd

Thank you for your code, it helps a lot to me. At last, I use node.js to 'exclude only the node_modules directory but not it's sub-directories for example only'

/**
* exclude some directory from timemachine
* usage: 
* node find_node_modules.js | sudo xargs -n 1 tmutil addexclusion -p
* defail for -p flag: https://www.macworld.com/article/2033804/control-time-machine-from-the-command-line.html
*/

const dir = '/Users/your_name'; // base directory
const exclusionNames = ['node_modules', '.git']; 

const { readdirSync, statSync } = require('fs');
const { sep } = require('path');

function findNodeModules(dir, list) {
    list = list || [];

    readdirSync(dir).forEach(subDir => {
        let newDir = `${dir}${sep}${subDir}`;
        
        if(statSync(newDir).isDirectory()) {
            if(exclusionNames.includes(subDir.toLowerCase())) {
                list.push(newDir);
                return;
            }

            findNodeModules(newDir, list);
        }
    });
}

let list = [];
findNodeModules(dir, list);

console.log(list.join('\n'));

This tool is great for this: https://github.com/stevegrunwell/asimov

@gimdongwoo The -prune argument will prevent find from descending into the node_modules directory once found, which otherwise litters the exclude list. So if youre like me and have a ton of projects in your working directory, rather than trying to guess the depth in the file structure with -maxdepth you can use -prune to search the whole directory to add clean entries.

This version of @fuerst's command will get you only the top level node_modules directory of each occurrence and functions like adding them in the Time Machine GUI.

find $(pwd) -type d -name node_modules -prune -exec tmutil addexclusion {} \;

I wrote a simple bash script which you can simply put in your development/work directory and run or add to your crontab.

#!/usr/bin/env bash

EXCLUDED_DIRECTORIES=( "env" "node_modules" "vendor" "venv" )

WORK_DIR=$(dirname "${BASH_SOURCE[0]}")
WORK_DIR=$(realpath "${WORK_DIR}")

for EXCLUDED_DIRECTORY in "${EXCLUDED_DIRECTORIES[@]}"; do
    find ${WORK_DIR} -maxdepth 2 -type d -name ${EXCLUDED_DIRECTORY} -prune -exec tmutil addexclusion {} \;
done

No need to pipe to xargs, just use find -exec like this:

find $(pwd) -type d -name node_modules -maxdepth 3 -exec tmutil addexclusion {} \;

Using $() for command substitution because back ticks are deprecated.

I cannot add exclude folders using your script but after some of arguments:

find $(pwd) -type d -name node_modules -maxdepth 3 -exec tmutil addexclusion -p {} \;

Thanks anyway.

For some reason, prepending -prune would return an empty list.

Got this one for you guys/gals, working with prune and confirmation.

find `pwd` -name 'node_modules' -prune -type d -exec tmutil addexclusion {} \; -exec tmutil isexcluded {} \;

edit: The directories are showing as excluded when you run tmutil isexcluded <dir> (as the included command does) on each directory.

They do not however populate the list inside the table located at:

"System Preferences..." >> "Time Machine" Preferences >> "Options..." >> "Exclude these items from backup:".

_{^{I am almost certain that's expected and it should all work fine, but...SHOULD IT NOT WORK I will report back}}

@giano thank you for the tool recommendation, it is great!

Using this tool from @moekhalil, only some of the dirs confirm as excluded:

[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /Users/jeremy/www/modules/node/deps/npm/node_modules
[Included]    /Users/jeremy/www/bric/node_modules

I ended up just taking the list generated from the tool, cleaning it up and pasting its content into /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist.

They had to be formatted like so. All mine were relative to home dir, so I put them in that stanza:

<string>www/bric/node_modules</string>

Then I verified the exclusions are working:

$ tmutil isexcluded /Users/jeremy/www/bric/node_modules
[Excluded]    /Users/jeremy/www/bric/node_modules
$

Alright, now here's a really good idea. Using .gitignore to generate excludes: https://github.com/samuelmeuli/tmignore

@glassdimly glad you found something that worked. Running it locally, with those directories seemed like it worked for me. Are you sure you ran it in the most common parent (i.e. in Users/jeremy/www) or higher?

// modified just to spit out isexcluded
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Included]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Included]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil addexclusion {} \; -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    src/jeremy/www/bric/node_modules

Use this to wrap commands like npm or cargo to exclude any dependency dirs as you go. This prevents situations where Time Machine backs up your dependency dirs after you installed dependencies to a new project but before your regular search command is executed.

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR=$1
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi
}

__npm_wrapper () {
    command npm "$@"
    EXIT_CODE=$?
    tmutil_exclude "node_modules" "package.json"
    return $EXIT_CODE
}

alias npm=__npm_wrapper

Would I need to run this everyone I add a new project as well, or is there a way to add a dynamic exclusion rule?

1 post above yours does this dynamically. Original post needs to be run periodically.

@poma I see, but how do I use it? Is it a bash script? Do I would add it to ~/.zshrc (for instance)? Would it be possible to adjust for yarn as well?

Yes add this to .zshrc for example. For yarn just copy alias and wrapper.

@poma awesome. What does the second parameter, package.json do? I'm trying to add another wrapper for cocoapods, where the equivalent would presumably be tmutil_exclude "Pods" "Podfile", and wasn't sure what it's used for.

It only excludes node_modules if a file named package.json is present in the same dir. Might not be as important for distinctly named node_modules dir but it's more relevant for example for rust's target dir which can have the same name as something useful outside of rust projects

@poma found the way which is LEAST obtrusive on all system resources. I extended this script to parse everything in the .gitignore. I am a NOOB in bash scripting and I'm sure it can be made simpler and I'd love to see that. Here it goes:

I have also added handling of .nosync for iCloud Drive

TRIMMED=""

trim() {
  local s2 s="$*"
  until s2="${s#[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  until s2="${s%[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  #echo "====${s}===="
  TRIMMED="$s"
}

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR="$1"
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi

    # Handles iCloud Drive
    CLOUD_FILE="${DIR}/.nosync"

    if [ -d "$DIR" ] && [ ! -f "${CLOUD_FILE}" ]; then
        touch "${CLOUD_FILE}"
        echo ".nosync added at ${CLOUD_FILE}"
    fi
}

__npm_wrapper () {
    # command pnpm "$@"
    EXIT_CODE=$?

    input=".gitignore"
    while IFS= read -r line
    do
        if [ ${line:0:1} = "/" ]; then
            trim ".${line}"
            tmutil_exclude "${TRIMMED}" "package.json"
        fi
    done < "$input"

    return $EXIT_CODE
}

alias pnpm=__npm_wrapper

Trying to use fd to replace find

fd -t d -d 5 -a --no-ignore --prune node_modules | xargs -I {} tmutil addexclusion {}

This thread shows up fairly high in my searches for excluding python virtualenvs from timemachine.

Just wanted to point out for folks looking to dig a little deeper that there is some good info out there on the xattr that's being set on the item leading to exclusion, including faster ways to find these files (with mdfind -- though this will skip files that are not indexed by spotlight) and likely some other opportunities to set the attribute directly with xattr if desired:

• https://apple.stackexchange.com/a/25833/27415
• https://www.heissenberger.at/en/blog/macos-exclude-node_modules-folder-from-time-machine/

Also, it appears that tmutil addexclusion supports multiple args:

$ mkdir -p {foo,bar}
$ ls
bar  foo
$ xattr *
$
$ tmutil addexclusion ./foo ./bar
$ xattr *
bar: com.apple.metadata:com_apple_backup_excludeItem
foo: com.apple.metadata:com_apple_backup_excludeItem

which means that many of the above commands could probably be done much more efficiently with find ... -exec tmutil addexclusion {} +, which collects arguments and runs the command once with collected arguments, as opposed to \; or piping to xargs / parallel, which will be separate invocations of tmutil.

EDIT: The equivalent for fd is --exec-batch