Exclude node_modules in timemachine

command.sh

find `pwd` -type d -maxdepth 3 -name 'node_modules' | xargs -n 1 tmutil addexclusion

I ended up just taking the list generated from the tool, cleaning it up and pasting its content into /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist.

They had to be formatted like so. All mine were relative to home dir, so I put them in that stanza:

<string>www/bric/node_modules</string>

Then I verified the exclusions are working:

$ tmutil isexcluded /Users/jeremy/www/bric/node_modules
[Excluded]    /Users/jeremy/www/bric/node_modules
$

Alright, now here's a really good idea. Using .gitignore to generate excludes: https://github.com/samuelmeuli/tmignore

@glassdimly glad you found something that worked. Running it locally, with those directories seemed like it worked for me. Are you sure you ran it in the most common parent (i.e. in Users/jeremy/www) or higher?

// modified just to spit out isexcluded
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Included]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Included]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil addexclusion {} \; -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    src/jeremy/www/bric/node_modules

Use this to wrap commands like npm or cargo to exclude any dependency dirs as you go. This prevents situations where Time Machine backs up your dependency dirs after you installed dependencies to a new project but before your regular search command is executed.

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR=$1
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi
}

__npm_wrapper () {
    command npm "$@"
    EXIT_CODE=$?
    tmutil_exclude "node_modules" "package.json"
    return $EXIT_CODE
}

alias npm=__npm_wrapper

Would I need to run this everyone I add a new project as well, or is there a way to add a dynamic exclusion rule?

1 post above yours does this dynamically. Original post needs to be run periodically.

@poma I see, but how do I use it? Is it a bash script? Do I would add it to ~/.zshrc (for instance)? Would it be possible to adjust for yarn as well?

Yes add this to .zshrc for example. For yarn just copy alias and wrapper.

@poma awesome. What does the second parameter, package.json do? I'm trying to add another wrapper for cocoapods, where the equivalent would presumably be tmutil_exclude "Pods" "Podfile", and wasn't sure what it's used for.

It only excludes node_modules if a file named package.json is present in the same dir. Might not be as important for distinctly named node_modules dir but it's more relevant for example for rust's target dir which can have the same name as something useful outside of rust projects

@poma found the way which is LEAST obtrusive on all system resources. I extended this script to parse everything in the .gitignore. I am a NOOB in bash scripting and I'm sure it can be made simpler and I'd love to see that. Here it goes:

I have also added handling of .nosync for iCloud Drive

TRIMMED=""

trim() {
  local s2 s="$*"
  until s2="${s#[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  until s2="${s%[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  #echo "====${s}===="
  TRIMMED="$s"
}

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR="$1"
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi

    # Handles iCloud Drive
    CLOUD_FILE="${DIR}/.nosync"

    if [ -d "$DIR" ] && [ ! -f "${CLOUD_FILE}" ]; then
        touch "${CLOUD_FILE}"
        echo ".nosync added at ${CLOUD_FILE}"
    fi
}

__npm_wrapper () {
    # command pnpm "$@"
    EXIT_CODE=$?

    input=".gitignore"
    while IFS= read -r line
    do
        if [ ${line:0:1} = "/" ]; then
            trim ".${line}"
            tmutil_exclude "${TRIMMED}" "package.json"
        fi
    done < "$input"

    return $EXIT_CODE
}

alias pnpm=__npm_wrapper

Trying to use fd to replace find

fd -t d -d 5 -a --no-ignore --prune node_modules | xargs -I {} tmutil addexclusion {}

This thread shows up fairly high in my searches for excluding python virtualenvs from timemachine.

Just wanted to point out for folks looking to dig a little deeper that there is some good info out there on the xattr that's being set on the item leading to exclusion, including faster ways to find these files (with mdfind -- though this will skip files that are not indexed by spotlight) and likely some other opportunities to set the attribute directly with xattr if desired:

• https://apple.stackexchange.com/a/25833/27415
• https://www.heissenberger.at/en/blog/macos-exclude-node_modules-folder-from-time-machine/

Also, it appears that tmutil addexclusion supports multiple args:

$ mkdir -p {foo,bar}
$ ls
bar  foo
$ xattr *
$
$ tmutil addexclusion ./foo ./bar
$ xattr *
bar: com.apple.metadata:com_apple_backup_excludeItem
foo: com.apple.metadata:com_apple_backup_excludeItem

which means that many of the above commands could probably be done much more efficiently with find ... -exec tmutil addexclusion {} +, which collects arguments and runs the command once with collected arguments, as opposed to \; or piping to xargs / parallel, which will be separate invocations of tmutil.

EDIT: The equivalent for fd is --exec-batch