peterdemartini/command.sh

maxxcrawford · 2018-06-15T21:27:20Z

Can this command be run from the root user directory, or is there a specific directory?

tomwidmer · 2018-06-21T11:07:51Z

To run it from a higher level if you have node projects in various subfolders, you can do:

find `pwd` -type d -maxdepth 8 -prune -name 'node_modules' | xargs -n 1 tmutil addexclusion

See also https://github.com/stevegrunwell/asimov

gimdongwoo · 2018-07-09T09:38:04Z

I think, this is work.

find `pwd` -type d -maxdepth 8 -name 'node_modules' | xargs -n 1 tmutil addexclusion

What is -prune?

fuerst · 2018-08-01T19:43:34Z

No need to pipe to xargs, just use find -exec like this:

find $(pwd) -type d -name node_modules -maxdepth 3 -exec tmutil addexclusion {} \;

Using $() for command substitution because back ticks are deprecated.

osopolar · 2018-08-15T20:36:46Z

And how to exclude only the node_modules directory but not it's sub-directories for example only /Users/Example/Projects/test/node_modules but not /Users/Example/Projects/test/node_modules/write/node_modules? I guess it does not harm but it helps to keep the list cleaner.

osopolar · 2018-08-15T23:30:41Z

You may use launchd to scan your project folder on a regular basis to exclude new node_modules directories, see https://gist.github.com/osopolar/9d8e8188091f7cbbfcee924ccfbb32dd

qgy18 · 2018-08-23T05:42:18Z

Thank you for your code, it helps a lot to me. At last, I use node.js to 'exclude only the node_modules directory but not it's sub-directories for example only'

/**
* exclude some directory from timemachine
* usage: 
* node find_node_modules.js | sudo xargs -n 1 tmutil addexclusion -p
* defail for -p flag: https://www.macworld.com/article/2033804/control-time-machine-from-the-command-line.html
*/

const dir = '/Users/your_name'; // base directory
const exclusionNames = ['node_modules', '.git']; 

const { readdirSync, statSync } = require('fs');
const { sep } = require('path');

function findNodeModules(dir, list) {
    list = list || [];

    readdirSync(dir).forEach(subDir => {
        let newDir = `${dir}${sep}${subDir}`;
        
        if(statSync(newDir).isDirectory()) {
            if(exclusionNames.includes(subDir.toLowerCase())) {
                list.push(newDir);
                return;
            }

            findNodeModules(newDir, list);
        }
    });
}

let list = [];
findNodeModules(dir, list);

console.log(list.join('\n'));

giano · 2018-08-29T10:08:24Z

This tool is great for this: https://github.com/stevegrunwell/asimov

dev01d · 2019-02-03T21:08:37Z

@gimdongwoo The -prune argument will prevent find from descending into the node_modules directory once found, which otherwise litters the exclude list. So if youre like me and have a ton of projects in your working directory, rather than trying to guess the depth in the file structure with -maxdepth you can use -prune to search the whole directory to add clean entries.

This version of @fuerst's command will get you only the top level node_modules directory of each occurrence and functions like adding them in the Time Machine GUI.

find $(pwd) -type d -name node_modules -prune -exec tmutil addexclusion {} \;

techouse · 2019-05-09T08:31:52Z

I wrote a simple bash script which you can simply put in your development/work directory and run or add to your crontab.

#!/usr/bin/env bash

EXCLUDED_DIRECTORIES=( "env" "node_modules" "vendor" "venv" )

WORK_DIR=$(dirname "${BASH_SOURCE[0]}")
WORK_DIR=$(realpath "${WORK_DIR}")

for EXCLUDED_DIRECTORY in "${EXCLUDED_DIRECTORIES[@]}"; do
    find ${WORK_DIR} -maxdepth 2 -type d -name ${EXCLUDED_DIRECTORY} -prune -exec tmutil addexclusion {} \;
done

chen86860 · 2019-05-28T12:19:24Z

No need to pipe to xargs, just use find -exec like this:
find $(pwd) -type d -name node_modules -maxdepth 3 -exec tmutil addexclusion {} \;
Using $() for command substitution because back ticks are deprecated.

I cannot add exclude folders using your script but after some of arguments:

find $(pwd) -type d -name node_modules -maxdepth 3 -exec tmutil addexclusion -p {} \;

Thanks anyway.

moekhalil · 2019-06-21T05:04:52Z

For some reason, prepending -prune would return an empty list.

Got this one for you guys/gals, working with prune and confirmation.

find `pwd` -name 'node_modules' -prune -type d -exec tmutil addexclusion {} \; -exec tmutil isexcluded {} \;

edit: The directories are showing as excluded when you run tmutil isexcluded <dir> (as the included command does) on each directory.

They do not however populate the list inside the table located at:

"System Preferences..." >> "Time Machine" Preferences >> "Options..." >> "Exclude these items from backup:".

_{^{I am almost certain that's expected and it should all work fine, but...}}^{_{SHOULD IT NOT WORK}} _{^{I will report back}}

codexp · 2019-07-25T08:20:27Z

@giano thank you for the tool recommendation, it is great!

glassdimly · 2019-10-05T19:56:50Z

Using this tool from @moekhalil, only some of the dirs confirm as excluded:

[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Included]    /Users/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /Users/jeremy/www/modules/node/deps/npm/node_modules
[Included]    /Users/jeremy/www/bric/node_modules

glassdimly · 2019-10-05T20:06:56Z

I ended up just taking the list generated from the tool, cleaning it up and pasting its content into /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist.

They had to be formatted like so. All mine were relative to home dir, so I put them in that stanza:

<string>www/bric/node_modules</string>

Then I verified the exclusions are working:

$ tmutil isexcluded /Users/jeremy/www/bric/node_modules
[Excluded]    /Users/jeremy/www/bric/node_modules
$

glassdimly · 2019-10-05T20:11:29Z

Alright, now here's a really good idea. Using .gitignore to generate excludes: https://github.com/samuelmeuli/tmignore

moekhalil · 2019-10-05T20:12:03Z

@glassdimly glad you found something that worked. Running it locally, with those directories seemed like it worked for me. Are you sure you ran it in the most common parent (i.e. in Users/jeremy/www) or higher?

// modified just to spit out isexcluded
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Included]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Included]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Included]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil addexclusion {} \; -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    /src/jeremy/www/bric/node_modules
🛡  src/jeremy  ➜  find `pwd` -name 'node_modules' -prune -type d -exec tmutil isexcluded {} \;
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/not-found/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/parent/node_modules
[Excluded]    /src/jeremy/www/modules/node/test/fixtures/module-require/child/node_modules
[Excluded]    /src/jeremy/www/modules/node/deps/npm/node_modules
[Excluded]    src/jeremy/www/bric/node_modules

poma · 2020-04-07T12:05:38Z

Use this to wrap commands like npm or cargo to exclude any dependency dirs as you go. This prevents situations where Time Machine backs up your dependency dirs after you installed dependencies to a new project but before your regular search command is executed.

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR=$1
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi
}

__npm_wrapper () {
    command npm "$@"
    EXIT_CODE=$?
    tmutil_exclude "node_modules" "package.json"
    return $EXIT_CODE
}

alias npm=__npm_wrapper

AdamGerthel · 2020-12-14T10:05:31Z

Would I need to run this everyone I add a new project as well, or is there a way to add a dynamic exclusion rule?

poma · 2020-12-14T11:02:19Z

1 post above yours does this dynamically. Original post needs to be run periodically.

AdamGerthel · 2020-12-14T11:06:35Z

@poma I see, but how do I use it? Is it a bash script? Do I would add it to ~/.zshrc (for instance)? Would it be possible to adjust for yarn as well?

poma · 2020-12-14T12:10:02Z

Yes add this to .zshrc for example. For yarn just copy alias and wrapper.

AdamGerthel · 2020-12-14T12:52:20Z

@poma awesome. What does the second parameter, package.json do? I'm trying to add another wrapper for cocoapods, where the equivalent would presumably be tmutil_exclude "Pods" "Podfile", and wasn't sure what it's used for.

poma · 2020-12-14T16:56:38Z

It only excludes node_modules if a file named package.json is present in the same dir. Might not be as important for distinctly named node_modules dir but it's more relevant for example for rust's target dir which can have the same name as something useful outside of rust projects

tomitrescak · 2022-03-09T04:09:36Z

@poma found the way which is LEAST obtrusive on all system resources. I extended this script to parse everything in the .gitignore. I am a NOOB in bash scripting and I'm sure it can be made simpler and I'd love to see that. Here it goes:

I have also added handling of .nosync for iCloud Drive

TRIMMED=""

trim() {
  local s2 s="$*"
  until s2="${s#[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  until s2="${s%[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
  #echo "====${s}===="
  TRIMMED="$s"
}

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR="$1"
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi

    # Handles iCloud Drive
    CLOUD_FILE="${DIR}/.nosync"

    if [ -d "$DIR" ] && [ ! -f "${CLOUD_FILE}" ]; then
        touch "${CLOUD_FILE}"
        echo ".nosync added at ${CLOUD_FILE}"
    fi
}

__npm_wrapper () {
    # command pnpm "$@"
    EXIT_CODE=$?

    input=".gitignore"
    while IFS= read -r line
    do
        if [ ${line:0:1} = "/" ]; then
            trim ".${line}"
            tmutil_exclude "${TRIMMED}" "package.json"
        fi
    done < "$input"

    return $EXIT_CODE
}

alias pnpm=__npm_wrapper

dj1020 · 2023-06-06T03:34:48Z

Trying to use fd to replace find

fd -t d -d 5 -a --no-ignore --prune node_modules | xargs -I {} tmutil addexclusion {}

n8henrie · 2023-12-14T15:03:13Z

This thread shows up fairly high in my searches for excluding python virtualenvs from timemachine.

Just wanted to point out for folks looking to dig a little deeper that there is some good info out there on the xattr that's being set on the item leading to exclusion, including faster ways to find these files (with mdfind -- though this will skip files that are not indexed by spotlight) and likely some other opportunities to set the attribute directly with xattr if desired:

n8henrie · 2023-12-14T15:09:33Z

Also, it appears that tmutil addexclusion supports multiple args:

$ mkdir -p {foo,bar}
$ ls
bar  foo
$ xattr *
$
$ tmutil addexclusion ./foo ./bar
$ xattr *
bar: com.apple.metadata:com_apple_backup_excludeItem
foo: com.apple.metadata:com_apple_backup_excludeItem

which means that many of the above commands could probably be done much more efficiently with find ... -exec tmutil addexclusion {} +, which collects arguments and runs the command once with collected arguments, as opposed to \; or piping to xargs / parallel, which will be separate invocations of tmutil.

EDIT: The equivalent for fd is --exec-batch

peterdemartini/command.sh

maxxcrawford commented Jun 15, 2018

tomwidmer commented Jun 21, 2018 •

edited

Loading

gimdongwoo commented Jul 9, 2018 •

edited

Loading

fuerst commented Aug 1, 2018

osopolar commented Aug 15, 2018

osopolar commented Aug 15, 2018

qgy18 commented Aug 23, 2018

giano commented Aug 29, 2018

dev01d commented Feb 3, 2019

techouse commented May 9, 2019 •

edited

Loading

chen86860 commented May 28, 2019

moekhalil commented Jun 21, 2019 •

edited

Loading

codexp commented Jul 25, 2019

glassdimly commented Oct 5, 2019

glassdimly commented Oct 5, 2019

glassdimly commented Oct 5, 2019

moekhalil commented Oct 5, 2019 •

edited

Loading

poma commented Apr 7, 2020 •

edited

Loading

AdamGerthel commented Dec 14, 2020

poma commented Dec 14, 2020

AdamGerthel commented Dec 14, 2020

poma commented Dec 14, 2020

AdamGerthel commented Dec 14, 2020 •

edited

Loading

poma commented Dec 14, 2020 •

edited

Loading

tomitrescak commented Mar 9, 2022 •

edited

Loading

dj1020 commented Jun 6, 2023

n8henrie commented Dec 14, 2023 •

edited

Loading

n8henrie commented Dec 14, 2023 •

edited

Loading

peterdemartini/command.sh

maxxcrawford commented Jun 15, 2018

tomwidmer commented Jun 21, 2018 • edited Loading

gimdongwoo commented Jul 9, 2018 • edited Loading

fuerst commented Aug 1, 2018

osopolar commented Aug 15, 2018

osopolar commented Aug 15, 2018

qgy18 commented Aug 23, 2018

giano commented Aug 29, 2018

dev01d commented Feb 3, 2019

techouse commented May 9, 2019 • edited Loading

chen86860 commented May 28, 2019

moekhalil commented Jun 21, 2019 • edited Loading

I am almost certain that's expected and it should all work fine, but...SHOULD IT NOT WORK I will report back

codexp commented Jul 25, 2019

glassdimly commented Oct 5, 2019

glassdimly commented Oct 5, 2019

glassdimly commented Oct 5, 2019

moekhalil commented Oct 5, 2019 • edited Loading

poma commented Apr 7, 2020 • edited Loading

AdamGerthel commented Dec 14, 2020

poma commented Dec 14, 2020

AdamGerthel commented Dec 14, 2020

poma commented Dec 14, 2020

AdamGerthel commented Dec 14, 2020 • edited Loading

poma commented Dec 14, 2020 • edited Loading

tomitrescak commented Mar 9, 2022 • edited Loading

dj1020 commented Jun 6, 2023

n8henrie commented Dec 14, 2023 • edited Loading

n8henrie commented Dec 14, 2023 • edited Loading

tomwidmer commented Jun 21, 2018 •

edited

Loading

gimdongwoo commented Jul 9, 2018 •

edited

Loading

techouse commented May 9, 2019 •

edited

Loading

moekhalil commented Jun 21, 2019 •

edited

Loading

_{^{I am almost certain that's expected and it should all work fine, but...}}^{_{SHOULD IT NOT WORK}} _{^{I will report back}}

moekhalil commented Oct 5, 2019 •

edited

Loading

poma commented Apr 7, 2020 •

edited

Loading

AdamGerthel commented Dec 14, 2020 •

edited

Loading

poma commented Dec 14, 2020 •

edited

Loading

tomitrescak commented Mar 9, 2022 •

edited

Loading

n8henrie commented Dec 14, 2023 •

edited

Loading

n8henrie commented Dec 14, 2023 •

edited

Loading