Exclude vcs (git, svn, hg) and node_modules from Backblaze backups on Mac

readme.md

Backblaze's bztransmit process loads a file called bzfileids.dat into RAM. This file is a list of all files Backblaze has previously uploaded, including a unique identifier for each file. On most systems, this files is under 100MB in size (paraphrased from Backblaze support rep Zack).

Mine had grown to 6GB. This means that anytime bztransmit runs, it will load this 6GB file into RAM while it is backing up. In doing so it was purging massive ammounts of memory causing behavior like Chrome (usign 10GB of memory on it's own) to hang/beachball for 30 seconds and then refresh all it's windows.

There is no way to alter this behavior once it's begun, aside from starting over with some files excluded. The index needs to be rebuilt from scratch without the excessibe file count, that also means you can't restart and "inherit" a previous backup.

In my case the biggest culprits were .git and node_modules, so I excluded those, started a new backup (transfered licnese) and spent a week hunting for fast internet I could stay connected to over night.

Here are those exclusions

Exclude Version Control System (VCS) folders like .git, .svn and .hg, and exclude node_modules from Backblaze backups on Mac

1. Edit the file at /Library/Backblaze.bzpkg/bzdata/bzexcluderules_editable.xml.
2. Add this rule inside the bzexclusions tag:

<!-- Below here is for node_modules. -->
<excludefname_rule plat="mac" osVers="*"  ruleIsOptional="t" skipFirstCharThenStartsWith="users/" contains_1="/node_modules/" contains_2="*" doesNotContain="*" endsWith="*" hasFileExtension="*" />

<!-- Below here is for Version Control Systems (VCS) for things like: .git, .svn, .hg -->
<excludefname_rule plat="mac" osVers="*"  ruleIsOptional="t" skipFirstCharThenStartsWith="users/" contains_1="/Sites/" contains_2="/.git/" doesNotContain="*" endsWith="*" hasFileExtension="*" />
<excludefname_rule plat="mac" osVers="*"  ruleIsOptional="t" skipFirstCharThenStartsWith="users/" contains_1="/Sites/" contains_2="/.svn/" doesNotContain="*" endsWith="*" hasFileExtension="*" />
<excludefname_rule plat="mac" osVers="*"  ruleIsOptional="t" skipFirstCharThenStartsWith="users/" contains_1="/Sites/" contains_2="/.hg/" doesNotContain="*" endsWith="*" hasFileExtension="*" />

Notes

It can take one to eight hours before new files are backed up. If you want to test this has worked, best to wait a day or so. https://help.backblaze.com/hc/en-us/articles/217665498-Why-hasn-t-Backblaze-backed-up-my-new-files-yet-

Adding this rule does not remove existing files/folders that have already backed up. You need to start over if you want to have a reasonably sized bztransmit RAM footprint.

References

• a. https://help.backblaze.com/hc/en-us/articles/220973007-Advanced-Topic-Setting-Custom-Exclusions-via-XML
• b. https://aoeex.com/phile/backblaze-custom-exclusions/ (examples for Windows)
• c. https://help.backblaze.com/hc/en-us/articles/217666158--Your-bzfileids-dat-file-is-too-large-

Bonus - What about Time Machine?

Check out @stevegrunwell's Asimov for doing something similar with Apple Time Machine https://github.com/stevegrunwell/asimov

What does the contains_1="/Sites/" do? Is that assuming that anything with a git folder in it will be in /Sites/? Seems oddly specific for a generalised gist...?

Omg you solved my question of how to do this with time machine before even asking. How do i find the bztransmit ram size? Is it just the size of the file?

Omg you solved my question of how to do this with time machine before even asking. How do i find the bztransmit ram size? Is it just the size of the file?

Been a while, glad this is still helpful. I probably was just looking at ram usage in Activity Monitor, maybe top (I actually use htop).