Skip to content

Instantly share code, notes, and snippets.

@webstory

webstory/README.md

Forked from magnetikonline/README.md
Created November 1, 2022 23:37
Show Gist options
  • Save webstory/3396dbfa242296963a61df85b27d77a1 to your computer and use it in GitHub Desktop.
Save webstory/3396dbfa242296963a61df85b27d77a1 to your computer and use it in GitHub Desktop.
Download ZIP
Backblaze B2 via rclone cheatsheet.
Raw
README.md

Backblaze B2 via rclone cheatsheet

Create B2 remote

  • Where "Key ID" is account.
  • Where "Application key" is key.
  • Using b2 as the remote name and used in all task examples (e.g. as b2:).
$ rclone config

# n) New remote

# name: b2

#XX / Backblaze B2
#   \ "b2"

# Account ID or Application Key ID
# Enter a string value. Press Enter for the default ("").
# account> 

# Application Key
# Enter a string value. Press Enter for the default ("").
# key>

All done, confirm settings:

$ cat ~/.config/rclone/rclone.conf

[b2]
type = b2
account = KEY_ID
key = APPLICATION_KEY
hard_delete = true

Tasks

List buckets

$ rclone lsd b2:

# -1 2019-08-27 23:02:27        -1 BUCKET_01
# -1 2019-08-27 23:02:27        -1 BUCKET_02
# -1 2019-08-27 23:02:27        -1 BUCKET_03

Synchronize local files to bucket

Will perform the following:

  • Push all files from /path/to/local/source to target bucket BUCKET_NAME.
  • Files found in target bucket not in source will be deleted.
  • Files considered identical if file size and modification date match.
  • Progress displayed to terminal, output sent to /path/to/rclone.log.
  • Using --fast-list to action rclone to pull all current target bucket files in a single/minimal number of API calls. Based on the number of target bucket files to consider this may have positive/negative execution time/cost benefit.
  • Use --transfers to control number of parallel file transfers to target bucket, tune based on available upstream bandwidth.
$ rclone sync \
	--fast-list \
	--log-file /path/to/rclone.log \
	--progress \
	--stats-one-line \
	--transfers 32 \
	--verbose \
		/path/to/local/source \
		b2:BUCKET_NAME

Synchronize local files to bucket - checksum

  • Identical to Synchronize local files to bucket, but using --checksum flag means files considered identical if file size and SHA-1 match.
  • A more thorough synchronization, but will take longer to execute as rclone must calculate SHA-1 checksums for every source file - B2 keeps a SHA-1 checksum for every target bucket file, so no additional overhead there.
$ rclone sync \
	--checksum \
	--fast-list \
	--log-file /path/to/rclone.log \
	--progress \
	--stats-one-line \
	--transfers 32 \
	--verbose \
		/path/to/local/source \
		b2:BUCKET_NAME

Verify local files against bucket

Will perform the following:

  • Verify all files at /path/to/local/source against target bucket BUCKET_NAME.
  • Files considered identical if file size and SHA-1 match.
  • To speed up the check, provide the --size-only flag, which will consider files identical if only file sizes match.
  • Progress displayed to terminal, output sent to /path/to/rclone.log.
  • Using --fast-list to action rclone to pull all current target bucket files in a single/minimal number of API calls. Based on the number of target bucket files to consider this may have positive/negative execution time/cost benefit.
$ rclone check \
	--fast-list \
	--log-file /temp/rclone.log \
	--progress \
	--verbose \
		/path/to/local/source \
		b2:BUCKET_NAME

Reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment