moontidef relic-yuexi

🤗Huggingface Model Downloader

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, this command-line tool smartly utilizes wget or aria2 for LFS files and git clone for the rest.

Features

⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.
🚀 Multi-threaded Download: Utilize multiple threads to speed up the download process.
🚫 File Exclusion: Use --exclude or --include to skip or specify files, save time for models with duplicate formats (e.g., *.bin or *.safetensors).
🔐 Auth Support: For gated models that require Huggingface login, use --hf_username and --hf_token to authenticate.
🪞 Mirror Site Support: Set up with HF_ENDPOINT environment variable.

	// Updated: Aug. 20, 2024
	// Run: node testRegex.js whatever.txt
	// Live demo: https://jina.ai/tokenizer
	// LICENSE: Apache-2.0 (https://www.apache.org/licenses/LICENSE-2.0)
	// COPYRIGHT: Jina AI
	const fs = require('fs');
	const util = require('util');

	// Define variables for magic numbers
	const MAX_HEADING_LENGTH = 7;