Luca Soldaini soldni

🤗Huggingface Model Downloader

Note

(2025-01-08) Add feature for 🏷️Tag(Revision) Selection, contributed by @Bamboo-D.
(2024-12-17) Add feature for ⚡Quick Startup and ⏭️Fast Resume, enabling skipping of downloaded files, while removing the git clone dependency to accelerate file list retrieval.

Considering the lack of multi-threaded download support in the official huggingface-cli, and the inadequate error handling in hf_transfer, This command-line tool leverages curl and aria2c for fast and robust downloading of models and datasets.

Features

⏯️ Resume from breakpoint: You can re-run it or Ctrl+C anytime.

	#!/bin/sh


	# this script does absolutely ZERO error checking. however, it worked
	# for me on a RHEL 6.3 machine on 2012-08-08. clearly, the version numbers
	# and/or URLs should be made variables. cheers, [email protected]


	mkdir mosh
	cd mosh

	import json
	from twitter import *

	# Twitter app credentials
	token = "..."
	token_secret = "..."
	consumer_key = "..."
	consumer_secret = "..."

	# Twitter client

	from typing import *
	import time
	import logging

	def logging_tqdm(
	i,
	*,
	logger: Optional[logging.Logger] = None,
	desc: str = "Working",
	total: Optional[int] = None,

	import asyncio
	import os
	import random
	from collections import deque
	import ssl
	from urllib.parse import urlparse

	import aiohttp
	import polars as pl
	import aiofiles

	#
	# Run like this:
	# beaker experiment logs <experiment id> \| python logs.py
	#

	import sys
	import re
	from collections import defaultdict, Counter
	from typing import List, Dict

	default['sshd']['sshd_config']['AuthenticationMethods'] = 'publickey,keyboard-interactive:pam'
	default['sshd']['sshd_config']['ChallengeResponseAuthentication'] = 'yes'
	default['sshd']['sshd_config']['PasswordAuthentication'] = 'no'