-
-
Save caniszczyk/3856584 to your computer and use it in GitHub Desktop.
| curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}' |
- Into $ORG folder
mkdir $ORG; cd $ORG - Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone
- Into $ORG folder
mkdir $ORG; cd $ORG- Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone
Github stopped https clones (username and passwords needed for every clone so instead use ssh clone
gh repo list $ORG --limit 9999 --json sshUrl | jq '.[]|.sshUrl' | xargs -n1 git clone
This also works whilst using gh command and auth instead of git
gh repo list $ORG --limit 9999 --json nameWithOwner | jq '.[]|.nameWithOwner' | xargs -n1 gh repo clone
this can be even simpler, assuming < 100 repos:
curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone
Just in case, if you need ssh url and a bit more than 100 repos:
curl -s "https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100&page=<1,2...>" | jq '.[].ssh_url' | xargs -n 1 git clone
Hi all
I wrote a simple script to clone all the repositories from GitHub, either from a user or whole organization. Please check it out, I'm sure you'll find it useful
Here's a fully working script with things from here, SO + some improvements by me. Needs gh cli.
If the repo exists it will check out the main/master branch and pull.
#!/bin/bash
ORG="myorg" # Your organization
CLONE_PATH="/home/user/Documents/..." # Path in your filesystem where you want to clone the repos
PER_PAGE=100 # per_page maxes out at 100
TOKEN="" # Set your token from GH
for ((PAGE=1; ; PAGE+=1)); do
# Page 0 and 1 are the same
# Change authorization method as needed
INPUT=$(curl -H "Authorization: token $TOKEN" -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[].clone_url")
if [[ -z "$INPUT" ]]; then
echo "All repos processed, stopped at page=$PAGE"
exit
fi
while read REPO_URL ; do
echo $REPO_URL
temp=${REPO_URL##*/}
repo_name=${temp%.*}
gh repo clone "$REPO_URL" "$CLONE_PATH/$repo_name" -- -q 2>/dev/null || (
cd "$CLONE_PATH/$repo_name"
# Handle case where local checkout is on a non-main/master branch
# - ignore checkout errors because some repos may have zero commits,
# so no main or master
git checkout -q main 2>/dev/null || true
git checkout -q master 2>/dev/null || true
git pull -q
)
done < <(echo "$INPUT")
done
And here's a "Perl-as-glue" version, that works on systems with command-line curl and git: https://gist.github.com/ology/b6087be12131bae0441d40d87505e658 :D
- | xargs -n1 git clone
we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone
though, it's asking for ssh passcode... any idea how to overcome that?
meh.. I have moved to ghorg. implemented it does everything I could do with my own scripts and is 10 times as fast to boot gh
…
On Mon, Jan 15, 2024 at 10:23 AM Vaisakh K M @.> wrote: @.* commented on this gist. ------------------------------ - | xargs -n1 git clone we can essentially remove jq and directly use gh gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone — Reply to this email directly, view it on GitHub https://gist.github.com/caniszczyk/3856584#gistcomment-4832274 or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNZZRED6QI6M2V6ZDBT4XTYOVXZXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTQNJWGU4DJJ3UOJUWOZ3FOKTGG4TFMF2GK . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .
thanks.. i didn't know that existed...
Windows / PowerShell
gh repo list theUSERorORG --json=nameWithOwner --limit 1000 -q ".[].nameWithOwner" | %{gh repo clone $_}
- | xargs -n1 git clone
we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clonethough, it's asking for ssh passcode... any idea how to overcome that?
If you don't want to use SSH protocol, just run this command1:
gh auth login
It's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:
What is your preferred protocol?
> http
sshSelect ssh, and it gh will save your preference for future gh commands. I'd recommend doing it this way so that gh can manage your credentials, BUT you can probably skip all the above by running2:
gh config set git_protocol ssh"gh config set git_protocol " (see diff below).ssh http
- gh config set git_protocol ssh
# use the below for http protocol (to bypass ssh requirement)
+ gh config set git_protocol http
Doy! 😄
Footnotes
I just want to say that this thread is a great example of the open source community. An 8-year thread on a platform, with everybody posting little bits of code and share with others for free, on the platform itself.
I agree.
To clone all repos from an organisation using HTTP
gh auth login
gh repo list <organisation> --json=url --limit 1000 -q ".[].url" | xargs -n1 git clone
1- Install jq, a lightweight and flexible command-line JSON processor
2- Fetch a list of repositories from the specified GitHub organization and clone them
brew install jq
curl -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1
Replace <ORG_NAME> with the actual name of the GitHub organization
- Use curl to retrieve the repository data from the GitHub API with a limit of 100 repositories per page
- Pipe the result to jq to extract the HTML URLs of all repositories
- Use xargs to clone each repository URL to the local machine with a shallow copy (depth=1), which retrieves only the latest commit history
- | xargs -n1 git clone
we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clonethough, it's asking for ssh passcode... any idea how to overcome that?
If you don't want to use SSH protocol, just run this command1:
gh auth loginIt's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:
What is your preferred protocol? > http sshSelect ssh, and it
ghwill save your preference for futureghcommands. I'd recommend doing it this way so thatghcan manage your credentials, BUT you can probably skip all the above by running2:gh config set git_protocol sshOops, typo! Instead of the one-liner immediately above, to use
httpinstead ofsshthe command should read"
gh config set git_protocol ssh **http**" (see diff below).- gh config set git_protocol ssh # use the below for http protocol (to bypass ssh requirement) + gh config set git_protocol httpDoy! 😄
Footnotes
1. [`gh auth login` (cli.github.com/manual)](https://cli.github.com/manual/gh_auth_login) [↩](#user-content-fnref-1-9e7baee31a028e56fce81af59af26625) 2. [`gh config set` (cli.github.com/manual)](https://cli.github.com/manual/gh_config_set#:~:text=%24%20gh%20config%20set%20git_protocol%20ssh%20%2D%2Dhost%20github.com) [↩](#user-content-fnref-2-9e7baee31a028e56fce81af59af26625)
got around by using ssh agent
eval "$(ssh-agent -s)" # Start the ssh-agent
ssh-add ~/.ssh/id_rsa # Add your SSH key:
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone
clone all repos