Skip to content

Instantly share code, notes, and snippets.

@caniszczyk

caniszczyk/clone-all-twitter-github-repos.sh

Created October 9, 2012 04:25
Show Gist options
  • Save caniszczyk/3856584 to your computer and use it in GitHub Desktop.
Save caniszczyk/3856584 to your computer and use it in GitHub Desktop.
Download ZIP
Clone all repos from a GitHub organization
Raw
clone-all-twitter-github-repos.sh
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@octohedron
Copy link

Here's a fully working script with things from here, SO + some improvements by me. Needs gh cli.

If the repo exists it will check out the main/master branch and pull.

#!/bin/bash

ORG="myorg" # Your organization
CLONE_PATH="/home/user/Documents/..." # Path in your filesystem where you want to clone the repos
PER_PAGE=100 # per_page maxes out at 100
TOKEN="" # Set your token from GH

for ((PAGE=1; ; PAGE+=1)); do
  # Page 0 and 1 are the same
  # Change authorization method as needed
  INPUT=$(curl -H "Authorization: token $TOKEN" -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[].clone_url")
  if [[ -z "$INPUT" ]]; then
    echo "All repos processed, stopped at page=$PAGE"
    exit
  fi
  while read REPO_URL ; do
    echo $REPO_URL
    temp=${REPO_URL##*/}
    repo_name=${temp%.*}
    gh repo clone "$REPO_URL" "$CLONE_PATH/$repo_name" -- -q 2>/dev/null || (
        cd "$CLONE_PATH/$repo_name"
        # Handle case where local checkout is on a non-main/master branch
        # - ignore checkout errors because some repos may have zero commits, 
        # so no main or master
        git checkout -q main 2>/dev/null || true
        git checkout -q master 2>/dev/null || true
        git pull -q
    )
  done < <(echo "$INPUT")
done

@ology
Copy link

ology commented Nov 4, 2023

And here's a "Perl-as-glue" version, that works on systems with command-line curl and git: https://gist.github.com/ology/b6087be12131bae0441d40d87505e658 :D

@Vaisakhkm2625
Copy link

Vaisakhkm2625 commented Jan 15, 2024 •
edited
Loading

  • | xargs -n1 git clone

we can essentially remove jq and directly use gh

gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

though, it's asking for ssh passcode... any idea how to overcome that?

@hotelzululima
Copy link

hotelzululima commented Jan 16, 2024 via email

@Vaisakhkm2625
Copy link

meh.. I have moved to ghorg. implemented it does everything I could do with my own scripts and is 10 times as fast to boot gh
…
On Mon, Jan 15, 2024 at 10:23 AM Vaisakh K M @.> wrote: @.* commented on this gist. ------------------------------ - | xargs -n1 git clone we can essentially remove jq and directly use gh gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone — Reply to this email directly, view it on GitHub https://gist.github.com/caniszczyk/3856584#gistcomment-4832274 or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNZZRED6QI6M2V6ZDBT4XTYOVXZXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTQNJWGU4DJJ3UOJUWOZ3FOKTGG4TFMF2GK . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

thanks.. i didn't know that existed...

@davidglezz
Copy link

Windows / PowerShell

gh repo list theUSERorORG --json=nameWithOwner --limit 1000 -q ".[].nameWithOwner"  | %{gh repo clone $_}

@pa-0
Copy link

pa-0 commented Jun 2, 2024 •
edited
Loading

  • | xargs -n1 git clone

we can essentially remove jq and directly use gh

gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

though, it's asking for ssh passcode... any idea how to overcome that?

@Vaisakhkm2625,

If you don't want to use SSH protocol, just run this command1:

gh auth login

It's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:

What is your preferred protocol?
> http
  ssh

Select ssh, and it gh will save your preference for future gh commands. I'd recommend doing it this way so that gh can manage your credentials, BUT you can probably skip all the above by running2:

gh config set git_protocol ssh
 Oops, typo! Instead of the one-liner immediately above, to use `http` instead of `ssh` the command should read

"gh config set git_protocol ssh http" (see diff below).

- gh config set git_protocol ssh
# use the below for http protocol (to bypass ssh requirement)
+ gh config set git_protocol http

Doy! 😄

Footnotes

  1. gh auth login (cli.github.com/manual) ↩

  2. gh config set (cli.github.com/manual) ↩

@sgscaffidi3
Copy link

I just want to say that this thread is a great example of the open source community. An 8-year thread on a platform, with everybody posting little bits of code and share with others for free, on the platform itself.

I agree.

@mkumarb
Copy link

mkumarb commented Jul 7, 2024 •
edited
Loading

To clone all repos from an organisation using HTTP

gh auth login
gh repo list <organisation> --json=url --limit 1000 -q ".[].url" | xargs -n1 git clone

@almgwary
Copy link

1- Install jq, a lightweight and flexible command-line JSON processor
2- Fetch a list of repositories from the specified GitHub organization and clone them

brew install jq

curl -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1

Replace <ORG_NAME> with the actual name of the GitHub organization

  • Use curl to retrieve the repository data from the GitHub API with a limit of 100 repositories per page
  • Pipe the result to jq to extract the HTML URLs of all repositories
  • Use xargs to clone each repository URL to the local machine with a shallow copy (depth=1), which retrieves only the latest commit history

@Vaisakhkm2625
Copy link

  • | xargs -n1 git clone

we can essentially remove jq and directly use gh

gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

though, it's asking for ssh passcode... any idea how to overcome that?

@Vaisakhkm2625,

If you don't want to use SSH protocol, just run this command1:

gh auth login

It's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:

What is your preferred protocol?
> http
  ssh

Select ssh, and it gh will save your preference for future gh commands. I'd recommend doing it this way so that gh can manage your credentials, BUT you can probably skip all the above by running2:

gh config set git_protocol ssh

Oops, typo! Instead of the one-liner immediately above, to use http instead of ssh the command should read

"gh config set git_protocol ssh **http**" (see diff below).

- gh config set git_protocol ssh
# use the below for http protocol (to bypass ssh requirement)
+ gh config set git_protocol http

Doy! 😄

Footnotes

1. [`gh auth login` (cli.github.com/manual)](https://cli.github.com/manual/gh_auth_login) [↩](#user-content-fnref-1-9e7baee31a028e56fce81af59af26625)

2. [`gh config set` (cli.github.com/manual)](https://cli.github.com/manual/gh_config_set#:~:text=%24%20gh%20config%20set%20git_protocol%20ssh%20%2D%2Dhost%20github.com) [↩](#user-content-fnref-2-9e7baee31a028e56fce81af59af26625)

got around by using ssh agent

eval "$(ssh-agent -s)"  # Start the ssh-agent
ssh-add ~/.ssh/id_rsa    # Add your SSH key:
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment