caniszczyk/clone-all-twitter-github-repos.sh

flavioespinoza · 2021-04-19T18:32:44Z

Create a bash alias/func in your `~/.bashrc` or `~/.zshrc` file

I solved this for my team by creating an alias/bash func in the ~/.bashrc or ~/.zshrc configuration file.

Config

MacOS Terminal

Open a new Terminal and edit your .bashrc configuration file:

sudo nano ~/.bashrc

-- or --

MacOS iTerm2 with oh-my-zsh

Open iTerm2 terminal and edit your .zshrc configuration file:

sudo nano ~/.zshrc

Create CloneAll Func

Add the following bash func as an alias:

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}

`.bashrc`

Save and close your ~/.bashrc file and then close the terminal, then open a new one.

You need to do this or the new alias won't initialize:

-- or --

`.zshrc`

Save and close your ~/.zshrc file and run the following command:

exec "$SHELL"

Run

CloneAll <your_github_org_name>

Example

If your GitHub repo URL is called https://github.com/flavioespinoza the command would be:

CloneAll flavioespinoza

Troubleshooting

Set Number of Repos to Clone

The per_page=200 at the end of the first variable ORG_URL sets the number of repos that will be cloned

ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=<number_of_repos>";  <---- make sure this is what you want

askfriends · 2021-06-20T21:17:35Z

How to run this on Windows10?
btw i want to clone/download all my own repos which include 3 or 4 private repos too.

Gesugao-san · 2021-08-06T18:11:25Z

@caniszczyk
Outdated.
Reason: ubygems is DEAD.
https://www.google.ru/search?q=ubygems
https://blog.engineyard.com/goodbye-ubygems

l0b0 · 2021-09-06T03:08:58Z

Parallel, properly paginated (100 per page is max) SSH cloning of public and private org repos - just provide $TOKEN and $ORG:

page=1
while links=($(curl -H "Authorization: token ${TOKEN}" -s "https://api.github.com/orgs/${ORG}/repos?per_page=100&page=${page}" | jq -rc '.[] | {ssh_url} | .ssh_url'));  [[ "$links" ]]
do
    GIT_TERMINAL_PROMPT=0 parallel git clone --depth=1 {} ::: "${links[@]}"
    ((++page))
done

redaphid · 2021-12-18T22:32:38Z

I just want to say that this thread is a great example of the open source community. An 8-year thread on a platform, with everybody posting little bits of code and share with others for free, on the platform itself.

redaphid · 2021-12-18T23:56:27Z

I just finished up a simple script if you have a similar oddball setup to me (jq,gh,fish shell). Fish is just used because I like the scripting language. (Not that I'm defensive about fish or anything. haha)

#!/usr/bin/env fish
# You need:
# - fish (https://fishshell.com/) (You don't need to use it as your shell, but it's a good shellscripting language)
# - gh - github cli (https://cli.github.com)
# - jq (https://stedolan.github.io/jq/)

set -q ORG[1]; or begin
	echo "I need the org name! I use the environment variable $ORG. example: ORG=octoblu ./download.fish" >&2
	exit 1
end

set repos (gh repo list $ORG --limit 9999 --json name)
set repos_to_clone (echo $repos | jq -r ".[].name")

echo "cloning repos:\n $repos_to_clone"
for u in $repos_to_clone
	echo "cloning $u"
	git clone https://github.com/$ORG/$u repos/$u; or echo "failed to clone $u"
	sleep 5
end

redaphid · 2021-12-19T00:33:16Z

fwiw you can also fork all the repos in an org to a new org like this:

#!/usr/bin/env fish
# You need:
# - fish (https://fishshell.com/) (You don't need to use it as your shell, but it's a good shellscripting language)
# - gh - github cli (https://cli.github.com)
# - jq (https://stedolan.github.io/jq/)

set -q OLD_ORG[1]; or begin
	echo "I need the org name! I use the environment variable $OLD_ORG. example: OLD_ORG=octoblu NEW_ORG=meshnu ./fork.fish" >&2
	exit 1
end

set -q NEW_ORG[1]; or begin
	echo "I need the org name! I use the environment variable $NEW_ORG. example: OLD_ORG=octoblu NEW_ORG=meshnu ./fork.fish" >&2
	exit 1
end

set repos (gh repo list $OLD_ORG --limit 9999 --json name)
set repos_to_clone (echo $repos | jq -r ".[].name")

echo "cloning repos:\n $repos_to_clone"
for u in $repos_to_clone
	echo "forking $u"
	gh repo fork $OLD_ORG/$u --clone=false --org $NEW_ORG; or echo "failed to clone $u"
	sleep 5
end

# echo $repos

gabrie30 · 2021-12-22T19:36:03Z

@askfriends ghorg will work on Windows and allows you to clone your own repos as well as from an org

patrickdevivo · 2022-01-22T23:37:30Z

This can also be done with a tool called mergestat, as described on this page of the docs.

For example like so:

mergestat "SELECT clone('https://github.com/mergestat/'|| name) AS path FROM github_org_repos('mergestat')" -v --clone-dir my-dir

An advantage of this approach is that pagination and API rate limits are taken care of by the tool, and arbitrary "sets" of repos can be cloned (filter out by certain criteria such as age or language).

Full disclosure, I am the maintainer/creator of mergestat, but wanted to share as this is a fairly frequent use-case/question we receive from people looking to batch query/work with repos belonging to an org.

doleron · 2022-01-24T12:48:00Z

Well, I used this repo: https://github.com/muhasturk/gitim

python3 -m gitim -o my_org -t mytoken

mirstan · 2022-01-29T04:49:11Z

Thanks for all the hints. This worked for me and assumes that you've installed gh and jq then authenticated using gh auth login :
gh repo list $ORG --limit 9999 --json sshUrl | jq '.[]|.sshUrl' | xargs -n1 git clone

ChaitanyaChandra · 2022-02-18T02:29:54Z

clone all repos

you need to install gh (github api)
windows : choco install gh
mac : brew install gh

username="ChaitanyaChandra" # user name or org name is required
limit=100
gh repo list $username --limit $limit | awk '{print $1}' > out.txt
while read -r line; do echo "git clone  https://github.com/$line.git" >> final.sh  ; done < out.txt
chmod +x final.sh
./final.sh

aclk · 2022-03-04T06:03:22Z

Into $ORG folder
mkdir $ORG; cd $ORG
Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone

ahmed2m · 2022-05-14T22:15:44Z

Into $ORG folder
mkdir $ORG; cd $ORG

Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone

Github stopped https clones (username and passwords needed for every clone so instead use ssh clone
gh repo list $ORG --limit 9999 --json sshUrl | jq '.[]|.sshUrl' | xargs -n1 git clone

RTC1 · 2022-06-29T16:13:31Z

This also works whilst using gh command and auth instead of git

gh repo list $ORG --limit 9999 --json nameWithOwner | jq '.[]|.nameWithOwner' | xargs -n1 gh repo clone

sikmir · 2022-10-07T06:44:04Z

this can be even simpler, assuming < 100 repos: curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone

Just in case, if you need ssh url and a bit more than 100 repos:
curl -s "https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100&page=<1,2...>" | jq '.[].ssh_url' | xargs -n 1 git clone

Danipulok · 2023-01-27T06:31:42Z

Hi all
I wrote a simple script to clone all the repositories from GitHub, either from a user or whole organization. Please check it out, I'm sure you'll find it useful

octohedron · 2023-07-03T19:37:17Z

Here's a fully working script with things from here, SO + some improvements by me. Needs gh cli.

If the repo exists it will check out the main/master branch and pull.

#!/bin/bash

ORG="myorg" # Your organization
CLONE_PATH="/home/user/Documents/..." # Path in your filesystem where you want to clone the repos
PER_PAGE=100 # per_page maxes out at 100
TOKEN="" # Set your token from GH

for ((PAGE=1; ; PAGE+=1)); do
  # Page 0 and 1 are the same
  # Change authorization method as needed
  INPUT=$(curl -H "Authorization: token $TOKEN" -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[].clone_url")
  if [[ -z "$INPUT" ]]; then
    echo "All repos processed, stopped at page=$PAGE"
    exit
  fi
  while read REPO_URL ; do
    echo $REPO_URL
    temp=${REPO_URL##*/}
    repo_name=${temp%.*}
    gh repo clone "$REPO_URL" "$CLONE_PATH/$repo_name" -- -q 2>/dev/null || (
        cd "$CLONE_PATH/$repo_name"
        # Handle case where local checkout is on a non-main/master branch
        # - ignore checkout errors because some repos may have zero commits, 
        # so no main or master
        git checkout -q main 2>/dev/null || true
        git checkout -q master 2>/dev/null || true
        git pull -q
    )
  done < <(echo "$INPUT")
done

ology · 2023-11-04T03:23:52Z

And here's a "Perl-as-glue" version, that works on systems with command-line curl and git: https://gist.github.com/ology/b6087be12131bae0441d40d87505e658 :D

Vaisakhkm2625 · 2024-01-15T18:23:12Z

| xargs -n1 git clone

we can essentially remove jq and directly use gh

gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

though, it's asking for ssh passcode... any idea how to overcome that?

hotelzululima · 2024-01-16T02:01:39Z

meh.. I have moved to ghorg. implemented it does everything I could do with my own scripts and is 10 times as fast to boot gh

…

On Mon, Jan 15, 2024 at 10:23 AM Vaisakh K M ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ - | xargs -n1 git clone we can essentially remove jq and directly use gh gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone — Reply to this email directly, view it on GitHub <https://gist.github.com/caniszczyk/3856584#gistcomment-4832274> or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNZZRED6QI6M2V6ZDBT4XTYOVXZXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTQNJWGU4DJJ3UOJUWOZ3FOKTGG4TFMF2GK> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

Vaisakhkm2625 · 2024-01-18T22:55:51Z

meh.. I have moved to ghorg. implemented it does everything I could do with my own scripts and is 10 times as fast to boot gh
…
On Mon, Jan 15, 2024 at 10:23 AM Vaisakh K M @.> wrote: @.* commented on this gist. ------------------------------ - | xargs -n1 git clone we can essentially remove jq and directly use gh gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone — Reply to this email directly, view it on GitHub https://gist.github.com/caniszczyk/3856584#gistcomment-4832274 or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNZZRED6QI6M2V6ZDBT4XTYOVXZXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTQNJWGU4DJJ3UOJUWOZ3FOKTGG4TFMF2GK . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

thanks.. i didn't know that existed...

davidglezz · 2024-01-20T19:43:09Z

Windows / PowerShell

gh repo list theUSERorORG --json=nameWithOwner --limit 1000 -q ".[].nameWithOwner"  | %{gh repo clone $_}

pa-0 · 2024-06-02T22:17:40Z

| xargs -n1 git clone

we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone
though, it's asking for ssh passcode... any idea how to overcome that?

@Vaisakhkm2625,

If you don't want to use SSH protocol, just run this command¹:

gh auth login

It's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:

What is your preferred protocol?
> http
  ssh

Select ssh, and it gh will save your preference for future gh commands. I'd recommend doing it this way so that gh can manage your credentials, BUT you can probably skip all the above by running²:

~~gh config set git_protocol ssh~~
Oops, typo! Instead of the one-liner immediately above, to use `http` instead of `ssh` the command should read

"gh config set git_protocol ssh http" (see diff below).

- gh config set git_protocol ssh
# use the below for http protocol (to bypass ssh requirement)
+ gh config set git_protocol http

Doy! 😄

sgscaffidi3 · 2024-06-26T19:30:48Z

I just want to say that this thread is a great example of the open source community. An 8-year thread on a platform, with everybody posting little bits of code and share with others for free, on the platform itself.

I agree.

mkumarb · 2024-07-07T13:25:56Z

To clone all repos from an organisation using HTTP

gh auth login
gh repo list <organisation> --json=url --limit 1000 -q ".[].url" | xargs -n1 git clone

almgwary · 2024-08-30T01:45:42Z

1- Install jq, a lightweight and flexible command-line JSON processor
2- Fetch a list of repositories from the specified GitHub organization and clone them

brew install jq

curl -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone --depth=1

Replace <ORG_NAME> with the actual name of the GitHub organization

Use curl to retrieve the repository data from the GitHub API with a limit of 100 repositories per page
Pipe the result to jq to extract the HTML URLs of all repositories
Use xargs to clone each repository URL to the local machine with a shallow copy (depth=1), which retrieves only the latest commit history

Vaisakhkm2625 · 2024-09-01T16:43:37Z

| xargs -n1 git clone

we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone
though, it's asking for ssh passcode... any idea how to overcome that?
@Vaisakhkm2625,

If you don't want to use SSH protocol, just run this command1:
gh auth login
It's interactive, so there are a couple of selections you need to make first i.e., selecting the host (GitHub or GitHub Enterprise) etc. but eventually you will reach:
What is your preferred protocol?
> http
  ssh
Select ssh, and it gh will save your preference for future gh commands. I'd recommend doing it this way so that gh can manage your credentials, BUT you can probably skip all the above by running2:
gh config set git_protocol ssh
Oops, typo! Instead of the one-liner immediately above, to use http instead of ssh the command should read

"gh config set git_protocol ssh **http**" (see diff below).
- gh config set git_protocol ssh
# use the below for http protocol (to bypass ssh requirement)
+ gh config set git_protocol http
Doy! 😄

Footnotes
1. [`gh auth login` (cli.github.com/manual)](https://cli.github.com/manual/gh_auth_login) [↩](#user-content-fnref-1-9e7baee31a028e56fce81af59af26625)

2. [`gh config set` (cli.github.com/manual)](https://cli.github.com/manual/gh_config_set#:~:text=%24%20gh%20config%20set%20git_protocol%20ssh%20%2D%2Dhost%20github.com) [↩](#user-content-fnref-2-9e7baee31a028e56fce81af59af26625)

got around by using ssh agent

eval "$(ssh-agent -s)"  # Start the ssh-agent
ssh-add ~/.ssh/id_rsa    # Add your SSH key:
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone

caniszczyk/clone-all-twitter-github-repos.sh

flavioespinoza commented Apr 19, 2021 •

edited

Loading

askfriends commented Jun 20, 2021

Gesugao-san commented Aug 6, 2021

l0b0 commented Sep 6, 2021 •

edited

Loading

redaphid commented Dec 18, 2021

redaphid commented Dec 18, 2021 •

edited

Loading

redaphid commented Dec 19, 2021 •

edited

Loading

gabrie30 commented Dec 22, 2021

patrickdevivo commented Jan 22, 2022

doleron commented Jan 24, 2022

mirstan commented Jan 29, 2022

ChaitanyaChandra commented Feb 18, 2022

aclk commented Mar 4, 2022

ahmed2m commented May 14, 2022

RTC1 commented Jun 29, 2022

sikmir commented Oct 7, 2022

Danipulok commented Jan 27, 2023

octohedron commented Jul 3, 2023

ology commented Nov 4, 2023

Vaisakhkm2625 commented Jan 15, 2024 •

edited

Loading

hotelzululima commented Jan 16, 2024 via email

Vaisakhkm2625 commented Jan 18, 2024

davidglezz commented Jan 20, 2024

pa-0 commented Jun 2, 2024 •

edited

Loading

sgscaffidi3 commented Jun 26, 2024

mkumarb commented Jul 7, 2024 •

edited

Loading

almgwary commented Aug 30, 2024

Vaisakhkm2625 commented Sep 1, 2024

caniszczyk/clone-all-twitter-github-repos.sh

flavioespinoza commented Apr 19, 2021 • edited Loading

Create a bash alias/func in your ~/.bashrc or ~/.zshrc file

Config

MacOS Terminal

MacOS iTerm2 with oh-my-zsh

Create CloneAll Func

.bashrc

.zshrc

Run

Example

Troubleshooting

Set Number of Repos to Clone

askfriends commented Jun 20, 2021

Gesugao-san commented Aug 6, 2021

l0b0 commented Sep 6, 2021 • edited Loading

redaphid commented Dec 18, 2021

redaphid commented Dec 18, 2021 • edited Loading

redaphid commented Dec 19, 2021 • edited Loading

gabrie30 commented Dec 22, 2021

patrickdevivo commented Jan 22, 2022

doleron commented Jan 24, 2022

mirstan commented Jan 29, 2022

ChaitanyaChandra commented Feb 18, 2022

clone all repos

aclk commented Mar 4, 2022

ahmed2m commented May 14, 2022

RTC1 commented Jun 29, 2022

sikmir commented Oct 7, 2022

Danipulok commented Jan 27, 2023

octohedron commented Jul 3, 2023

ology commented Nov 4, 2023

Vaisakhkm2625 commented Jan 15, 2024 • edited Loading

hotelzululima commented Jan 16, 2024 via email

Vaisakhkm2625 commented Jan 18, 2024

davidglezz commented Jan 20, 2024

pa-0 commented Jun 2, 2024 • edited Loading

Footnotes

sgscaffidi3 commented Jun 26, 2024

mkumarb commented Jul 7, 2024 • edited Loading

almgwary commented Aug 30, 2024

Vaisakhkm2625 commented Sep 1, 2024

Footnotes

flavioespinoza commented Apr 19, 2021 •

edited

Loading

Create a bash alias/func in your `~/.bashrc` or `~/.zshrc` file

`.bashrc`

`.zshrc`

l0b0 commented Sep 6, 2021 •

edited

Loading

redaphid commented Dec 18, 2021 •

edited

Loading

redaphid commented Dec 19, 2021 •

edited

Loading

Vaisakhkm2625 commented Jan 15, 2024 •

edited

Loading

pa-0 commented Jun 2, 2024 •

edited

Loading

mkumarb commented Jul 7, 2024 •

edited

Loading