Clone all repositories of a Github user

clone.bash

curl -s https://api.github.com/users/milanboers/repos | grep \"clone_url\" | awk '{print $2}' | sed -e 's/"//g' -e 's/,//g' | xargs -n1 git clone

Hi all I wrote a simple script to clone all the repositories from GitHub, either from a user or whole organization. Please check it out, I'm sure you'll find it useful

Nice script though! Keep in mind that you just crawl 100 each and you don't "loop" over that. So if you have > 100 you will only see 100 per page and no more pages. Hope it makes sense.

I finally switched to 'gickup' in a docker container for this (highly recommend!) which automagically downloads everything you have and anything you have access to.

Hi all I wrote a simple script to clone all the repositories from GitHub, either from a user or whole organization. Please check it out, I'm sure you'll find it useful

Nice script though! Keep in mind that you just crawl 100 each and you don't "loop" over that. So if you have > 100 you will only see 100 per page and no more pages. Hope it makes sense.

I encountered the same issue and made this helper has a bunch of other useful utilities too for managing your github projects

I encountered the same issue and made this helper has a bunch of other useful utilities too for managing your github projects

((page_count = public_repos / 100 + 1))

+10 for this smart move, you fetched the API repo amount and then use it for pagination to expand the crawl limits.

Will try your tool at some time, atm no ongoing tasks for that. But still good job! 👍

In my case, the pages have 30 entries, not 100. In any case, I think this solution is independent of that:

gubAll() {
  page=1
  repos=0
  entries=`curl -s https://api.github.com/users/$1/repos?page=$page | grep \"clone_url\"`
  while [[ -n $entries ]]
  do
    # echo "$entries"
    repos=$(( $repos + $(echo "$entries" | wc -l) ))
    echo "$entries" | awk '{print $2}' | sed -e 's/"//g' -e 's/,//g' | xargs -n1 git clone
    page=$(( $page + 1 ))
    entries=`curl -s https://api.github.com/users/$1/repos?page=$page | grep \"clone_url\"`
  done
  echo Pages: $(( $page -1 )), repos: $repos.
}

Then:

$ gubAll milanboers

will grab all 32 repos (at this point in time) from milanboers.

This simultaneously downloads as many repos as possible from any organization using parallel

ORGANIZATION=myorg
NUM_REPOS=2000

gh repo list $ORGANIZATION -L $NUM_REPOS --json name |
    jq -r --arg org $ORGANIZATION '$org + "/" + .[].name' |
    parallel gh repo clone