GitHub Repository Cloner

repocloner.sh

#!/bin/bash

# Default values
name="embedded"
language="language:C"
division=6
output_file=""
page=10
start_date=$(date +%Y-%m-%d)
end_date=$start_date

# Process command line arguments
while getopts ":n:l:d:o:p:s:e:" opt; do
  case $opt in
    n)
      name=$OPTARG
      ;;
    l)
      language="language:$OPTARG"
      ;;
    d)
      division=$OPTARG
      ;;
    o)
      output_file=$OPTARG
      ;;
    p)
      page=$OPTARG
      ;;
    s)
      start_date=$OPTARG
      ;;
    e)
      end_date=$OPTARG
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
    :)
      echo "Option -$OPTARG requires an argument." >&2
      exit 1
      ;;
  esac
done

# Validate output file argument
if [ -z "$output_file" ]; then
  echo "Please provide an output file path using the -o option."
  echo "Usage: ./script.sh -o <output_file> [optional_arguments]"
  exit 1
fi

# Calculate the division in seconds
division_seconds=$((division * 24 * 60 * 60))

# Convert start and end dates to timestamps
start_timestamp=$(date -d "$start_date" +%s)
end_timestamp=$(date -d "$end_date" +%s)

# Loop over the date ranges and make the API requests
# Sleep commands are used to cool down the usage of the GitHub API
current_timestamp=$start_timestamp
while [[ $current_timestamp -le $end_timestamp ]]; do
  current_date=$(date -d @"$current_timestamp" +"%Y-%m-%d")
  next_timestamp=$((current_timestamp + division_seconds))
  next_date=$(date -d @"$next_timestamp" +"%Y-%m-%d")

  echo "Date Range: $current_date to $next_date"

  for ((i = 1; i <= page; i++)); do
    curl "https://api.github.com/search/repositories?q=$name+$language+created:$current_date..$next_date&per_page=100&page=$i" |
      jq -r '.items[].ssh_url' >>"$output_file"
    sleep 5
  done

  current_timestamp=$next_timestamp
  sleep 10
done

Repository Cloner

This script allows you to clone repositories from GitHub based on specified criteria.

Usage

./script.sh -o <output_file> [optional_arguments]

Command Line Arguments

The following command line arguments are available:

-o <output_file>: Specifies the path to the output file where the cloned repository URLs will be stored. (Required)
Optional arguments:

-n : Specifies the repository name to search for. (Default: embedded)
-l : Specifies the programming language of the repositories. (Default: language:C)
-d : Specifies the division in days for date ranges. (Default: 6)
-p : Specifies the number of pages to retrieve from the GitHub API. (Default: 10)
-s <start_date>: Specifies the start date for the cloning process. (Default: current date)
-e <end_date>: Specifies the end date for the cloning process. (Default: current date)
Note: Date arguments should be provided in the format "YYYY-MM-DD".

Example
To clone Python repositories related to machine learning created between 2002-01-01 and 2023-06-14 with a division of 6 days per date range, you can use the following command:

./script.sh -o python-ml-repos.txt -n ML -l language:Python -d 6 -s 2002-01-01 -e 2023-06-14

The cloned repository URLs will be stored in the specified output file.

Feel free to customize the content and provide additional information as needed.