MossabDiae

The benfits:

1. less loading time (google download speed is better than your upload speed)
2. one click data loading
3. making data online you can work from other machines.
4. we'll be using private github repository so data is private.

1- Get a github token key

2- compress your data as zip , make a private github repository and upload it

MossabDiae

#!/usr/bin/env python

import requests
from bs4 import BeautifulSoup
import click 

def get_page(url):
    """
    take url get the webgape , store it's html , return that
    """

MossabDiae

#!/usr/bin/env sh
# dependencies: jq (sudo apt install jq)
# disabled parallel downloading to keep local network / aws happy.

download_code () {
    # clean double quotes if found
    local OUTPUT=${1//\"/}
    local dest=./lambda_functions/$OUTPUT.zip
    local URL=$(aws lambda get-function --function-name $OUTPUT --query 'Code.Location')