nemani/download_all_lambda_functions.sh

TheNetJedi · 2019-02-04T04:34:26Z

Fantastic!

Selimcse98 · 2019-06-14T02:01:58Z

Awesome! It worked nicely. Thanks a lot!

Ton-Anne · 2019-06-20T08:40:17Z

"doth" line 11 might be a typo?
Other than that very nice! Thanks a lot.

arorasahil1207 · 2019-10-07T08:22:43Z

I am new here can you tell me where you have put the access key/token ?

nemani · 2019-10-07T15:15:38Z

@arorasahil1207 as mentioned aws configure should do the trick

arorasahil1207 · 2019-10-14T12:42:00Z

I don't know how to work with bash can you please elaborate so that i can understand the same,thanks in advance

nemani · 2019-10-14T20:45:05Z

First you need a bash shell.

Next you need to install aws cli which allows you to interact with AWS services via the CLI and bash scripts.

https://docs.aws.amazon.com/cli/latest/userguide/install-linux-al2017.html

Once this is done, you need to run the command aws configure to setup the initial configuration.

It will ask you inputs like text mode and the IAM config keys to use with AWS CLI.

Then you can follow the script above.

SIVA451 · 2020-04-13T14:36:02Z

For me, it showing as an empty folder while unzipping it in windows OS.

pgorpuni · 2020-05-06T11:11:38Z

I am getting below error. I did AWS configure also.
An error occurred (ValidationException) when calling the GetFunction operation: 1 validation error detected: Value 'DB_User:' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-.]+)(:($LATEST|[a-zA-Z0-9-]+))?
xargs: wget: No such file or directory

nemani · 2020-05-07T06:20:35Z

@pgorpuni looks like the error is due to a lambda named 'DB_User'. The regex constraints for the name are by AWS and this does not match it. Don't think it is an error with the script. Try changing the name of the lambda.

I am getting below error. I did AWS configure also.
An error occurred (ValidationException) when calling the GetFunction operation: 1 validation error detected: Value 'DB_User:' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-.]+)(:($LATEST|[a-zA-Z0-9-]+))?
xargs: wget: No such file or directory

For me, it showing as an empty folder while unzipping it in windows OS.

@SIVA451 sorry I dont use windows so cant help by writing a script for it.
The above script is meant for bash (or similiar like zsh) shells which are commonly on linux.

sudeep-quantelai · 2020-12-05T08:24:56Z

I changed the original code to extract the function (starting with a specified prefix) from the zip file and extract the python code and rename it as <function_name>.lambda

download_code () {
    local OUTPUT=$1
        aws lambda get-function --function-name $OUTPUT --query 'Code.Location' | xargs wget -O ./lambda_functions/$OUTPUT.zip
        unzip ./lambda_functions/$OUTPUT.zip -d ./lambda_functions
        mv ./lambda_functions/lambda_function.py ./lambda_functions/$OUTPUT.lambda
        rm ./lambda_functions/$OUTPUT.zip
}

mkdir -p lambda_functions
for run in $(aws lambda list-functions --query 'Functions[?starts_with(FunctionName, `<your pattern here>`) == `true`].FunctionName' --output text);
do
        download_code "$run"
done

echo "done"

handeglc · 2021-05-12T11:52:06Z

I am getting below error. I did AWS configure also.
An error occurred (ValidationException) when calling the GetFunction operation: 1 validation error detected: Value 'DB_User:' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:lambda:)?([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-.]+)(:($LATEST|[a-zA-Z0-9-]+))?
xargs: wget: No such file or directory

this worked on my mac with aws-cli/2.2.3

download_code () {
    local OUTPUT=$1
    aws lambda get-function --function-name $OUTPUT --query 'Code.Location' | xargs wget -O ./lambda_functions/$OUTPUT.zip  
}

mkdir -p lambda_functions

for run in $(aws lambda list-functions  --query 'Functions[].FunctionName' --output text);
do
	download_code "$run" 
	echo $run
done

natanloterio · 2021-12-25T16:33:34Z

This is another version:


download_code () {
    local OUTPUT=$1
    aws lambda get-function --function-name $OUTPUT --query 'Code.Location' | xargs wget -O ./lambda_functions/$OUTPUT.zip  
}

mkdir -p lambda_functions

for run in $(aws lambda list-functions | grep FunctionName | awk -F'"' '{print $4}');
      do
           download_code "$run" &
      done
  
      echo "Completed Downloading all the Lamdba Functions!"

After that, you could unzip them in separated folders using:


find . -name '*.zip' -exec sh -c 'unzip -d "${1%.*}" "$1"' _ {} \;

abhay330 · 2022-09-22T14:20:26Z

If someone is looking for a Pythonic way to do this : https://github.com/abhay330/download_all_lambda_function

avicoder · 2022-12-12T04:37:39Z

It crashed my 64GB RAM mac, Twice :/

behrens-cloud · 2022-12-21T15:11:52Z

It crashed my 64GB RAM mac, Twice :/

I can verify that the original version does that on Macs. @handeglc removed the & to not run in background. I did the same and also added a loop to capture all regions. It could be improved even more of course. Thanks @nemani for posting this gist!

# Downloads all aws-lambda functions to a subdirectory
# Assumes you have a role that has at least read access to lambda.
# Credits to https://gist.github.com/nemani/defdde356b6678352bcd4af69b7fe529

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT
function ctrl_c() {
        echo "** CTRL-C Detected - aborting..."
        exit -1
}

download_region(){
    REGION=$1
    echo Searching region: $REGION
    for code in $(aws lambda list-functions --region $REGION --query 'Functions[].FunctionName' --output text);
    do
        download_path=./lambda_functions/$REGION/$code
        mkdir -p $download_path
        aws lambda get-function --function-name $code --region $REGION --query 'Code.Location' | xargs wget --quiet -O $download_path/$code.zip
        unzip -o -f -d $download_path $download_path/$code.zip
    done
}

# search all regions
for region in $(aws ec2 describe-regions --query "Regions[].{Name:RegionName}" --output text | sort -r); do
    download_region $region
done

echo "Completed Downloading all the Lamdba Functions!"

MossabDiae · 2023-01-24T13:31:01Z

Thanks for sharing, the code didn't work as stated in some previous replies I believe because of the filtering criteria to get the lambda function name.
AWS CLI is returning a JSON format which can be better explored using jq

If the above code didn't work for you too, try my implementation of the fix here.

hrpitta · 2023-02-17T16:15:00Z

@abhay330 thanks man! Worked very well

bf4 · 2023-02-27T15:26:26Z

very nice collaboration. I extended it in a fork https://gist.github.com/bf4/825009c3e49bf53259e8c18c4c38f190

#!/bin/bash
#
# Usage:
#  ./download_all.sh
#  ./download_all.sh download us-east-1 my-function
#  ./download_all.sh help
#
# Downloads all aws-lambda functions to a subdirectory
# Assumes you have a role that has at least read access to lambda.
# Credits to https://gist.github.com/nemani/defdde356b6678352bcd4af69b7fe529

set -euo pipefail
IFS=$'\n\t'
# Influenced by https://dev.to/thiht/shell-scripts-matter

abort=0
show_help() {
 abort=${abort:0}
 sed -ne '/^#/!q;s/.\{1,2\}//;1d;p' < "$0" >&$((abort+1))
 exit $abort
}

# shellcheck disable=SC2034
export AWS_PROFILE="${AWS_PROFILE:-your-profile-here}"
readonly DOWNLOAD_PATH_BASE="./lambda_functions"

# trap ctrl-c and call ctrl_c()
trap ctrl_c INT
ctrl_c() {
  echo "** CTRL-C Detected - aborting..."
  exit 1
}

get_function_names() {
  local region; region="$1"
  aws lambda list-functions --region "$region" --query 'Functions[].FunctionName' --output text
}

get_function() {
  local region; region="$1"
  local function_name; function_name="$2"
  local download_path
  download_path="${DOWNLOAD_PATH_BASE}/$region/$function_name"
  mkdir -p "$download_path"
  aws lambda get-function --function-name "$function_name" --region "$region" --query 'Code.Location' \
    | xargs wget --quiet -O "$download_path/$function_name.zip"

  unzip -o -f -d "$download_path" "$download_path/$function_name.zip"
}

download_region(){
  local region; region="$1"
  echo "searching region: $region"
  for function_name in $(get_function_names "$region"); do
    get_function "$region" "$function_name"
  done
}

get_regions(){
  aws ec2 describe-regions --query "Regions[].{Name:RegionName}" --output text | sort -r
}

main() {
  # search all regions
  for region in $(get_regions); do
    download_region "$region"
  done

  echo "Completed Downloading all the Lamdba Functions!"
}

case "${1:-}" in
  help|-h|--help)
    show_help
  ;;

  download)
    shift
    get_function "$1" "$2"
  ;;

  *)
    shift
    main
  ;;
esac

psamson · 2023-07-04T16:35:07Z

The original script worked for me on a Mac M1, ...BUT it deleted all the Lambda functions on AWS after downloading them.

nemani/download_all_lambda_functions.sh

TheNetJedi commented Feb 4, 2019

Selimcse98 commented Jun 14, 2019

Ton-Anne commented Jun 20, 2019

arorasahil1207 commented Oct 7, 2019

nemani commented Oct 7, 2019

arorasahil1207 commented Oct 14, 2019

nemani commented Oct 14, 2019

SIVA451 commented Apr 13, 2020

pgorpuni commented May 6, 2020

nemani commented May 7, 2020 •

edited

Loading

sudeep-quantelai commented Dec 5, 2020

handeglc commented May 12, 2021 •

edited

Loading

natanloterio commented Dec 25, 2021 •

edited

Loading

abhay330 commented Sep 22, 2022

avicoder commented Dec 12, 2022

behrens-cloud commented Dec 21, 2022 •

edited

Loading

MossabDiae commented Jan 24, 2023

hrpitta commented Feb 17, 2023

bf4 commented Feb 27, 2023

psamson commented Jul 4, 2023

	# Parallelly download all aws-lambda functions
	# Assumes you have ran `aws configure` and have output-mode as "text"
	# Works with "aws-cli/1.16.72 Python/3.6.7 Linux/4.15.0-42-generic botocore/1.12.62"

	download_code () {
	local OUTPUT=$1
	aws lambda get-function --function-name $OUTPUT --query 'Code.Location' \| xargs wget -O ./lambda_functions/$OUTPUT.zip
	}

	mkdir -p lambda_functions

	for run in $(aws lambda list-functions \| cut -f 6 \| xargs);
	do
	download_code "$run" &
	done

	echo "Completed Downloading all the Lamdba Functions!"

	# If you want to download only ones with specific prefix
	# https://github.com/sambhajis-gdb/download_all_lambda_function/blob/master/get_lambda_with_prefix.sh
	# Credits to https://stackoverflow.com/users/6908140/sambhaji-sawant for the

nemani/download_all_lambda_functions.sh

TheNetJedi commented Feb 4, 2019

Selimcse98 commented Jun 14, 2019

Ton-Anne commented Jun 20, 2019

arorasahil1207 commented Oct 7, 2019

nemani commented Oct 7, 2019

arorasahil1207 commented Oct 14, 2019

nemani commented Oct 14, 2019

SIVA451 commented Apr 13, 2020

pgorpuni commented May 6, 2020

nemani commented May 7, 2020 • edited Loading

sudeep-quantelai commented Dec 5, 2020

handeglc commented May 12, 2021 • edited Loading

natanloterio commented Dec 25, 2021 • edited Loading

abhay330 commented Sep 22, 2022

avicoder commented Dec 12, 2022

behrens-cloud commented Dec 21, 2022 • edited Loading

MossabDiae commented Jan 24, 2023

hrpitta commented Feb 17, 2023

bf4 commented Feb 27, 2023

psamson commented Jul 4, 2023

nemani commented May 7, 2020 •

edited

Loading

handeglc commented May 12, 2021 •

edited

Loading

natanloterio commented Dec 25, 2021 •

edited

Loading

behrens-cloud commented Dec 21, 2022 •

edited

Loading