A bash script which supports uploading blobs to Azure Storage: ./upload-to-azure.sh [filename]

upload-to-azure.sh

#!/usr/bin/env bash

FILENAME=${1}
# expected to be defined in the environment
#  - AZURE_STORAGE_ACCOUNT
#  - AZURE_CONTAINER_NAME
#  - AZURE_ACCESS_KEY

# inspired by
# https://stackoverflow.com/questions/20103258/accessing-azure-blob-storage-using-bash-curl

authorization="SharedKey"

HTTP_METHOD="PUT"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2015-02-21"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
x_ms_blob_type_h="x-ms-blob-type:BlockBlob"

FILE_LENGTH=$(wc --bytes < ${FILENAME})
FILE_TYPE=$(file --mime-type -b ${FILENAME})
FILE_MD5=$(md5sum -b ${FILENAME} | awk '{ print $1 }')

# Build the signature string
canonicalized_headers="${x_ms_blob_type_h}\n${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${AZURE_STORAGE_ACCOUNT}/${AZURE_CONTAINER_NAME}/${FILE_MD5}"

#######
# From: https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
#
#StringToSign = VERB + "\n" +
#               Content-Encoding + "\n" +
#               Content-Language + "\n" +
#               Content-Length + "\n" +
#               Content-MD5 + "\n" +
#               Content-Type + "\n" +
#               Date + "\n" +
#               If-Modified-Since + "\n" +
#               If-Match + "\n" +
#               If-None-Match + "\n" +
#               If-Unmodified-Since + "\n" +
#               Range + "\n" +
#               CanonicalizedHeaders +
#               CanonicalizedResource;
string_to_sign="${HTTP_METHOD}\n\n\n${FILE_LENGTH}\n\n${FILE_TYPE}\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $AZURE_ACCESS_KEY | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(printf  "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)

authorization_header="Authorization: $authorization $AZURE_STORAGE_ACCOUNT:$signature"
OUTPUT_FILE="https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_CONTAINER_NAME}/${FILE_MD5}"

curl -X ${HTTP_METHOD} \
    -T ${FILENAME} \
    -H "$x_ms_date_h" \
    -H "$x_ms_version_h" \
    -H "$x_ms_blob_type_h" \
    -H "$authorization_header" \
    -H "Content-Type: ${FILE_TYPE}" \
    ${OUTPUT_FILE}

if [ $? -eq 0 ]; then
    echo ${OUTPUT_FILE}
    exit 0;
fi;
exit 1

Getting error

<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:f93e3dbc-e01e-0045-098e-07d78b000000
Time:2018-06-19T05:28:40.8713951Z</Message><MaxLimit>67108864</MaxLimit>

Hello @rtyler.
I am trying to create a file system on an adlsgen 2 account with hierarchical namespace enabled.
However I am getting a different error than @codegagan

curl: (6) Could not resolve host: PUT; Unknown error
{"error":{"code":"AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:27d7af5c-e01f-008f-7f44-5406ff000000\nTime:2019-08-16T15:06:27.7274914Z"}}https://storageaccountname.dfs.core.windows.net/filesystem?resource=filesystem

Could you advise on what might be wrong?

thanks

useless

Exactly what I was looking for and works perfectly.

https://gist.github.com/rtyler/30e51dc72bed23718388c43f9c11da76#gistcomment-3000362

@saikovvuri I am facing the same error. Were you able to resolve the error.? If so, how.?

@gunnypatel should you open up any firewall ports before pushing the file.?

You shouldn't have to do anything with the firewall as long as you can make HTTPS calls to Azure. If you're getting an authentication errors then there's something wrong with the header being sent to Azure. Check to make sure that the time on your system is correct. I think if it's off by a few minutes it could cause an issue with the authentication headers.

Thank you, this was very helpful! 🙌

This code works completely fine... But my requirement is i need to push 0 byte file to azure blob storage. when i try to do that, it is throwing an error. does someone have any idea what i need to change in this script ?