A few years ago, I helped a non-profit organization build and deploy a series of web applications and micro-services on top of AWS. One of the services was responsible for sending out email notifications to people via AWS SES.
Back then, I wrote a really simple program in Go that made direct calls to the AWS API. Instead of using an official library, I read the API specification, learned how to authenticate requests, and implemented a fairly trivial SDK. The code was succinct, readable, and worked perfectly for 3 years without any issues.
That is until this past week…
AWS recently updated the specification, we’re now at Signature Version 4 (SigV4), for how API requests must be formed and signed by clients. In this walkthrough, I’ll share the full Go source code (~130 LOC) that I put together to make valid authenticated calls to AWS for SES. You don’t need to be knowledgeable about Go to grok code in this post. In fact, code in this post will translate pretty cleanly to most languages - even the esoteric ones. The specification I’ll be following and writing code to can be found here: Signature Version 4 signing process - Amazon General Reference.
Let’s start by walking through properties that I established at global scope:
// AWS Configuration
const awsAccessKeyID = ""
const awsSecretAccessKey = ""
const awsSESRegion = "us-west-2"For the AWS initiated, the above configuration should make sense. For those new to the game, you’ll need to fill in the appropriate access and secret keys as well as the AWS region that you’ll be using for SES. It’s important to mention that this is just an example program and you should not have your AWS access and secret keys embedded in your code. There are better ways to handle the security of these credentials.
Let’s start with main, the program’s entry point:
func main() {
from, to, subject := "[email protected]", "[email protected]", "Test"
body := fmt.Sprintf("Hi %s!", to)
s, err := mail(from, to, subject, body)
fmt.Println(s, err)
}We’re keeping things simple for our local mail func and only asking for the sender, recipient, subject, and body. The func signature is:
func mail(from, to, subject, body string) (string, error)The returned string will be the body of the AWS API HTTP request and error will be any error that we encounter on the way. In the development of this func, we’ll need to add 2 more supporting functions for deriving the signing key, but we’ll get there in a bit. With this boilerplate out of the way, let’s dive deep into the mail func, the heart of this program.
func mail(from, to, subject, body string) (string, error) {
data := make(url.Values)
data.Add("Action", "SendEmail")
data.Add("Source", from)
data.Add("Destination.ToAddresses.member.1", to)
data.Add("Message.Subject.Data", subject)
data.Add("Message.Body.Text.Data", body)
data.Add("AWSAccessKeyId", awsAccessKeyID)
encodedData := data.Encode()
reqBody := strings.NewReader(encodedData)
sesHost := fmt.Sprintf("email.%s.amazonaws.com", awsSESRegion)
sesEndpoint := fmt.Sprintf("https://%s", awsSESRegion)
req, err := http.NewRequest("POST", sesEndpoint, reqBody)
if err != nil {
return "", err
}
// ...At the top of the function, we build up a list of value that will eventually be form-urlencoded. I used all of the parameters defined above in my older library, which maps to the SES API, and did not have to make any changes in the move to SigV4. I’ve defined the sesHost and sesEndpoint variables - they’ll be re-used later when we build out the canonical request for signing.
There are 4 tasks to building an authenticated AWS API request:
- Creating a canonical request
- Creating a string to sign
- Computing the signature
- Adding the signature to the HTTP request
The rest of this post will break down each of these steps and share the relevant Go code for each of these. At the end, I’ll share the entire Go mail function. If you’re looking for how it all comes together, skip to the bottom of this post.
The corresponding task specification can be found here: Task 1: Create a canonical request for Signature Version 4 - Amazon General Reference
This portion is fairly straightforward: we want to generate a canonical request that adheres to the following specification:
CanonicalRequest =
HTTPRequestMethod + '\n' +
CanonicalURI + '\n' +
CanonicalQueryString + '\n' +
CanonicalHeaders + '\n' +
SignedHeaders + '\n' +
HexEncode(Hash(RequestPayload))
Here’s the Go source code for this specific task:
// Now
now := time.Now().UTC()
date := now.Format("20060102T150405Z")
// Canonical Headers
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("host", sesHost)
req.Header.Set("x-amz-date", date)
// Headers
canonicalHeaders := fmt.Sprintf("host:%s\nx-amz-date:%s\n", sesHost, date)
signedHeaders := "host;x-amz-date"
// Compute Payload Hash
payloadHash := hashAndEncode(encodedData)
// Canonical Request
canonicalRequest := fmt.Sprintf("%s\n%s\n%s\n%s\n%s\n%s",
"POST", "/", "", canonicalHeaders, signedHeaders, payloadHash)In the source code above, we get there by executing the following steps:
- Generate a date stamp that will used in a header as well as the canonical request
- Set the simple headers:
Content-Type,host, andx-amz-date; some languages/frameworks automatically set thehostheader but I’m adding it here in case yours does not - Compute the payload hash of the SES parameters that we’ve already encoded for the HTTP body
- Generate the canonical request string, which we’ll use in the following steps
I wrote a simple function for SHA256 hashing and returning the hex encoded string:
func hashAndEncode(s string) string {
h := sha256.New()
h.Write([]byte(s))
return hex.EncodeToString(h.Sum(nil))
}The corresponding task specification can be found here: Task 2: Create a string to sign for Signature Version 4 - Amazon General Reference
In this portion we’re trying to generate a string to sign that meets the following specification:
StringToSign =
Algorithm + \n +
RequestDateTime + \n +
CredentialScope + \n +
HashedCanonicalRequest
Here’s the Go source code for this specific task:
// Compute Canonical Request Hash
canonicalRequestHash := hashAndEncode(canonicalRequest)
// Signing String
algorithm := "AWS4-HMAC-SHA256"
credentialDate := now.Format("20060102")
credentialScope := fmt.Sprintf("%s/%s/ses/aws4_request",
credentialDate, awsSESRegion)
stringToSign := fmt.Sprintf("%s\n%s\n%s\n%s",
algorithm, date, credentialScope, canonicalRequestHash)In the source code above, we get there by executing the following steps:
- Compute the canonical request hash
- Initialize and format variables for the string to sign; note, the credential date is formatted differently from the
x-amz-dateheader - Generate the string to sign, which we’ll use in the following steps
Now we’re off to task 3 - arguably the most interesting of all the tasks that we’ve discussed here.
The corresponding task specification can be found here: Task 3: Calculate the signature for Amazon Signature Version 4 - Amazon General Reference
For this task, we’re generate a signing key based on the following specification:
kSecret = your secret access key
kDate = HMAC("AWS4" + kSecret, Date)
kRegion = HMAC(kDate, Region)
kService = HMAC(kRegion, Service)
kSigning = HMAC(kService, "aws4_request")
Unlike the previous 2 tasks, we’ll break this up into 2 parts. I’ll start with sharing the bits from our mail function to generate the signature. Here’s the Go source code:
// Compute Signature
signingKey := getSignatureKey(awsSecretAccessKey, credentialDate,
awsSESRegion, "ses")
signatureSHA := hmac.New(sha256.New, signingKey)
signatureSHA.Write([]byte(stringToSign))
signatureString := hex.EncodeToString(signatureSHA.Sum(nil))Now, here’s the Go code for getSignatureKey and sign:
func sign(key []byte, message string) []byte {
h := hmac.New(sha256.New, key)
h.Write([]byte(message))
return h.Sum(nil)
}
func getSignatureKey(key, dateStamp, region, service string) []byte {
kSecret := []byte(fmt.Sprintf("AWS4%s", key))
kDate := sign(kSecret, dateStamp)
kRegion := sign(kDate, region)
kService := sign(kRegion, service)
kSigning := sign(kService, "aws4_request")
return kSigning
}You’ll notice a few hardcoded elements based on the spec, like AWS4 prefixing the secret access key or aws4_request as the final element in signing the message.
Now, we’ll head to the final task, adding the signature to the request.
The corresponding task specification can be found here: Task 4: Add the signature to the HTTP request - Amazon General Reference
We’re ultimately building a value for an HTTP Authorization header that meets the following specification:
Authorization: algorithm Credential=access key ID/credential scope, SignedHeaders=SignedHeaders, Signature=signature
Here’s the Go source code for this specific task:
// Set Authorization Header
authorizationHeader := fmt.Sprintf("%s Credential=%s/%s, SignedHeaders=%s, Signature=%s", algorithm, awsAccessKeyID, credentialScope, signedHeaders, signatureString)
req.Header.Set("Authorization", authorizationHeader)There you have it! These four tasks make up all the relevant bits to make an authenticated AWS API request to send an email with SES.
The final source code for this program can be found here: https://gist.github.com/anandkunal/b67eb94454b77cfc2b50026989586cc0
I had a ton of fun putting this together and learned more about AWS’ latest specification. While I could have used the official AWS Go module to get the job done quickly and efficiently, I found this to be a really great way to learn a new protocol. I hope this helps those that are upgrading their scripts or curious to see what it looks like to implement AWS SigV4 in Go.
AWS has solid documentation. Aside from the links that were already mentioned above, I relied on the following to help me get this program built:
- https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-ses-api-authentication.html
- https://docs.aws.amazon.com/ses/latest/DeveloperGuide/using-ses-api-requests.html
- https://docs.amazonaws.cn/en_us/general/latest/gr/sigv4_signing.html
- https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
- https://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-common-coding-mistakes