Solution to AWS Lambda node.js UnrecognizedClientException "The security token included in the request is invalid."

troubleshooting.md

Troubleshooting AWS unauthorized errors in lambda requests

This is mainly for node.js but might apply to other environments. Unsure.

If you are running a AWS Lambda function that calls another AWS service and getting an error about invalid tokens or other access denied errors, do this:

Check IAM

The role assigned to your lambda function will need permission to perform the actions. Check IAM and make sure the role has all the permissions.

I usually give Full Access to the service in question (e.g. DynamoDB) if I suspect a problem here. I then work backward toward the granular rules. (Sometimes services require more permissions than you expect.)

Verify your settings and incoming data

Example below. Check the logs for results.

module.exports.handler = function(event, context) {
  // incoming data ok?
  console.log('event', JSON.stringify(event, null, 2));
  // env vars ok? (mostly set by AWS lambda, but might have some of yours)
  console.log('env', JSON.stringify(process.env, null, 2));
  // anything weird with the AWS service instance?
  console.log('dynamodb client', JSON.stringify(db._dynamodbClient));
  // what about the service config?
  console.log('dynamodb service', JSON.stringify(db._dynamodbClient.service, null, 2));
  // ...

Beware env credentials

This has bit me a couple times now and the solution isn't immediately obvious. Lambda automatically generates temporary aws credentials for the lambda function to use. These temporary credentials get added to process.env by Lambda

For some reason though, if you try to set these manually when creating a dynamodb client for example, it won't work.

It seems like best practice here is to always store credentials in environment and let the AWS sdk detect them.

Example:

This will work on local but fail on lambda:

var dynamodb = new AWS.DynamoDB.DocumentClient({
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  region: process.env.AWS_DEFAULT_REGION,
});

This works on lambda AND local (as long as your env vars are named properly):

var dynamodb = new AWS.DynamoDB.DocumentClient({ region: process.env.AWS_DEFAULT_REGION });

I'm not exactly sure why this works, but even moving that creation inside the handler scope doesn't fix it, so it doesn't seem to be a race with client creation and process.env being ready.

Edit: @epiphone might have the reason in his comment below. this could be session token being required in prod but not local. e.g. adding sessionToken: process.env.AWS_SESSION_TOKEN to your params. NOTE: if you're using cognito with an assumed identity this might also be the issue since cognito requests require the session token IIRC.

AWS.config.update()

I see a lot of people still landing on this gist and referencing AWS.config.update(). DO NOT USE THIS. ANYWHERE. It's cancer. Instead use the constructor of the service to pass this info in. e.g. new AWS.Lambda({ region: '...' })

The reason it's horrible is it's a race condition built directly into the mediocre (and that's being generous) aws-sdk js v2 client.

update() only impacts things created after it is called and not any clients created before. And then (unless it's changed) it's still possible to override these settings directly on the constructor. tl;dr it's cancer and it should be avoided.

Thank you so much! Trying to over-configure with AWS.Config.Update when i should've just let it do its thing

This, as the second result on my google search was so much more helpful than the first result in solving my bug.

I've been trying to fix this for days. Thanks a lot!

I had this bug for TextractClient when using .dev.env file. It works fine on .prod.env but .dev.env just refused to load the Textract client with this cryptic error.

I found a solution:

Simply don't try to set the credentials manually in the constructor. Let this beast of an object just pick up the credentials from the environment that were loaded with dotenv. Somehow this works and now all the tests pass in dev mode.

Scratch that. I got it working in local development but the Docker instance doesn't work. Sad.