Solution to AWS Lambda node.js UnrecognizedClientException "The security token included in the request is invalid."

troubleshooting.md

Troubleshooting AWS unauthorized errors in lambda requests

This is mainly for node.js but might apply to other environments. Unsure.

If you are running a AWS Lambda function that calls another AWS service and getting an error about invalid tokens or other access denied errors, do this:

Check IAM

The role assigned to your lambda function will need permission to perform the actions. Check IAM and make sure the role has all the permissions.

I usually give Full Access to the service in question (e.g. DynamoDB) if I suspect a problem here. I then work backward toward the granular rules. (Sometimes services require more permissions than you expect.)

Verify your settings and incoming data

Example below. Check the logs for results.

module.exports.handler = function(event, context) {
  // incoming data ok?
  console.log('event', JSON.stringify(event, null, 2));
  // env vars ok? (mostly set by AWS lambda, but might have some of yours)
  console.log('env', JSON.stringify(process.env, null, 2));
  // anything weird with the AWS service instance?
  console.log('dynamodb client', JSON.stringify(db._dynamodbClient));
  // what about the service config?
  console.log('dynamodb service', JSON.stringify(db._dynamodbClient.service, null, 2));
  // ...

Beware env credentials

This has bit me a couple times now and the solution isn't immediately obvious. Lambda automatically generates temporary aws credentials for the lambda function to use. These temporary credentials get added to process.env by Lambda

For some reason though, if you try to set these manually when creating a dynamodb client for example, it won't work.

It seems like best practice here is to always store credentials in environment and let the AWS sdk detect them.

Example:

This will work on local but fail on lambda:

var dynamodb = new AWS.DynamoDB.DocumentClient({
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  region: process.env.AWS_DEFAULT_REGION,
});

This works on lambda AND local (as long as your env vars are named properly):

var dynamodb = new AWS.DynamoDB.DocumentClient({ region: process.env.AWS_DEFAULT_REGION });

I'm not exactly sure why this works, but even moving that creation inside the handler scope doesn't fix it, so it doesn't seem to be a race with client creation and process.env being ready.

Edit: @epiphone might have the reason in his comment below. this could be session token being required in prod but not local. e.g. adding sessionToken: process.env.AWS_SESSION_TOKEN to your params. NOTE: if you're using cognito with an assumed identity this might also be the issue since cognito requests require the session token IIRC.

AWS.config.update()

I see a lot of people still landing on this gist and referencing AWS.config.update(). DO NOT USE THIS. ANYWHERE. It's cancer. Instead use the constructor of the service to pass this info in. e.g. new AWS.Lambda({ region: '...' })

The reason it's horrible is it's a race condition built directly into the mediocre (and that's being generous) aws-sdk js v2 client.

update() only impacts things created after it is called and not any clients created before. And then (unless it's changed) it's still possible to override these settings directly on the constructor. tl;dr it's cancer and it should be avoided.

I still have the problem.
My lambda works correctly for a moment (sometimes 2 hours, sometimes 10 min), then it stops again with the error message:
[UnrecognizedClientException: The security token included in the request is invalid]
In another lambda (with less requests to DynamoDB), it's always working well.

To resolve issues with not being able to specify aws credentials for the client(like if you want to assume a role to call cross account)

var creds = new AWS.Credentials({accessKeyId: access_id, secretAccessKey: access_key, sessionToken: session_token});
var lambda = new AWS.Lambda({credentials: creds, signatureCache: false});

wow alexa skill kit and lambda was working on my local but not on lambda following your advise , I have changed my dynamodb setting from

AWS.config.update({
        credentials: {
            accessKeyId: process.env.AWS_ACCESS_KEY_ID,
            secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
        },
        region: process.env.AWS_DEFAULT_REGION
    });

to

AWS.config.update({
        region: process.env.AWS_DEFAULT_REGION
    });

and it worked. took me hours to find this post , thanks

I am using :

AWS.config.update({
region: process.env.AWS_DEFAULT_REGION
});

Gut still getting UnrecognizedClientException "The security token included in the request is invalid." periodically. Even tried setting the DynamoDB retry options as:
var dynamodb = new AWS.DynamoDB({maxRetries: 5, retryDelayOptions: {base: 2000} });
var docClient = new AWS.DynamoDB.DocumentClient({service : dynamodb});

Seems like the Lambda updates the temporary aws credentials periodically and the AWS.DynamoDB.DocumentClient calls fail as this is happening. Has somebody run into this situation and what was the fix? All help will be appreciated.

Has anyone solved this issue? I'm having the same problem as well. Can't access my DynamoDB.DocumentClient in Lambda even though I've made sure the correct execution role policy is set in the Lambda console. The issue is driving me bonkers.

This will work on local but fail on lambda:
var dynamodb = new AWS.DynamoDB.DocumentClient({
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
region: process.env.AWS_DEFAULT_REGION,
});

Try adding sessionToken: process.env.AWS_SESSION_TOKEN to the DocumentClient constructor params.

thanks @epiphone! added your note to the doc since that's likely the root cause. cool, thanks!

I have a lambda which accesses the Neptune , same error, if I switch to hardcode my admin key and secret, it works, however when I use the key and secret provided by lambda (process.env.*), it says The security token included in the request is invalid, I am sure I have all the policy attached to the lambda execute role, I am stuck now.

Thank you so much! Trying to over-configure with AWS.Config.Update when i should've just let it do its thing

This, as the second result on my google search was so much more helpful than the first result in solving my bug.

I've been trying to fix this for days. Thanks a lot!

I had this bug for TextractClient when using .dev.env file. It works fine on .prod.env but .dev.env just refused to load the Textract client with this cryptic error.

I found a solution:

Simply don't try to set the credentials manually in the constructor. Let this beast of an object just pick up the credentials from the environment that were loaded with dotenv. Somehow this works and now all the tests pass in dev mode.

Scratch that. I got it working in local development but the Docker instance doesn't work. Sad.