Lambda calling lambda with trace id

index.js

// Lambda 1 - arv-foo
// Synchronous fn that will back API Gateway endpoint.

const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();

const getRandomInt = (max) => Math.floor(Math.random() * Math.floor(max));

exports.handler = async (event, context) => {
    console.log("event:", event)
    const traceId = (event.headers || {})["x-amzn-trace-id"] || getRandomInt(999999);
    
    const barPayload = {
        FunctionName: "arv-bar",
        Payload: JSON.stringify({ 
            caller: "Called from arv-foo",
            headers: {
               "x-amzn-trace-id": traceId, 
            },
        }),
        InvocationType: "Event",
    }
    
    const barResponse = await lambda.invoke(barPayload).promise();
    console.log('arv-bar response:', barResponse);

    const response = {
        statusCode: 200,
        body: JSON.stringify({ 
            message: 'Lambda arv-foo returning response',
            traceId,
        }),
    };
    return response;
};




// Lambda 2 - arv-bar
// Asynchronous worker fn that will do some hrd work.
exports.handler = async (event) => {
    console.log('arv-bar event:', event);
    await new Promise(r => setTimeout(r, 2000));
    const response = {
        statusCode: 200,
        body: JSON.stringify('Lambda arv-bar finished.'),
    };
    return response;
};



// Lambda 2 clouldwatch logs example:
2020-09-08T14:21:18.219Z	62b1bc63-2862-43a6-b0e6-44f1633b2e64	INFO	arv-bar event: {
  caller: 'Called from arv-foo',
  headers: { 'x-amzn-trace-id': 917093 }
}

I've messing around with X-Ray and Lambdas more this weekend...here's what I learned:

An alternative to grabbing the trace-id from the header is to grab it from process.env._X_AMZN_TRACE_ID.

AWS's default tracing allows me to use X-Ray to trace invocation and run of my Lambdas. The free tier is pretty accommodating, but even if I overshoot the limit it only costs something like $5 for one million traces.

I am manually chaining multiple Lambdas together, and want to capture a clean trace of my entire workflow:
API Gateway -> Lambda 1 -> Lambda 2 -> S3

But this requires wrapping any calls to aws-sdk (like Lambda.invoke()) so X-Ray can observe and capture each request:

	const AWSXRay = require('aws-xray-sdk');
	const AWS = AWSXRay.captureAWS(require('aws-sdk'));

Alas, X-Ray SDK is not available to Lambda in the runtime like AWS-SDK is. I'd need to bundle 'aws-xray-sdk' as an additional Layer in each Lambda. This can be done easily via the serverless framework, however each Layer adds bloat, and eventually cost, to the workflow. But unless I do this X-Ray will not associate my downstream Lambas with the original 'calling' Lambda, so my X-ray graphs and data will be separated:

Here is a blog post that goes into a lot more detail about what's going on here. It looks like AWS has resolved some of the pain points, but I don't see any solution for producing a clean graph that shows all the services stitched together nicely in X-Ray.

I think my best bet is to create a Log Group in CloudWatch that collects logs containing a custom Trace ID. But I do think the X-Ray tracing would be nice, especially if I went the next step to use multi-part uploading and inspect the upload jobs for performance, timeouts, etc. Doing this 'manually' in CloudWatch would mean a lot of custom logging and more time than I'd like to invest in the project.

Thought I'd share what I found! LMK if anything seems off to you in here :)