Change Python logger format in AWS Lambda

change_lambda_logger_format.py

# Python logger in AWS Lambda has a preset format. To change the format of the logging statement, 
# remove the logging handler & add a new handler with the required format

import logging
import sys

def setup_logging():
    logger = logging.getLogger()
    for h in logger.handlers:
      logger.removeHandler(h)
    
    h = logging.StreamHandler(sys.stdout)
    
    # use whatever format you want here
    FORMAT = '%(asctime)s %(message)s'
    h.setFormatter(logging.Formatter(FORMAT))
    logger.addHandler(h)
    logger.setLevel(logging.INFO)
    
    return logger

def lambda_handler(event, context):
    logger = setup_logging()
    logger.info("This is a test log statement!")
    return
    
# Expected output from Lambda:
# 
# START RequestId: 1a2b3c4d-abcd-1234-efgh-1a2b3c4d5e6f Version: $LATEST
# 2017-10-06 22:40:59,653 This is a test log statement!
# END RequestId: 1a2b3c4d-abcd-1234-efgh-1a2b3c4d5e6f
# REPORT RequestId: 1a2b3c4d-abcd-1234-efgh-1a2b3c4d5e6f	Duration: 0.41 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 21 MB

A bit inefficient but this worked for me:

log = logging.getLogger()
for h in log.handlers:
    h.setFormatter(logging.Formatter("%(aws_request_id)s [%(levelname)s] %(message)s"))

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

Why not just create a logger, add your handler and then set propagate to False?

    FORMAT = '%(asctime)s %(message)s'
    h = logging.StreamHandler(sys.stdout)
    h.setFormatter(logging.Formatter(FORMAT))

    logger = logging.getLogger(__name__)
    logger.addHandler(h)
    logger.setLevel(logging.INFO)
    logger.propagate = False

This would prevent anything being logged using logger from propagating up to parent loggers. In the Lambda case the logger that Lambda adds.

Or... use the aws-lambda-powertools logger which has a lot of this, structured logs, and you get things like setting correlation ids.

And you can customize the formatter if you really need via https://awslabs.github.io/aws-lambda-powertools-python/latest/core/logger/#bring-your-own-formatter

I prefer keep the number of 3rd party dependencies in my lambdas low but I do like to have control of the logging configuration. I have had some success using something similar to the following:

import logging
import logging.config


config = {
    'version': 1,
    'formatters': {
      'lambda': {
        'class': "logging.Formatter",
        'format': "%(asctime)s %(levelname)s %(aws_request_id)s %(module)s:%(lineno)d %(message)s"
      }
    },
    'filters': {
        'lambda': {
            '()': '__main__.LambdaLoggerFilter'
        }
    },
    'handlers': {
      'lambda': {
        'class': "logging.StreamHandler",
        'level': "DEBUG",
        'formatter': "lambda",
        'filters': ['lambda']
      }
    },
    'loggers': {
      'lambda': {
        'level': "DEBUG",
        'handlers': ["lambda"],
        'propagate': False
      }
    }
}

logging.config.dictConfig(config)

LOG = logging.getLogger('lambda')

def handle(event, context):
    LOG.info('Logging from Lambda with Request ID.')

Obviously this comes with the risk of assuming AWS do not change the runtime bootstrapping code (tested on python3.8 runtime).

I was able to retain aws_request_id in the log output:
https://gist.github.com/matthewpick/3aa01abfeda36eae717837a99994d3ed

has anyone managed to find a nice way to get "aws_request_id" when using the approach of removing the default handler and using your own?

Since the default handler's filter injects the aws_request_id into the record, if you remove the default handler, you will not be able to retrieve aws_request_id. Therefore, instead of removing the default handler and adding your own, you should just modify the formatter using setFormatter.

Also you can see the code of default handler and filter
https://github.com/aws/aws-lambda-python-runtime-interface-client/blob/849e874de01776cb386c18fb2c1731d43cd2b2f4/awslambdaric/bootstrap.py#L339C1-L342C20