jplsightm

import json

def dummy(element):
    return element

def get_value(_dict):
    try:
        return _dict.get('value', _dict)
    except AttributeError:
        return _dict

jplsightm

import json
from datetime import datetime
from random import randint
from time import sleep

a = {'timestamp': datetime.now().strftime("%m/%d/%Y, %H:%M:%S"), 'text': 'testing for the win, {}'.format(randint(0,10)), 'host': 'a_computer', 'attribute': 'abc'}
b = {'timestamp': datetime.now().strftime("%m/%d/%Y, %H:%M:%S"), 'text': 'here_we_are for the win, {}'.format(randint(0,10)), 'host': 'a_computer', 'attribute': 'abc'}
c = {'timestamp': datetime.now().strftime("%m/%d/%Y, %H:%M:%S"), 'text': 'testing for the win, {}'.format(randint(0,10)), 'host': 'b_computer', 'attribute': 'abc'}
d = {'timestamp': datetime.now().strftime("%m/%d/%Y, %H:%M:%S"), 'text': '{} - what now'.format(randint(0,10)), 'host': 'b_computer', 'attribute': 'abc'}
e = {'timestamp': datetime.now().strftime("%m/%d/%Y, %H:%M:%S"), 'text': 'testing for the win, {}'.format(randint(0,10)), 'host': 'c_computer', 'attribute': 'zyx'}

jplsightm

Enabling message search and pattern recognition with JSON logs in Datadog

It is generally recomended that logs sent to Datadog should be in a json format. This allows for attributes to be easily identified and thus avoids the need to created (what at times can be complex) grok processors. These attributes are important, because you can then create facets to help you identify the logs that are important to you without needing to understand a complex query langauge. The challenge with this is that the message of the log does not appear to be available. This can be easily resolved using the Message Remapper.

Setup and defining the problem.

I have created some sample logs that I am forwarding onto Datadog. A sample of how these logs look can be found below:

{"attribute": "abc", "timestamp": "10/14/2020, 14:15:04", "host": "a_computer", "text": "testing for the win, 10"}