converts looker logs from plain text log to json like format - usefull for importing logs to EBK or ELK

looker_log_jsonify.py

#!/usr/bin/env python
from json import dumps
import sys


while True:
    string = sys.stdin.readline()
    if not string:
        break

    try:
        level = 'UNKNOWN'
        if '[DEBUG|' in string:
            level = 'DEBUG'
        if '[VERBOSE|' in string:
            level = 'VERBOSE'
        if '[INFO|' in string:
            level = 'INFO'
        if '[WARN|' in string:
            level = 'WARN'
        if '[ERROR|' in string:
            level = 'ERROR'
        if '[FATAL|' in string:
            level = 'FATAL'

        tmp_string = string[string.find('] ::') - 20:string.find('] ::')]
        looker_log_type = string[string.find('] ::') - 20:string.find('] ::')][tmp_string.rfind('|') + 1:255].strip()

        result = dict(
            createtime=string[0:string.find('[')].strip().replace(' ', 'T').replace('T+', ' +'),
            looker_log_type=looker_log_type,
            level=level,
            message=string[string.find('::') + 3:string.find('::') + 1000].strip()
        )

        print(dumps(result))
    except Exception as error:
        print(error)

triggered via:

cd /tmp/ && wget https://gist.githubusercontent.com/sisu-frank-kutzey/02d54375ae3aed6d393701ab9cbdf8c0/raw/aae3e6c6bdb73844be40c370b31a55e5ec99003b/looker_log_jsonify.py -O /tmp/looker_log_jsonify.py && tail -f /var/log/looker.log | egrep --line-buffered 'WARN|ERROR|CRITICAL' | python /tmp/looker_log_jsonify.py >> /var/log/looker.log.json

Thanks for this @sisu-frank-kutzey! I wanted to capture all level of logs, including some tricky INFO messages that had queries with newlines in them, causing the log file to have multiple lines per log. Here is the tweaked code to do that:

try:
    level = 'UNKNOWN'
    if '[DEBUG|' in string:
      level = 'DEBUG'
    if '[VERBOSE|' in string:
      level = 'VERBOSE'
    if '[INFO|' in string:
      level = 'INFO'
    if '[WARN|' in string:
      level = 'WARN'
    if '[ERROR|' in string:
      level = 'ERROR'
    if '[FATAL|' in string:
      level = 'FATAL'

    if '::' in string: #this is a log start line
      #print the previous result
      print(dumps(result))

      #start building the next one
      tmp_string = string[string.find('] ::') - 20:string.find('] ::')]
      looker_log_type = string[string.find('] ::') - 20:string.find('] ::')][tmp_string.rfind('|') + 1:255].strip()
      message=string[string.find('::') + 3:string.find('::') + 1000].strip()

      result = dict(
        createtime=string[0:string.find('[')].strip().replace(' ', 'T').replace('T+', ' +'),
        looker_log_type=looker_log_type,
        level=level,
        message=string[string.find('::') + 3:string.find('::') + 1000].strip()
      )
    else:
        result['message'] = result['message'] += string
  except Exception as error:
    sys.stderr.write(error)