Anil Yadav anilyadav

parsing mysql query logs with logstash

The problem is that some lines in the file are missing timestamps when they aren't continuations of any previous line. It's dumb, really.

The mysql query log is seriously bullshit format, but nothing logstash can't unscrew.

The main goal here is to show how we can fix the 'missing timestamp' problem.

% ruby bin/logstash agent -e '

Sed Cheat Sheet

Sed command line options

sed [options] sed-command [input-file]

Option	Description	Example

SQL to MongoDB Mapping Chart

SQL to MongoDB Mapping Chart¶

In addition to the charts that follow, you might want to consider the Frequently Asked Questions section for a selection of common questions about MongoDB.

Executables¶

The following table presents the MySQL/Oracle executables and the corresponding MongoDB executables.

	#!/bin/sh
	#
	# Script to run innobackupex script (for all databases on server), check for success, and apply logs to backups.
	#
	# (C)2010 Owen Carter @ Mirabeau BV
	# This script is provided as-is; no liability can be accepted for use.
	# You are free to modify and reproduce so long as this attribution is preserved.
	#

	# contribution: 2011-12-07 Tom De Vylder - modified to work with PerconaDB

	#!/bin/sh
	#
	# Script to prepare and restore full and incremental backups created with innobackupex-runner.
	#
	# This script is provided as-is; no liability can be accepted for use.
	#

	INNOBACKUPEX=innobackupex-1.5.1
	INNOBACKUPEXFULL=/usr/bin/$INNOBACKUPEX
	TMPFILE="/tmp/innobackupex-restore.$$.tmp"

	# Elasticsearch Cheatsheet - an overview of commonly used Elasticsearch API commands

	# cat paths
	/_cat/allocation
	/_cat/shards
	/_cat/shards/{index}
	/_cat/master
	/_cat/nodes
	/_cat/indices
	/_cat/indices/{index}

	-- show running queries (pre 9.2)
	SELECT procpid, age(query_start, clock_timestamp()), usename, current_query
	FROM pg_stat_activity
	WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%'
	ORDER BY query_start desc;

	-- show running queries (9.2)
	SELECT pid, age(query_start, clock_timestamp()), usename, query
	FROM pg_stat_activity
	WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'


	FILE SPACING:

	# double space a file
	sed G

	# double space a file which already has blank lines in it. Output file
	# should contain no more than one blank line between lines of text.
	sed '/^$/d;G'

	#!/bin/bash -e

	mkdir report
	rpm -qa > ./report/rpm.txt
	journalctl -b > ./report/journalctl.txt
	df -k > ./report/df.txt
	netstat > ./report/netstat.txt

	sar -P ALL > ./report/sar.txt
	iostat -d 1 10 > ./report/iostat.txt

	rm $HOME/.mysql_history
	ln -s /dev/null $HOME/.mysql_history