Skip to content

Instantly share code, notes, and snippets.

View portante's full-sized avatar

Peter Portante portante

View GitHub Profile
file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %.10000m%n"
@portante
portante / ProcessJSONLogs
Created March 14, 2017 22:25
Simple Python program to process log files that have unsorted JSON documents, one per line
#!/usr/bin/env python
import os, sys, json
from operator import itemgetter
lines = []
try:
with open(sys.argv[1], "r") as fp:
for line in fp.readlines():
doc = json.loads(line)
@portante
portante / gist:4754141ecf663ac49969142b059a52cd
Created March 10, 2017 02:35
Example of update HIL class master branch with upstream master
$ git co -b update-class-master class/master
Branch update-class-master set up to track remote branch master from class.
Switched to a new branch 'update-class-master'
$ git branch --set-upstream-to=upstream/master
Branch update-class-master set up to track remote branch master from upstream.
$ git status
On branch update-class-master
Your branch is behind 'upstream/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
nothing to commit, working tree clean
@portante
portante / glusterfs.grok
Created September 27, 2016 18:39
GlusterFS grok pattens
GTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
GMESSAGEID \[MSGID: %{INT:msgid}\]
GSRCINFO \[%{NOTSPACE:srcfile}:%{INT:linenum}:%{NOTSPACE:funcname}\]
GLOGMSGID \[%{GTIMESTAMP:timestamp}\] %{NOTSPACE:level}%{SPACE}%{GMESSAGEID}%{SPACE}%{GSRCINFO}%{SPACE}%{NOTSPACE:subvol}:%{SPACE}%{GREEDYDATA:message}
GLOGNOMSGID \[%{GTIMESTAMP:timestamp}\] %{NOTSPACE:level} %{GSRCINFO} %{NOTSPACE:subvol}:%{SPACE}%{GREEDYDATA:message}
GLOGREPEAT The message \"%{NOTSPACE:level}%{SPACE}%{GMESSAGEID}%{SPACE}%{GSRCINFO}%{SPACE}%{NOTSPACE:subvol}:%{SPACE}%{GREEDYDATA:message}\" repeated %{INT:repeat_count} times between \[%{GTIMESTAMP:timestamp}\] and \[%{GTIMESTAMP:repeat_end}\]
GLUSTERSTACK \[%{GTIMESTAMP:timestamp}\] %{NOTSPACE:level} %{GSRCINFO} \((?'traceback'-->%{SPACE}?%{PATH}%{SPACE}?\(%{NOTSPACE}\) \[%{NOTSPACE}\](%{SPACE}|%{SPACE}\g'traceback'))%{SPACE}\)%{SPACE}%{GREEDYDATA:message}
GLUSTERASSERT \[%{GTIMESTAMP:timestamp}\] %{NOTSPACE:level} %{GSRCINFO} \(--> %{INT}-%{NOTSPACE:component}?: %{GREEDYDATA:message}
@portante
portante / dotrace
Last active August 4, 2016 16:39
A quick tool to issue a stat on files give a set of GFIDs from Gluster volume heal info
#!/bin/bash
function getparent {
#echo $1 $2
local link=$(readlink -n -v $1/$2)
if [ "x$link" == "x" ]; then
return
fi
p_gfid_dn=$(dirname $1/$link)
p_gfid=$(basename $p_gfid_dn)
if [ "$p_gfid" != "00000000-0000-0000-0000-000000000001" ]; then
@portante
portante / elasticsearch.patriots.yml
Created July 8, 2016 04:04
This is the ES .yml file for a master node in our 6 node ES cluster. We have three master & data nodes, and 3 data nodes. Each node has 12 1 TB disks, mounted at /srv/es/{0..B}. We put the ES work directory on the first disk. This is currently used by ES 1.5.x running under RHEL 7.2.
cluster.name: elasticsearch.patriots
action.disable_delete_all_indices: true
bootstrap.mlockall: true
node.name: "brady"
node.master: true
node.data: true
@portante
portante / es-create-index-aliases.py
Created July 8, 2016 03:57
This is a simple script which creates index aliases for index names of the pattern: "<prefix>[.name].(sar|sosreport)-*", where the "name" is optional. We have daily indices for *.sar-* indices, and monthly indices for *.sosreport-* indices. This will create aliases for various relative groupings based on the table.
#!/usr/bin/env python3
#
# es-create-index-aliases
#
# Create the index aliases, making it easier to reference groupings of the
# time series data found there-in.
#
from __future__ import print_function
@portante
portante / fluent.conf
Created April 5, 2016 10:15
A fluent.conf for OpenStack in need of a position file.
# Nova compute
<source>
@type tail
path /var/log/nova/nova-compute.log
tag nova.compute
# The first line of a multiline log entry begins with a timestamp, YYYY-MM-DD
# HH:MM:SS, and an optional sub-second value using 3 or 6 digits, followed by
# pid, level, and class. To determine a "first line" we only need to match
# the beginning part of it, so for OpenStack log lines, those four fields are
@portante
portante / comment.txt
Last active March 18, 2016 14:47
Comment on Patch Set #4 posted March 10th for http://review.gluster.org/#/c/12250/
> I like where this is going. Perhaps we could consider the steps an
> admin has to take and orient the commands to address what they need
> to do.
>
> First, having to "kill -9" a process as an interface to stopping a
> brick seems dicey.
>
> So to that end, perhaps we could two sets of commands, the first to
> tell gluster to take the brick out of service, allowing the admin
> to then replace it, and a second command to brick the brick back
@portante
portante / 00-tags.conf
Last active December 19, 2015 15:18
EPEL 7 rsyslog configuration for forwarding log data as JSON
# LIVES in /etc/rsyslog.d/ ***
#
# A simple string of "tags" that system administrators can use to identify the
# various roles a system can take on, or categories to consider it under. Use
# whitespace to separate the tags.
set $!tags = "EXAMPLETAG EXAMPLETAGTWO EXAMPLETAGTHREE";