Strips duplicated and repeated lines from stdin (such as a log output). Can also handles multiline repeats, up to a configurable memory limit.

logdeduplicator.py

import os
import sys

max_memory = int(os.environ.get('MAX_MEMORY', '10'))

recent_lines = []

for line in sys.stdin:
	if line not in recent_lines:
		sys.stdout.write(line)
	recent_lines.append(line)
	if len(recent_lines) > max_memory:
		recent_lines.pop(0)