Skip to content

Instantly share code, notes, and snippets.

@timconradinc
timconradinc / gist:5593704
Created May 16, 2013 18:01
elasticsearch.yml
# Force all memory to be locked, forcing the JVM to never swap
bootstrap.mlockall: true
# Indexing Settings for Writes
indices.memory.index_buffer_size: 50%
index.refresh_interval: 30
index.translog.flush_threshold_ops: 50000
index.store.compress.stored: true
index.routing.allocation.total_shards_per_node: 6
# this configuration is to parse the syslog files and then put them into redis.
input {
file {
# this input is a dump of sendmail and proofpoint logs, it's really messy
type => "proofpoint_raw_type"
tags => "proofpoint_raw"
path => ["/home/conrad/testlog.log"]
}
#stdin {
# ElasticSearch Service
description "ElasticSearch"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
ElasticSearch Quick Guide
ElasticSearch for Logstash Overview
There are two ways to send data to ElasticSearch from Logstash. The first is the 'elasticsearch' output and the other is
the 'elasticsearch_http' output. In a nutshell, the 'elasticsearch' output is tightly coupled with your elasticsearch
cluster, and the 'elasticsearch_http' output isn't.
What does this mean? The 'elasticsearch' output will *always* start up a local ElasticSearch node and try to join it to
your ElasticSearch cluster. This has the end goal of making Logstash aware of your cluster - if a node goes down,
Failed to index an event, will retry {:exception=>org.elasticsearch.transport.RemoteTransportException: [logstash-es.example.edu][inet[/10.9.1.7:9300]][index],
May 30 23:59:51 pp-serve02 sendmail[3083]: r4V3xnMc003080: to=<a@example.edu>, delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=151450, relay=exch-hubeb1.exchmail.example.edu. [10.1.1.1], dsn=2.0.0, stat=Sent (<2e7250bda7ce4a90a1fbd35e26f57227@2128> [InternalId=41607625] Queued mail for delivery)
May 30 23:59:59 pp-serve03 sendmail[27124]: r4V3xxd8027124: Milter: connect: host=mail.example.org, addr=11.11.11.11, reject=550 5.7.0 Local Policy Violation - Blocked - see https://support.proofpoint.com/dnsbl-lookup.cgi?ip=11.11.11.11
May 30 23:59:45 pp-serve01 sendmail[17669]: r4V3xhGo017465: to=<sh@exmaple.edu>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=131517, relay=exch-hubab1.exchmail.example.edu. [1.1.1.1], dsn=2.0.0, stat=Sent (<!&!AAAAAAAAAAAYAAAAAAAAAH59QL7H5PFGpbUlvo6UBNjCgAAAEAAAAHiJqcYxittJt5L0Gxv4jpABAAAAAA==@exxample.com> [InternalId=33718689] Queued mail for delivery)
creates
@fields.(<!&!AAAAAAAAAAAYAAAAAAAAAH59QL7H5PFGpbUlvo6UBNjCgAAAEAAAAHiJqcYxittJt5L0Gxv4jpABAAAAAA=
input {
#
# this configuration is for both the info- and notice- files that
# proofpoint puts out.
#
# They need to have both the type of 'proofpoint_raw_type' and a tag
# of 'proofpoint_raw' for this all to work properly.
#
#file {
# type => "proofpoint_raw_type"
mutate {
type => "PaloAltoTestTrafficLog"
# for dns queries
replace => [ "SourceAddressFQDN", "%{SourceAddress}" ]
replace => [ "DestinationAddressFQDN", "%{DestinationAddress}" ]
}
dns {
type => "PaloAltoTestTrafficLog"
reverse => [ "SourceAddressFQDN","DestinationAddressFQDN" ]
HotKeySet("{ESC}", "Terminate")
While 1
MouseMove(Random(211,378),Random(464,592))
Local $max = Random(1,50)
Local $i = 1
While $i < $max
MouseDown("left")
MouseUp("left")
$i = $i + 1
function bob {
Param([string]$a, [string]$b)
Write-Host "a: $a b: $b"
}
bob "one" "two"