imweijh

On Tue Oct 27, 2015, history.state.gov began buckling under load, intermittently issuing 500 errors. Nginx's error log was sprinkled with the following errors:

2015/10/27 21:48:36 [crit] 2475#0: accept4() failed (24: Too many open files) 2015/10/27 21:48:36 [alert] 2475#0: *7163915 socket() failed (24: Too many open files) while connecting to upstream...

An article at http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/ provided directions that mostly worked. Below are the steps we followed. The steps that diverged from the article's directions are marked with an *.

1. • Instead of using su to run ulimit on the nginx account, use ps aux | grep nginx to locate nginx's process IDs. Then query each process's file handle limits using cat /proc/pid/limits (where pid is the process id retrieved from ps). (Note: sudo may be necessary on your system for the cat command here, depending on your system.)
2. Added fs.file-max = 70000 to /etc/sysctl.conf
3. Added `nginx soft nofile 1

imweijh

/var/www/example.com/logs/*.log {
        daily
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate

imweijh

#!/bin/bash
echo "Building NGINX along with Echo module"

# install prerequisites
yum -y install gcc gcc-c++ make zlib-devel pcre-devel openssl-devel

# download the Echo module
curl -L -O 'https://github.com/openresty/echo-nginx-module/archive/v0.58.tar.gz'
tar -xzvf v0.58.tar.gz && rm v0.58.tar.gz
mv echo-nginx-module-0.58 /tmp/echo-nginx-module

imweijh

{
  "metrics" : {
    "order" : 0,
    "template" : "metrics-*",
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {

imweijh

--[[
Provides custom authorization for nginx.
See the `nginx_authorize_by_lua.conf` for the Nginx config. This lua file is referenced in the config
See testWebserverAccess.sh for unit tests.

To Run nginx (make sure you have the lua, config and htpasswd file):
$ /usr/local/openresty/nginx/sbin/nginx -c /etc/nginx/conf/nginx_authorize_by_lua.conf

Logs are available at: /usr/local/openresty/nginx/logs/lua.log
To write to the log:

imweijh

yum install -y wget
wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
mv rsyslog.repo /etc/yum.repos.d/rsyslog.repo
yum info rsyslog --skip-broken
yum install -y rsyslog
rsyslogd -version

imweijh

input {
  generator {
    count => 1
    lines => [
      "qid=ABCD1 first",
      "qid=XYZ2 first",
      "qid=ABCD1 second",
      "qid=XYZ2 second",
      "qid=ZZZZ first",
      "qid=ABCD1 third"

imweijh

AUDIT type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): user pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} auid=%{NUMBER:audit_audid} subj=%{WORD:audit_subject} msg=%{GREEDYDATA:audit_message} 
AUDITLOGIN type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): login pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} old auid=%{NUMBER:old_auid} new auid=%{NUMBER:new_auid} old ses=%{NUMBER:old_ses} new ses=%{NUMBER:new_ses}

imweijh

$thres=0.02, .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).color(#eee).lines(10).label('Prediction'), .es(index='metricbeat*',metric='max:system.cpu.user.pct').color(#666).lines(1).label(Actual), .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).subtract(.es(index='metricbeat*',metric='max:system.cpu.user.pct')).abs().if(lt, $thres, null, .es(index='metricbeat*',metric='max:system.cpu.user.pct')).points(10,3,0).color(#c66).label('Anomaly').title('max:system.cpu.user.pct / @rmoff')

imweijh

package main

import (
	"log"
	"bufio"
	"time"
	"os"
	"fmt"
	"io"
	"net"