Logging Elasticsearch HTTP API Requests with Nginx

nginx.conf

worker_processes  1;

error_log /var/log/nginx/error.log;

events {
    worker_connections 1024;
}

http {
    log_format es '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent'
    '"$http_referer" "$http_user_agent" {$request_body}';
    access_log /var/log/nginx/access.log es;

    upstream elasticsearch {
        server 10.250.250.1:9200;
    }

    server {
        listen 8080;
        location / {
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_connect_timeout 5s;
            proxy_read_timeout 10s;
            proxy_pass http://elasticsearch;
        }
    }
}

This will create an nginx server listening on port 8080 that will proxy all requests it receives to Elasticsearch servers set in the upstream config section. It extends the standard combined log format to add the request body to the end of the log message. The request body is hex-encoded in the log file for safety. You'll need to parse them out when analysing the log file. You can replace \x with % to match URL-encoding, or just do the hex conversion yourself. If you put more than one server in the upstream section, the request are round-robined between them.

A more complete example can be found here.

Log messages in access.log look like the following:

10.250.250.1 - - [30/Oct/2015:03:50:58 +0000] "PUT /twitter/tweet/1 HTTP/1.1" 201 122"-" "curl/7.40.0" {{\x0A    \x22user\x22 : \x22kimchy\x22,\x0A    \x22post_date\x22 : \x222009-11-15T14:12:12\x22,\x0A    \x22message\x22 : \x22trying out Elasticsearch\x22\x0A}}

That's the log produced by the following request:

curl -XPUT http://localhost:8080/twitter/tweet/1 -d '{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

Hi, How to print "\x0A" directly, do not line feed?

like this:
log_format log_json escape=json '{"timestamp": "$time_local",'
'"remote_addr": "$remote_addr",'
'"referer": "$http_referer",'
'"request": "$request",'
'"status": "$status",'
'"byte": "$body_bytes_sent",'
'"agent": "$http_user_agent",'
'"x_forwarded_for": "$http_x_forwarded_for",'
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"up_resp_time": "$upstream_response_time",'
'"request_body": "$request_body",'
'"request_time": "$request_time"}';

make sure your nginx version >= 1.11.8
ref: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format