| op rate : 50451 [WRITE:50451] | |
| partition rate : 50451 [WRITE:50451] | |
| row rate : 50451 [WRITE:50451] | |
| latency mean : 3.9 [WRITE:3.9] | |
| latency median : 2.0 [WRITE:2.0] | |
| latency 95th percentile : 6.2 [WRITE:6.2] | |
| latency 99th percentile : 40.2 [WRITE:40.2] | |
| latency 99.9th percentile : 82.8 [WRITE:82.8] | |
| latency max : 393.4 [WRITE:393.4] |
| FROM ubuntu:14.04.1 | |
| MAINTAINER Wei-Ming Wu <[email protected]> | |
| ADD chkconfig /sbin/chkconfig | |
| ADD init.ora / | |
| ADD initXETemp.ora / | |
| @PARTS@ | |
| RUN cat /@[email protected]?? > /@[email protected] |
| input { | |
| elasticsearch { | |
| hosts => [ "HOSTNAME_HERE" ] | |
| port => "9200" | |
| index => "INDEXNAME_HERE" | |
| size => 1000 | |
| scroll => "5m" | |
| docinfo => true | |
| scan => true | |
| } |
Spin up three m3.medium EC2 ubuntu 14.04 instances with public DNS enabled and configure them for high network traffic by increasing these limits:
Added fs.file-max=80000 to /etc/sysctl.conf
Added the following lines to /etc/security/limits.conf
OK, you can pretty much ignore what I wrote below this update, because it doesn't really apply anymore.
I wrote this over a year ago, and at the time I had spent a couple of weeks trying to get Kafka 0.8 working with .NET and then Node.js with much frustration and very little success. I was rather angry. It keeps getting linked, though, and just popped up on Hacker News, so here's sort of an update, although I haven't used Kafka at all this year so I don't really have any new information.
In the end, we managed to get things working with a Node.js client, although we continued to have problems, both with our code and with managing a Kafka/Zookeeper cluster generally. What made it worse was that I did not then, and do not now, believe that Kafka was the correct solution for that particular problem at that particular company. What they were trying to achieve could have been done more simply with any number of other messaging systems, with a subscriber reading messages off and writing
| [ | |
| { | |
| "_id": "Wildfly-Monitoring-Dashboard", | |
| "_type": "dashboard", | |
| "_source": { | |
| "title": "Wildfly Monitoring Dashboard", | |
| "hits": 0, | |
| "description": "", | |
| "panelsJSON": "[{\"col\":7,\"id\":\"Active-Sessions-Statistics-jboss-hibernate4.war\",\"row\":3,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":10,\"id\":\"Datasource-Pool-Statistics-jboss-hibernate4.war\",\"row\":3,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":7,\"id\":\"Http-Requests-By-Client-IP\",\"row\":1,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":10,\"id\":\"Http-Response-Status\",\"row\":1,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":7,\"id\":\"Prepared-Statement-Cache-Statistics-jboss-as-hibernate4\",\"row\":5,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"id\":\"ActiveMQ-Broker-Memory-Metrics\",\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"ActiveMQ-Broker-Total-Metrics\",\"row\":4,\"size_x\" |
| #!/bin/bash -xv | |
| ### Country list | |
| # CHANGE WITH YOUR COUNTRIES | |
| COUNTRIES="europe/france/guadeloupe europe/france/guyane europe/france/mayotte europe/france/martinique europe/france/reunion europe/france" | |
| # SMALL SUBSET FOR TESTING: | |
| COUNTRIES="europe/france/guyane europe/france/mayotte" | |
| NOMINATIM="/var/www/Nominatim" | |
| cd $NOMINATIM/data |
| # Author: Aram Grigorian <[email protected]> | |
| # https://github.com/aramg | |
| # https://github.com/opendns | |
| # | |
| # By default, nginx will close upstream connections after every request. | |
| # The upstream-keepalive module tries to remedy this by keeping a certain minimum number of | |
| # persistent connections open at all times to upstreams. These connections are re-used for | |
| # all requests, regardless of downstream connection source. There are options available | |
| # for load balacing clients to the same upstreams more consistently. | |
| # This is all designed around the reverse proxy case, which is nginxs main purpose. |
| # command to generate dhparams.pen | |
| # openssl dhparam -out /etc/nginx/conf.d/dhparams.pem 2048 | |
| limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m; | |
| limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s; | |
| limit_req_status 444; | |
| limit_conn_status 503; | |
| proxy_cache_path /var/lib/nginx/proxy levels=1:2 keys_zone=backcache:8m max_size=50m; | |
| proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args"; |
