The thing that trips most people up is that the parameters to the web cli (--backend elasticsearch://blah:9300/blah) do NOT set the output destination.
The web app is its own process with its own args. It knows nothing about the agent config file.
Remember that you could simply use the same jar like so:
java -jar logstash.jar web --backend elasticsearch://elasticsearch:9300/clustername
and run the web interface without the agent process.
What also trips people up is port assignments. Logstash connects to elasticsearch as a transport client. This means it becomes a part of the elasticsearch cluster but does not store any data.
If you're running agent, web and an external ES process on the same machine, all three processes will try and use port 9300. This won't work. The problem is that it MIGHT work depending on startup order. If you start ES first, it will grab port 9300. When logstash comes up, the agent and web processes will increment to 9301 and 9302.
Now let's imagine you shut everything down and start logstash first. Likely agent will get 9300 and web will get 9301. I'm not positive here but ES will I THINK jump to 9302. The problem is that you've told the web to talk to 9300 which is now the agent.
The best bet here if you're running on the same machine is to lock ES to a different port and use that. That way there's never an issue. This could cause problems when you start to cluster though. So really if you're running ES externally from logstash (i.e. not embedded) run it on a different instance/machine.
Elasticsearch maintains compatibility only between minor versions. All clients and servers connecting via transport MUST be on the same major version.
This means that 0.18.x nodes can talk to a 0.18.y server regardless of what x and y are. A 0.19.x client or server cannot talk to a 0.18.x client or server.
If you're trying to run ES on the same machine as your logstash web or agent process, just use the embedded ES. You aren't buying yourself anything.
I am not very clear on the entire process of installing logstash,elasticsearch & kibana. I realize that logstash has huge memory requirements and installing all these components on one machine is not recommended. I have logstash, elasticsearch and kibana all installed on one server. I send messages to logstash which then gets indexed in elasticsearch, via a tcp port connection. I would like to send messages to logstash via another tcp port but not sure if that can work. Do you think this kind of setup is ok and worth running?I will delete the daily indexes to maintain disk space. I cannot recall how to assign logstash to listen to a specific port. Do you have any ideas or advice about this?