tldr: check to make sure you are pointing your log requests to the http input port you set in the pipeline instad of the logstash api port
Some days you just want to hit your head against a wall. I have been having issues trying to figure out why my logstash logging endpoint was not working for the last month or two. (its been lower priority among other things.)
I recently migrated my stuff to a new kubernetes environment, and wanted to try out ECK to deploy my elasticsearch logstash etc. stuff. (Not that worth over just using bitnami helm unless you are deploying a LOT of elasticsearch clusters)
The one issue I was having was that I was unable to figure out how to get my logstash instance working again. All of my logs were 404ing.
After a lot of digging and port forwarding shenanigans in kubernetes, I FINALLY noticed the following log line when restarting my logstash instance.
Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
I had been pointing my services at port 9600 this entire time, which returns an ok as its a good port, but port 9600 was the logstash API port, not the http input port. This means that all of my logs were attempting to be routed to the api, which of course it would not have a post endpoint at /. Logstash would have no idea what to do with that endpoint.
Fixing the service redirect and making sure the ingress points to the right api, I now have a functioning logging endpoint again. Woo!
End result? check your ports to make sure you are connecting to the right ones beginning to end next time.