All on Ruby 1.9.3p395 from Sidekiq workers on Heroku unless otherwise stated.
-
Feedzirra
This uses curb to fetch feeds, which worked fine locally but under load would eventually segfault. This happened on our own roll-your-own VPS and on Heroku.
-
Net::HTTP (this was Ruby 2.0pxxx)
After a long run downloading feeds we would start to see HTTP requests taking minutes with a 30 second timeout. Eventually all of our workers in Sidekiq would be stalled. I have read conflicting information about whether timeout.rb is to be trusted in MRI 1.9+, but it seems that in our case it shouldn't have been.
We observed other instabilities on Ruby 2.0 (pg prepared statement errors) in the same setup, so it's conceivable that some pathologocal interaction code have cause what we observed (hung threads holding the GIL could do this).
-
Faraday/Net::HTTP
Strangely, this combination kind of worked except we'd see a bunch of responses in the logs that had a nil status. This was absolutely reproducable locally in the Rails console so I'll get an example together and file a bug report.
-
Faraday/em_http
I had my doubts as to whether or not this would work due to not being entirely certain about the relationship between the EM reactor and the threads running in Sidekiq. I don't recall the specific failure.
-
Typhoeus
This segfaulted almost immediately. This almost certainly to do with whatever version of libcurl it was using on Heroku.
Perhaps related: typhoeus/typhoeus#301 And (section "Multithreading Issues"): http://curl.haxx.se/libcurl/c/libcurl-tutorial.html