translation.rdoc

[ruby-dev:39421] exception from open-uri¶ ↑

Yusuke ENDOH writes:

Periodically when fetching web pages using open-uri, he notices that many different kinds of exceptions can be raised.

So far, he’s experienced these below:

• Errno::ETIMEDOUT

• OpenURI::HTTPError

• Errno::ECONNRESET

• Timeout::Error

• EOFError

• Errno::EHOSTUNREACH

• SocketError

Because of this, he has a few questions:

1) Are there any other exceptions open-uri can cause? (besides Errno::*)

2) Are any of these exceptions unintentional? EOFError seems suspicious. If this is a bug, could be caused by timeout or Net::HTTP, but he doesn’t know how to reproduce.

3) Under these conditions, Yusuke wonders how are open-uri users are supposed to do to deal with these issues. Yusuke lists four ways he can think of:

A. rescue from StandardError

B. Write code to deal with each individual exception

C. Programs that require robustness are based on the assumption that you
can't use open-uri or Net::HTTP.

D. Programs that require robustness are based on the assumption that you
can't use Ruby.

Yusuke then pastes various exceptions thrown from open-uri:

/usr/lib/ruby/1.8/net/http.rb:560:in ‘initialize’: Connection timed out - connect(2) (Errno::ETIMEDOUT)

from /usr/lib/ruby/1.8/net/http.rb:560:in `open'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/open-uri.rb:277:in ‘open_http’: 400 Bad Request (OpenURI::HTTPError)

from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/net/protocol.rb:135:in ‘sysread’: Connection reset by peer (Errno::ECONNRESET)

from /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
from /usr/lib/ruby/1.8/timeout.rb:62:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'
from /usr/lib/ruby/1.8/open-uri.rb:248:in `open_http'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/timeout.rb:60:in ‘rbuf_fill’: execution expired (Timeout::Error)

from /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'
from /usr/lib/ruby/1.8/open-uri.rb:248:in `open_http'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/net/protocol.rb:135:in ‘sysread’: end of file reached (EOFError)

from /usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
from /usr/lib/ruby/1.8/timeout.rb:62:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'
from /usr/lib/ruby/1.8/open-uri.rb:248:in `open_http'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/net/http.rb:560:in ‘initialize’: No route to host - connect(2) (Errno::EHOSTUNREACH)

from /usr/lib/ruby/1.8/net/http.rb:560:in `open'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

/usr/lib/ruby/1.8/net/http.rb:560:in ‘initialize’: getaddrinfo: Temporary failure in name resolution (SocketError)

from /usr/lib/ruby/1.8/net/http.rb:560:in `open'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'

[ruby-dev:39422] Tanaka Akira responds to Yusuke [ruby-dev:39421]¶ ↑

Tanaka says that coincidentally, he’s experienced the same problems with a program called “samidare”.

He notes a couple exceptions he’s experienced from his log:

Errno::ECONNREFUSED Errno::ENETUNREACH

He also notes that he complained about EOFError to Aoki san in the past.

> 3) Under these conditions, Yusuke wonders how are open-uri users are > supposed to do to deal with these issues. Yusuke lists four ways he can > think of: > > A. rescue from StandardError

Tanaka says that he uses this method for dealing with errors.

[ruby-dev:39426] Yusuke responds to Tanaka [ruby-dev:39422]¶ ↑

Yusuke thanks for Tanaka for the extra list of exceptions, and will deal with them for the time being.

Yusuke mentions that he’s thinking about filing the EOFError as a bug on redmine.

Yusuke notes that when using open-uri, the recommended idiom is to wrap with “rescue StandardError”. However, if someone wants to use open-uri with a block, that idiom would be inconvenient:

open("http://...") do |fh|
  ... # (A)
end

would become:

begin
  open("http://...") do |fh|
    ... # (A)
  end
rescue StandardError
  ... # (B)
end

He goes on to say that any exceptions caused in (A) would be rescued in (B). If an exceptional bug is caused in (A), it would be caught in (B), and he does not want that.

Yusuke suggests that maybe if any exceptions happen within open-uri, everything is captured, then turned in to errors with names like OpenURI::Error. Yusuke goes on to say that the user then only has to rescue OpenURI::Errors, and that might be easier.

[ruby-dev:39427] Tanaka responds to Yusuke’s email [ruby-dev:39426]¶ ↑

Tanaka says that he doesn’t particularly recommend the “rescue StandardError” pattern.

Tanaka does not think these problems are particular to open-uri.

He goes goes on to say that the problem is that open-uri block and rescue block are different, and hope there is a more general way to solve the problem.

He thinks a solution at this point is to use open-uri without a block. Also, if the thing to be read isn’t that big, he recommends using URI#read (samidare uses it).

Tanaka also says that if one wants to use open-uri with a block, maybe try this untested code:

done = false begin

open(uri) {|f|
  done = true
  ...
}

rescue StandardError

raise if done
...

end

[ruby-dev:39428] Yusuke response do Tanaka’s email [ruby-dev:39427]¶ ↑

Yusuke agrees with Tanaka that the exceptions are not just from open-uri, and he realized that right after sending his email.

Yusuke says that he thought open-uri may not be as valuable as Kernel#open, but now understands that they may have the same problems.

What pattern are people using in Ruby 1.9?

Any updates for Ruby 2.3?