Diagnostics for EAGAIN error in GHC Builder on SmartOS

diagnostics.md

Builder Client encounters errors like:

builder-client: fd:11: hGetLine: resource exhausted (Resource temporarily unavailable)

The string resource exhausted comes from the Show instance for IOErrorType:

• https://github.com/ghc/ghc/blob/f0fcc41d755876a1b02d1c7c79f57515059f6417/libraries/base/GHC/IO/Exception.hs#L290

The string Resource temporarily unavailable comes from calling strerror on the underlying SmartOS system:

• https://github.com/ghc/ghc/blob/41f5b7e3e0648302b9c5dc485917a391d21d15a1/libraries/base/Foreign/C/Error.hs#L468
• https://github.com/illumos/illumos-gate/blob/7256a34efe9df75b638b9e812912ef7c5c68e208/usr/src/lib/libc/port/gen/errlist#L50

EAGAIN is mapped to ResourceExhausted in the Foreign C Error mapping in base:

• https://github.com/ghc/ghc/blob/41f5b7e3e0648302b9c5dc485917a391d21d15a1/libraries/base/Foreign/C/Error.hs#L480

The error message as whole is formatted by the Show instance of IOException:

• https://github.com/ghc/ghc/blob/f0fcc41d755876a1b02d1c7c79f57515059f6417/libraries/base/GHC/IO/Exception.hs#L321-334

hGetLine calls hGetLineBuffered calls hGetLineBufferedLoop calls maybeFillReadBuffer which considers anything other than EOF an error:

• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLine
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLineBuffered
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLineBufferedLoop
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#maybeFillReadBuffer

This flaw might be in using hGetLine on a non-blocking Handle at which point the error lies in ghc-builder and not in hGetLine itself.

This appears to be triggered by getLines in Builder.Command in GHC builder:

• https://github.com/haskell/ghc-builder/blob/master/common/Builder/Command.hs

It is calling hGetLine on the handles returned from runInteractiveProcess which are binary by default:

• http://hackage.haskell.org/package/process-1.2.0.0/docs/src/System-Process.html#runInteractiveProcess

References:

• https://us-east.manta.joyent.com/smartosman/public/man2/read.2.html
• https://us-east.manta.joyent.com/smartosman/public/man2/open.2.html
• https://us-east.manta.joyent.com/smartosman/public/man2/Intro.2.html

Further information:

Prelude> import GHC.IO.Handle
Prelude GHC.IO.Handle> import System.Process
Prelude GHC.IO.Handle System.Process> let (prog, args) = ("yes", [])
Prelude GHC.IO.Handle System.Process> hGetLine hIn
*** Exception: fd:9: hGetLine: illegal operation (handle is not open for reading)
Prelude GHC.IO.Handle System.Process> hIn
{handle: fd:9}
Prelude GHC.IO.Handle System.Process> hGetLine hOut
"y"
Prelude GHC.IO.Handle System.Process> hGetLine hErr

That last operation blocks in GHCi indicating that it's correcting handling the EAGAIN. DTrace shows that the read call made by hGetline hErr is returning EAGAIN. This is expected.

Something different is occuring when run under SMF. The issue of EAGAIN leaking out as an IOError only occurs when the builder is run under SMF.

This issue does not occur if LC_ALL is set to en_US.UTF-8.

And now to close the loop as recommended by @rmustacc on #smartos.

This is easily reproducible in a login shell with "C" locale:

[root@ghc-builder-x86-64-smartos ~/builder]# unset LANG
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_CTYPE
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_NUMERIC
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_TIME
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_COLLATE
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_MONETARY
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_MESSAGES
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_ALL
[root@ghc-builder-x86-64-smartos ~/builder]# locale
LANG=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=
[root@ghc-builder-x86-64-smartos ~/builder]# builder-client --do-build
[2014-05-19 23:14:26] [smartos-x86_64-head] Connecting...
[2014-05-19 23:14:27] [smartos-x86_64-head] Connected.
[2014-05-19 23:14:27] [smartos-x86_64-head] Sending: "PROTO 0.3"
Received: "200 Protocol version OK"
[2014-05-19 23:14:27] [smartos-x86_64-head] Sending: "START SSL"
Received: "200 Welcome to SSL"
[2014-05-19 23:14:28] [smartos-x86_64-head] Sending: "AUTH smartos-x86_64-head nottheactualpassword"
Received: "200 authenticated"
[2014-05-19 23:14:28] [smartos-x86_64-head] Sending: "BUILD INSTRUCTIONS"
Received: "202 What sort?"
Received: "201 Instructions follow"
Received: "200 That's it"
Running "git clone"
Running "create mk/build.mk"
Running "get subrepos"
Running "repo versions"
Running "touching clean-check files"
Running "setting version date"
Running "booting"
builder-client: fd:12: hGetLine: resource exhausted (Resource temporarily unavailable)
[2014-05-19 23:22:09] [smartos-x86_64-head] Sending: "UPLOAD 75 1"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:13] [smartos-x86_64-head] Sending: "UPLOAD 75 2"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:17] [smartos-x86_64-head] Sending: "UPLOAD 75 3"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:20] [smartos-x86_64-head] Sending: "UPLOAD 75 4"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:24] [smartos-x86_64-head] Sending: "UPLOAD 75 5"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:28] [smartos-x86_64-head] Sending: "UPLOAD 75 6"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:32] [smartos-x86_64-head] Sending: "UPLOAD 75 7"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:36] [smartos-x86_64-head] Sending: "RESULT 75"
Received: "202 Send instructions"
Received: "202 Send result"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:36] [smartos-x86_64-head] Sending: "RESET TIME"
Received: "200 Done"