Diagnostics for EAGAIN error in GHC Builder on SmartOS

diagnostics.md

Builder Client encounters errors like:

builder-client: fd:11: hGetLine: resource exhausted (Resource temporarily unavailable)

The string resource exhausted comes from the Show instance for IOErrorType:

• https://github.com/ghc/ghc/blob/f0fcc41d755876a1b02d1c7c79f57515059f6417/libraries/base/GHC/IO/Exception.hs#L290

The string Resource temporarily unavailable comes from calling strerror on the underlying SmartOS system:

• https://github.com/ghc/ghc/blob/41f5b7e3e0648302b9c5dc485917a391d21d15a1/libraries/base/Foreign/C/Error.hs#L468
• https://github.com/illumos/illumos-gate/blob/7256a34efe9df75b638b9e812912ef7c5c68e208/usr/src/lib/libc/port/gen/errlist#L50

EAGAIN is mapped to ResourceExhausted in the Foreign C Error mapping in base:

• https://github.com/ghc/ghc/blob/41f5b7e3e0648302b9c5dc485917a391d21d15a1/libraries/base/Foreign/C/Error.hs#L480

The error message as whole is formatted by the Show instance of IOException:

• https://github.com/ghc/ghc/blob/f0fcc41d755876a1b02d1c7c79f57515059f6417/libraries/base/GHC/IO/Exception.hs#L321-334

hGetLine calls hGetLineBuffered calls hGetLineBufferedLoop calls maybeFillReadBuffer which considers anything other than EOF an error:

• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLine
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLineBuffered
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#hGetLineBufferedLoop
• http://hackage.haskell.org/package/base-4.7.0.0/docs/src/GHC-IO-Handle-Text.html#maybeFillReadBuffer

This flaw might be in using hGetLine on a non-blocking Handle at which point the error lies in ghc-builder and not in hGetLine itself.

This appears to be triggered by getLines in Builder.Command in GHC builder:

• https://github.com/haskell/ghc-builder/blob/master/common/Builder/Command.hs

It is calling hGetLine on the handles returned from runInteractiveProcess which are binary by default:

• http://hackage.haskell.org/package/process-1.2.0.0/docs/src/System-Process.html#runInteractiveProcess

References:

• https://us-east.manta.joyent.com/smartosman/public/man2/read.2.html
• https://us-east.manta.joyent.com/smartosman/public/man2/open.2.html
• https://us-east.manta.joyent.com/smartosman/public/man2/Intro.2.html

This issue does not occur if LC_ALL is set to en_US.UTF-8.

And now to close the loop as recommended by @rmustacc on #smartos.

This is easily reproducible in a login shell with "C" locale:

[root@ghc-builder-x86-64-smartos ~/builder]# unset LANG
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_CTYPE
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_NUMERIC
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_TIME
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_COLLATE
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_MONETARY
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_MESSAGES
[root@ghc-builder-x86-64-smartos ~/builder]# unset LC_ALL
[root@ghc-builder-x86-64-smartos ~/builder]# locale
LANG=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=
[root@ghc-builder-x86-64-smartos ~/builder]# builder-client --do-build
[2014-05-19 23:14:26] [smartos-x86_64-head] Connecting...
[2014-05-19 23:14:27] [smartos-x86_64-head] Connected.
[2014-05-19 23:14:27] [smartos-x86_64-head] Sending: "PROTO 0.3"
Received: "200 Protocol version OK"
[2014-05-19 23:14:27] [smartos-x86_64-head] Sending: "START SSL"
Received: "200 Welcome to SSL"
[2014-05-19 23:14:28] [smartos-x86_64-head] Sending: "AUTH smartos-x86_64-head nottheactualpassword"
Received: "200 authenticated"
[2014-05-19 23:14:28] [smartos-x86_64-head] Sending: "BUILD INSTRUCTIONS"
Received: "202 What sort?"
Received: "201 Instructions follow"
Received: "200 That's it"
Running "git clone"
Running "create mk/build.mk"
Running "get subrepos"
Running "repo versions"
Running "touching clean-check files"
Running "setting version date"
Running "booting"
builder-client: fd:12: hGetLine: resource exhausted (Resource temporarily unavailable)
[2014-05-19 23:22:09] [smartos-x86_64-head] Sending: "UPLOAD 75 1"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:13] [smartos-x86_64-head] Sending: "UPLOAD 75 2"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:17] [smartos-x86_64-head] Sending: "UPLOAD 75 3"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:20] [smartos-x86_64-head] Sending: "UPLOAD 75 4"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:24] [smartos-x86_64-head] Sending: "UPLOAD 75 5"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:28] [smartos-x86_64-head] Sending: "UPLOAD 75 6"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:32] [smartos-x86_64-head] Sending: "UPLOAD 75 7"
Received: "202 Send name"
Received: "202 Send subdir"
Received: "202 Send program"
Received: "202 Send args"
Received: "202 Send mailOutput"
Received: "202 Send start time"
Received: "202 Send end time"
Received: "202 Send exit code"
Received: "202 Send output"
Received: "202 Send fileUploaded"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:36] [smartos-x86_64-head] Sending: "RESULT 75"
Received: "202 Send instructions"
Received: "202 Send result"
Received: "200 Got it, thanks!"
[2014-05-19 23:22:36] [smartos-x86_64-head] Sending: "RESET TIME"
Received: "200 Done"