Testing gnu parallel behaviour

random_duration_script.py

#!/usr/bin/python3
import sys
from time import sleep
import random

for i in range(random.randint(3, 8)):
    sleep(random.random() / 10)
    print(f"{sys.argv[1]}, step {i}")
    if i == 2 and ("arg1" in sys.argv[1] or "arg2" in sys.argv[1]):
        raise ValueError(f"{sys.argv[1]} raised an exception")

Testing how parallel deals with collecting stdout, stderr and error codes in the case where we require running multiple commands on each process (in this case an echo header and footer as per our requirement to use parallel as part of github actions).

$ parallel 'echo ::group::{}; ./random_duration_script.py {}; echo ::endgroup::' ::: arg1 arg2 arg3 arg4 arg5
::group::arg1 
arg1 , step 0
arg1 , step 1
arg1 , step 2
::endgroup::
Traceback (most recent call last):
  File "./random_duration_script.py", line 10, in <module>
    raise ValueError(f"{sys.argv[1]} raised an exception")
ValueError: arg1  raised an exception
::group::arg2 
arg2 , step 0
arg2 , step 1
arg2 , step 2
::endgroup::
Traceback (most recent call last):
  File "./random_duration_script.py", line 10, in <module>
    raise ValueError(f"{sys.argv[1]} raised an exception")
ValueError: arg2  raised an exception
::group::arg4 
arg4 , step 0
arg4 , step 1
arg4 , step 2
arg4 , step 3
arg4 , step 4
arg4 , step 5
::endgroup::
::group::arg3 
arg3 , step 0
arg3 , step 1
arg3 , step 2
arg3 , step 3
arg3 , step 4
arg3 , step 5
::endgroup::
::group::arg5 
arg5 , step 0
arg5 , step 1
arg5 , step 2
arg5 , step 3
arg5 , step 4
::endgroup::

$ echo $?
0

Note how stderr is output after stdout but remains ordered as we expect.
However, we see that the error code is lost in this case due to the echo immediately following the exception raised by random_duration_script.py.

Removing the commands wrapping the call to random_duration_script.py confirms that the error code is otherwise preserved even when processes complete successfully following a failing process:

$ parallel './random_duration_script.py {}' ::: arg1 arg2 arg3 arg4 arg5 2&> /dev/null; echo $?
2

(note that I pipe stdout and stderr to /dev/null here as it is unimportant in this exploration of error code handling of 'parallel')

We can retain our error code AND still echo our 'endgroup' with a minor change:

$ parallel 'echo ::group::{}; ./random_duration_script.py {} && echo ::endgroup:: || (echo ::endgroup:: && exit 1) ' ::: arg1 arg2 arg3 arg4 arg5 2&> /dev/null; echo $?
2