mpirun2.log

host: master-node00
host: slave-node00
host: slave-node01

==================================================================================================
mpiexec options:
----------------
  Base path: /usr/lib64/mpi/gcc/mpich/bin/
  Launcher: (null)
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.xz=00;31:*.avi=01;35:*.bmp=01;35:*.dl=01;35:*.fli=01;35:*.gif=01;35:*.gl=01;35:*.jpg=01;35:*.jpeg=01;35:*.mkv=01;35:*.mng=01;35:*.mov=01;35:*.mp4=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.svg=01;35:*.tga=01;35:*.tif=01;35:*.webm=01;35:*.webp=01;35:*.wmv=01;35:*.xbm=01;35:*.xcf=01;35:*.xpm=01;35:*.aiff=00;32:*.ape=00;32:*.au=00;32:*.flac=00;32:*.m4a=00;32:*.mid=00;32:*.mp3=00;32:*.mpc=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:*.wma=00;32:*.wv=00;32:
    HOSTTYPE=x86_64
    LESSCLOSE=lessclose.sh %s %s
    XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB
    LANG=POSIX
    WINDOWMANAGER=xterm
    LESS=-M -I -R
    SYSTEMD_EXEC_PID=1167
    HOSTNAME=master-node00
    INVOCATION_ID=ba01c2f9ddf24d7086835d320cfe21c5
    CSHEDIT=emacs
    GPG_TTY=/dev/tty1
    LESS_ADVANCED_PREPROCESSOR=no
    COLORTERM=1
    MACHTYPE=x86_64-suse-linux
    XDG_VTNR=1
    MINICOM=-c on
    OSTYPE=linux
    XDG_SESSION_ID=4
    USER=root
    PAGER=less
    MORE=-sl
    PWD=/root
    HOME=/root
    LC_CTYPE=en_US.UTF-8
    JOURNAL_STREAM=8:16836
    HOST=master-node00
    XNLSPATH=/usr/X11R6/lib/X11/nls
    XDG_SESSION_TYPE=tty
    XDG_DATA_DIRS=/usr/share
    PROFILEREAD=true
    FROM_HEADER=
    MAIL=/var/mail/root
    LESSKEY=/etc/lesskey.bin
    SHELL=/bin/bash
    TERM=linux
    XDG_SESSION_CLASS=user
    LS_OPTIONS=-A -N --color=tty -T 0
    PYTHONSTARTUP=/etc/pythonstart
    SHLVL=1
    XDG_SEAT=seat0
    MANPATH=/usr/share/man:/usr/local/man
    LOGNAME=root
    DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
    XDG_RUNTIME_DIR=/run/user/0
    XDG_CONFIG_DIRS=/etc/xdg
    PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin
    G_BROKEN_FILENAMES=1
    HISTSIZE=1000
    CPU=x86_64
    LESSOPEN=lessopen.sh %s
    _=/usr/lib64/mpi/gcc/mpich/bin/mpirun

  Hydra internal environment:
  ---------------------------
    GFORTRAN_UNBUFFERED_PRECONNECTED=y


    Proxy information:
    *********************
      [1] proxy: master-node00 (1 cores)
      Exec list: /tmp/simple_mpi (1 processes); 

      [2] proxy: slave-node00 (1 cores)
      Exec list: /tmp/simple_mpi (1 processes); 

      [3] proxy: slave-node01 (1 cores)
      Exec list: /tmp/simple_mpi (1 processes); 


==================================================================================================

[mpiexec@master-node00] Timeout set to -1 (-1 means infinite)
[mpiexec@master-node00] Got a control port string of master-node00:44143

Proxy launch args: /usr/lib64/mpi/gcc/mpich/bin/hydra_pmi_proxy --control-port master-node00:44143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 

Arguments being passed to proxy 0:
--version 3.4.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname master-node00 --global-core-map 0,1,3 --pmi-id-map 0,0 --global-process-count 3 --auto-cleanup 1 --pmi-kvsname kvs_3290_0_1387041771_master-node00 --pmi-process-mapping (vector,(0,3,1)) --global-inherited-env 52 'LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.xz=00;31:*.avi=01;35:*.bmp=01;35:*.dl=01;35:*.fli=01;35:*.gif=01;35:*.gl=01;35:*.jpg=01;35:*.jpeg=01;35:*.mkv=01;35:*.mng=01;35:*.mov=01;35:*.mp4=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.svg=01;35:*.tga=01;35:*.tif=01;35:*.webm=01;35:*.webp=01;35:*.wmv=01;35:*.xbm=01;35:*.xcf=01;35:*.xpm=01;35:*.aiff=00;32:*.ape=00;32:*.au=00;32:*.flac=00;32:*.m4a=00;32:*.mid=00;32:*.mp3=00;32:*.mpc=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:*.wma=00;32:*.wv=00;32:' 'HOSTTYPE=x86_64' 'LESSCLOSE=lessclose.sh %s %s' 'XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB' 'LANG=POSIX' 'WINDOWMANAGER=xterm' 'LESS=-M -I -R' 'SYSTEMD_EXEC_PID=1167' 'HOSTNAME=master-node00' 'INVOCATION_ID=ba01c2f9ddf24d7086835d320cfe21c5' 'CSHEDIT=emacs' 'GPG_TTY=/dev/tty1' 'LESS_ADVANCED_PREPROCESSOR=no' 'COLORTERM=1' 'MACHTYPE=x86_64-suse-linux' 'XDG_VTNR=1' 'MINICOM=-c on' 'OSTYPE=linux' 'XDG_SESSION_ID=4' 'USER=root' 'PAGER=less' 'MORE=-sl' 'PWD=/root' 'HOME=/root' 'LC_CTYPE=en_US.UTF-8' 'JOURNAL_STREAM=8:16836' 'HOST=master-node00' 'XNLSPATH=/usr/X11R6/lib/X11/nls' 'XDG_SESSION_TYPE=tty' 'XDG_DATA_DIRS=/usr/share' 'PROFILEREAD=true' 'FROM_HEADER=' 'MAIL=/var/mail/root' 'LESSKEY=/etc/lesskey.bin' 'SHELL=/bin/bash' 'TERM=linux' 'XDG_SESSION_CLASS=user' 'LS_OPTIONS=-A -N --color=tty -T 0' 'PYTHONSTARTUP=/etc/pythonstart' 'SHLVL=1' 'XDG_SEAT=seat0' 'MANPATH=/usr/share/man:/usr/local/man' 'LOGNAME=root' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus' 'XDG_RUNTIME_DIR=/run/user/0' 'XDG_CONFIG_DIRS=/etc/xdg' 'PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin' 'G_BROKEN_FILENAMES=1' 'HISTSIZE=1000' 'CPU=x86_64' 'LESSOPEN=lessopen.sh %s' '_=/usr/lib64/mpi/gcc/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /root --exec-args 1 /tmp/simple_mpi 

Arguments being passed to proxy 1:
--version 3.4.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname slave-node00 --global-core-map 0,1,3 --pmi-id-map 0,1 --global-process-count 3 --auto-cleanup 1 --pmi-kvsname kvs_3290_0_1387041771_master-node00 --pmi-process-mapping (vector,(0,3,1)) --global-inherited-env 52 'LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.xz=00;31:*.avi=01;35:*.bmp=01;35:*.dl=01;35:*.fli=01;35:*.gif=01;35:*.gl=01;35:*.jpg=01;35:*.jpeg=01;35:*.mkv=01;35:*.mng=01;35:*.mov=01;35:*.mp4=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.svg=01;35:*.tga=01;35:*.tif=01;35:*.webm=01;35:*.webp=01;35:*.wmv=01;35:*.xbm=01;35:*.xcf=01;35:*.xpm=01;35:*.aiff=00;32:*.ape=00;32:*.au=00;32:*.flac=00;32:*.m4a=00;32:*.mid=00;32:*.mp3=00;32:*.mpc=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:*.wma=00;32:*.wv=00;32:' 'HOSTTYPE=x86_64' 'LESSCLOSE=lessclose.sh %s %s' 'XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB' 'LANG=POSIX' 'WINDOWMANAGER=xterm' 'LESS=-M -I -R' 'SYSTEMD_EXEC_PID=1167' 'HOSTNAME=master-node00' 'INVOCATION_ID=ba01c2f9ddf24d7086835d320cfe21c5' 'CSHEDIT=emacs' 'GPG_TTY=/dev/tty1' 'LESS_ADVANCED_PREPROCESSOR=no' 'COLORTERM=1' 'MACHTYPE=x86_64-suse-linux' 'XDG_VTNR=1' 'MINICOM=-c on' 'OSTYPE=linux' 'XDG_SESSION_ID=4' 'USER=root' 'PAGER=less' 'MORE=-sl' 'PWD=/root' 'HOME=/root' 'LC_CTYPE=en_US.UTF-8' 'JOURNAL_STREAM=8:16836' 'HOST=master-node00' 'XNLSPATH=/usr/X11R6/lib/X11/nls' 'XDG_SESSION_TYPE=tty' 'XDG_DATA_DIRS=/usr/share' 'PROFILEREAD=true' 'FROM_HEADER=' 'MAIL=/var/mail/root' 'LESSKEY=/etc/lesskey.bin' 'SHELL=/bin/bash' 'TERM=linux' 'XDG_SESSION_CLASS=user' 'LS_OPTIONS=-A -N --color=tty -T 0' 'PYTHONSTARTUP=/etc/pythonstart' 'SHLVL=1' 'XDG_SEAT=seat0' 'MANPATH=/usr/share/man:/usr/local/man' 'LOGNAME=root' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus' 'XDG_RUNTIME_DIR=/run/user/0' 'XDG_CONFIG_DIRS=/etc/xdg' 'PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin' 'G_BROKEN_FILENAMES=1' 'HISTSIZE=1000' 'CPU=x86_64' 'LESSOPEN=lessopen.sh %s' '_=/usr/lib64/mpi/gcc/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /root --exec-args 1 /tmp/simple_mpi 

Arguments being passed to proxy 2:
--version 3.4.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname slave-node01 --global-core-map 0,1,3 --pmi-id-map 0,2 --global-process-count 3 --auto-cleanup 1 --pmi-kvsname kvs_3290_0_1387041771_master-node00 --pmi-process-mapping (vector,(0,3,1)) --global-inherited-env 52 'LS_COLORS=no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.xz=00;31:*.avi=01;35:*.bmp=01;35:*.dl=01;35:*.fli=01;35:*.gif=01;35:*.gl=01;35:*.jpg=01;35:*.jpeg=01;35:*.mkv=01;35:*.mng=01;35:*.mov=01;35:*.mp4=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.svg=01;35:*.tga=01;35:*.tif=01;35:*.webm=01;35:*.webp=01;35:*.wmv=01;35:*.xbm=01;35:*.xcf=01;35:*.xpm=01;35:*.aiff=00;32:*.ape=00;32:*.au=00;32:*.flac=00;32:*.m4a=00;32:*.mid=00;32:*.mp3=00;32:*.mpc=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:*.wma=00;32:*.wv=00;32:' 'HOSTTYPE=x86_64' 'LESSCLOSE=lessclose.sh %s %s' 'XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB' 'LANG=POSIX' 'WINDOWMANAGER=xterm' 'LESS=-M -I -R' 'SYSTEMD_EXEC_PID=1167' 'HOSTNAME=master-node00' 'INVOCATION_ID=ba01c2f9ddf24d7086835d320cfe21c5' 'CSHEDIT=emacs' 'GPG_TTY=/dev/tty1' 'LESS_ADVANCED_PREPROCESSOR=no' 'COLORTERM=1' 'MACHTYPE=x86_64-suse-linux' 'XDG_VTNR=1' 'MINICOM=-c on' 'OSTYPE=linux' 'XDG_SESSION_ID=4' 'USER=root' 'PAGER=less' 'MORE=-sl' 'PWD=/root' 'HOME=/root' 'LC_CTYPE=en_US.UTF-8' 'JOURNAL_STREAM=8:16836' 'HOST=master-node00' 'XNLSPATH=/usr/X11R6/lib/X11/nls' 'XDG_SESSION_TYPE=tty' 'XDG_DATA_DIRS=/usr/share' 'PROFILEREAD=true' 'FROM_HEADER=' 'MAIL=/var/mail/root' 'LESSKEY=/etc/lesskey.bin' 'SHELL=/bin/bash' 'TERM=linux' 'XDG_SESSION_CLASS=user' 'LS_OPTIONS=-A -N --color=tty -T 0' 'PYTHONSTARTUP=/etc/pythonstart' 'SHLVL=1' 'XDG_SEAT=seat0' 'MANPATH=/usr/share/man:/usr/local/man' 'LOGNAME=root' 'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus' 'XDG_RUNTIME_DIR=/run/user/0' 'XDG_CONFIG_DIRS=/etc/xdg' 'PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin' 'G_BROKEN_FILENAMES=1' 'HISTSIZE=1000' 'CPU=x86_64' 'LESSOPEN=lessopen.sh %s' '_=/usr/lib64/mpi/gcc/mpich/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /root --exec-args 1 /tmp/simple_mpi 

[mpiexec@master-node00] Launch arguments: /usr/lib64/mpi/gcc/mpich/bin/hydra_pmi_proxy --control-port master-node00:44143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 0 
[mpiexec@master-node00] Launch arguments: /usr/bin/ssh -x slave-node00 "/usr/lib64/mpi/gcc/mpich/bin/hydra_pmi_proxy" --control-port master-node00:44143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 1 
[mpiexec@master-node00] Launch arguments: /usr/bin/ssh -x slave-node01 "/usr/lib64/mpi/gcc/mpich/bin/hydra_pmi_proxy" --control-port master-node00:44143 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --pmi-port 0 --gpus-per-proc -2 --proxy-id 2 
[proxy:0:0@master-node00] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1 
[proxy:0:0@master-node00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:0@master-node00] got pmi command (from 0): get_maxes

[proxy:0:0@master-node00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:0@master-node00] got pmi command (from 0): get_appnum

[proxy:0:0@master-node00] PMI response: cmd=appnum appnum=0
[proxy:0:0@master-node00] got pmi command (from 0): get_my_kvsname

[proxy:0:0@master-node00] PMI response: cmd=my_kvsname kvsname=kvs_3290_0_1387041771_master-node00
[proxy:0:0@master-node00] got pmi command (from 0): get
kvsname=kvs_3290_0_1387041771_master-node00 key=PMI_process_mapping 
[proxy:0:0@master-node00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:0@master-node00] got pmi command (from 0): put
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-0 value=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000 
[proxy:0:0@master-node00] cached command: -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000
[proxy:0:0@master-node00] PMI response: cmd=put_result rc=0 msg=success
[proxy:0:0@master-node00] got pmi command (from 0): barrier_in

[proxy:0:0@master-node00] flushing 1 put command(s) out
[proxy:0:0@master-node00] forwarding command (cmd=put -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000) upstream
[proxy:0:0@master-node00] forwarding command (cmd=barrier_in) upstream
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=put -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:2@slave-node01] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1 
[proxy:0:2@slave-node01] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:2@slave-node01] got pmi command (from 4): get_maxes

[proxy:0:2@slave-node01] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:2@slave-node01] got pmi command (from 4): get_appnum

[proxy:0:2@slave-node01] PMI response: cmd=appnum appnum=0
[proxy:0:2@slave-node01] got pmi command (from 4): get_my_kvsname

[proxy:0:2@slave-node01] PMI response: cmd=my_kvsname kvsname=kvs_3290_0_1387041771_master-node00
[proxy:0:2@slave-node01] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=PMI_process_mapping 
[proxy:0:2@slave-node01] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:2@slave-node01] got pmi command (from 4): put
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-2 value=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000 
[proxy:0:2@slave-node01] cached command: -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000
[proxy:0:2@slave-node01] PMI response: cmd=put_result rc=0 msg=success
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=put -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:2@slave-node01] got pmi command (from 4): barrier_in

[proxy:0:2@slave-node01] flushing 1 put command(s) out
[proxy:0:2@slave-node01] forwarding command (cmd=put -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000) upstream
[proxy:0:2@slave-node01] forwarding command (cmd=barrier_in) upstream
[proxy:0:1@slave-node00] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1 
[proxy:0:1@slave-node00] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0
[proxy:0:1@slave-node00] got pmi command (from 4): get_maxes

[proxy:0:1@slave-node00] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024
[proxy:0:1@slave-node00] got pmi command (from 4): get_appnum

[proxy:0:1@slave-node00] PMI response: cmd=appnum appnum=0
[proxy:0:1@slave-node00] got pmi command (from 4): get_my_kvsname

[proxy:0:1@slave-node00] PMI response: cmd=my_kvsname kvsname=kvs_3290_0_1387041771_master-node00
[proxy:0:1@slave-node00] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=PMI_process_mapping 
[proxy:0:1@slave-node00] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,3,1))
[proxy:0:1@slave-node00] got pmi command (from 4): put
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-1 value=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000 
[proxy:0:1@slave-node00] cached command: -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000
[proxy:0:1@slave-node00] PMI response: cmd=put_result rc=0 msg=success
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=put -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000
[proxy:0:1@slave-node00] got pmi command (from 4): barrier_in

[proxy:0:1@slave-node00] flushing 1 put command(s) out
[proxy:0:1@slave-node00] forwarding command (cmd=put -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000) upstream
[proxy:0:1@slave-node00] forwarding command (cmd=barrier_in) upstream
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@master-node00] PMI response to fd 6 pid 4: cmd=keyval_cache -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000 -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000 -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000 
[mpiexec@master-node00] PMI response to fd 15 pid 4: cmd=keyval_cache -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000 -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000 -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000 
[mpiexec@master-node00] PMI response to fd 7 pid 4: cmd=keyval_cache -allgather-shm-1-0=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000 -allgather-shm-1-2=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000 -allgather-shm-1-1=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000 
[mpiexec@master-node00] PMI response to fd 6 pid 4: cmd=barrier_out
[mpiexec@master-node00] PMI response to fd 15 pid 4: cmd=barrier_out
[mpiexec@master-node00] PMI response to fd 7 pid 4: cmd=barrier_out
[proxy:0:0@master-node00] PMI response: cmd=barrier_out
[proxy:0:0@master-node00] got pmi command (from 0): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-0 
[proxy:0:0@master-node00] PMI response: cmd=get_result rc=0 msg=success value=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000
[proxy:0:0@master-node00] got pmi command (from 0): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-1 
[proxy:0:0@master-node00] PMI response: cmd=get_result rc=0 msg=success value=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000
[proxy:0:1@slave-node00] PMI response: cmd=barrier_out
[proxy:0:0@master-node00] got pmi command (from 0): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-2 
[proxy:0:0@master-node00] PMI response: cmd=get_result rc=0 msg=success value=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000
[proxy:0:1@slave-node00] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-0 
[proxy:0:1@slave-node00] PMI response: cmd=get_result rc=0 msg=success value=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000
[proxy:0:1@slave-node00] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-1 
[proxy:0:1@slave-node00] PMI response: cmd=get_result rc=0 msg=success value=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000
[proxy:0:1@slave-node00] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-2 
[proxy:0:1@slave-node00] PMI response: cmd=get_result rc=0 msg=success value=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000
[proxy:0:2@slave-node01] PMI response: cmd=barrier_out
[proxy:0:2@slave-node01] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-0 
[proxy:0:2@slave-node01] PMI response: cmd=get_result rc=0 msg=success value=0071CE83B96F5E925A4008EDBC2E4287103C61A7D377CC2B32004C3E5077CCAB33004F230088DF0C0080020000C04108EDBC2E4287103C618D5377CC2B32004C3E5077CCAB33004F23008818800000000000002200637577CC2B3200F8D74F00000000004F030088F9F84C12CA2D76F323120102EDBC2E4287103CE1980000F000000000CF1917B75138539E3ECBD7E037370113230082B153230600020A000211CF1917B7513846CA34CBD7E037370013230082ECDB2488EDBC2E4287103C61478B95BFD63400242E5077CCAB330092210084DF0C0000
[proxy:0:2@slave-node01] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-1 
[proxy:0:2@slave-node01] PMI response: cmd=get_result rc=0 msg=success value=00ED565AB35BFF4AF94008C42E82AA45471E48A7D377CC2B32004C3E5077CCAB33004F230088930A0000010000C04108C42E82AA45471E488D5377CC2B32004C3E5077CCAB33004F2300882E000000000000002200637577CC2B3200F8D74F00000000004F0300882D7C776EB487ADE9230600020A00020FCF1917B7513846CA34CBD7E0373700132300828C8923120102C42E82AA45471EC8980000F000000000CF1917B75138539E3ECBD7E037370113230082A8AB2488C42E82AA45471E48478B95BFD63400242E5077CCAB330092210084930A0000
[proxy:0:2@slave-node01] got pmi command (from 4): get
kvsname=kvs_3290_0_1387041771_master-node00 key=-allgather-shm-1-2 
[proxy:0:2@slave-node01] PMI response: cmd=get_result rc=0 msg=success value=008608362DECABCABE4008E1D4AFA60A20C87AA7D377CC2B32004C3E5077CCAB33004F230088010B0000010000C04108E1D4AFA60A20C87A8D5377CC2B32004C3E5077CCAB33004F2300883A000000000000002200637577CC2B3200F8D74F00000000004F03008836EB0CCCA103C6A823120102E1D4AFA60A20C8FA980000F000000000CF1917B75138539E3ECBD7E03737011323008285D1230600020A000210CF1917B7513846CA34CBD7E037370013230082CA952488E1D4AFA60A20C87A478B95BFD63400242E5077CCAB330092210084010B0000
Hello, World.  I am 0 of 3
Hello, World.  I am 2 of 3
Hello, World.  I am 1 of 3
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1@slave-node00] got pmi command (from 4): barrier_in

[proxy:0:1@slave-node00] forwarding command (cmd=barrier_in) upstream
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:2@slave-node01] got pmi command (from 4): barrier_in

[proxy:0:2@slave-node01] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@master-node00] got pmi command (from 0): barrier_in

[proxy:0:0@master-node00] forwarding command (cmd=barrier_in) upstream
[mpiexec@master-node00] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@master-node00] PMI response to fd 6 pid 0: cmd=barrier_out
[mpiexec@master-node00] PMI response to fd 15 pid 0: cmd=barrier_out
[mpiexec@master-node00] PMI response to fd 7 pid 0: cmd=barrier_out
[proxy:0:0@master-node00] PMI response: cmd=barrier_out
[proxy:0:1@slave-node00] PMI response: cmd=barrier_out
[proxy:0:2@slave-node01] PMI response: cmd=barrier_out
[proxy:0:1@slave-node00] got pmi command (from 4): finalize

[proxy:0:1@slave-node00] PMI response: cmd=finalize_ack
[proxy:0:2@slave-node01] got pmi command (from 4): finalize

[proxy:0:2@slave-node01] PMI response: cmd=finalize_ack
[proxy:0:0@master-node00] got pmi command (from 0): finalize

[proxy:0:0@master-node00] PMI response: cmd=finalize_ack