Running commands as child processes on new tmux panes

trun.bash

#!/bin/bash
set -euo pipefail
export LC_ALL=C
export LANG=C

if !(type reptyr > /dev/null 2>&1); then
  echo "Please install 'reptyr' command." >&2
  exit 1
fi

if [[ -z ${TMUX+x} ]]; then
  echo "Please run this command within a tmux session." >&2
  exit 1
fi

if [[ $# < 1 ]]; then
  echo "Usage: $0 COMMANDS..." >&2
  echo >&2
  echo "Example: $0 bash" >&2
  echo "         $0 bash -c \"echo hoge; sleep 3\"" >&2
  echo "         $0 gdb <file>" >&2
  exit 1
fi

# Put quotes correctly in the command
CMDS=()
for i in "$@"; do
  if [[ $i =~ [[:space:]] ]]; then
    i=\"$i\"
  fi
  CMDS+=("$i")
done
COMMANDS="(echo; "${CMDS[@]}")"

# Compile a program to call 'ioctl' syscall
SET_PTY_CMD=$(mktemp)
gcc -o $SET_PTY_CMD -x c - <<EOF
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/ioctl.h>

int main(int argc, char **argv) {
  if (argc < 3) {
    fprintf(stderr, "Usage: %s /dev/pts/XX COMMANDS\n", argv[0]);
    exit(1);
  }

  int fd = open(argv[1], O_RDWR);
  if (fd == -1) {
    perror("open");
    exit(1);
  }

  if (ioctl(fd, TIOCSCTTY, 0) != 0) {
    perror("ioctl");
    exit(1);
  }

  dup2(fd, STDIN_FILENO);
  dup2(fd, STDOUT_FILENO);
  dup2(fd, STDERR_FILENO);

  argv += 2;
  execvp(argv[0], argv);
  return 0;
}
EOF

# temp file for forwarding /dev/pts/xxx of 'reptyr' to host
TMPFILE=$(mktemp)
touch $TMPFILE
trap "rm -f $SET_PTY_CMD $TMPFILE" EXIT

# split the tmux pane and run 'reptyr -l' on it
tmux split-window bash -c "trap \"rm -f $TMPFILE\" EXIT; reptyr -l | tee $TMPFILE"

# '--follow=name' rather than '-f' to exit when the file is deleted
tail --follow=name $TMPFILE |
  head -1 |
  stdbuf -oL grep 'Opened a new pty:' |
  stdbuf -oL cut -d ' ' -f 5 |
  xargs -I DEVPTS setsid $SET_PTY_CMD DEVPTS bash -c "$COMMANDS"

Purpose

Mainly for debugging MPI programs.

tmpi is a great tool to debug MPI programs, but environment variables are not correctly inherited because processes running on new tmux panes are not children of the calling process.
In addition, it now supports only OpenMPI.

We can easily resolve these issues by replacing the controlling terminal using reptyr command.
This script (trun) allows for running commands as child processes on new tmux panes.
trun is independent of any MPI implementations.

Preliminaries

Install reptyr command by sudo apt install reptyr or something like that.

How to Use

Suppose that you installed this script as trun.

Running within a tmux session

trun bash

will split the current tmux pane and launch bash on a new tmux pane as a child process.

You can easily debug MPI programs by running

mpirun trun gdb <program>

Advanced Usage

A wrapper script of trun can be used for convenience.

For example, if you only need to focus on specific MPI ranks, you can write the following script to focus on, e.g., MPI rank 0.

#!/bin/bash

if [[ $PMI_RANK == 0 ]]; then
  trun gdb --args "$@"
else
  "$@"
fi

This script (mpidebug) assumes MPICH (PMI_RANK environment variable).

Running

mpirun mpidebug <commands...>

will launch a gdb session only for the process with MPI rank 0 on a new tmux pane.

Extending the script allows for more flexible control over which MPI process to be monitored by gdb (e.g., only rank 0 and 4).

References

• tmpi
• reptyr: Changing a process's controlling terminal
  • Great explanation on how setsid and ioctl syscalls should be used