README.rst

git push/fetch over stdio

Git uses send-pack and fetch-pack for transferring data over the smart protocol,. These plumbing commands are a bit too smart for their own good, in the sense that they always try to run their peer commands (receive-pack and upload-pack respectively), either on the local host or over ssh to the remote host (for file:// and ssh:// remotes, respectively). In some cases it may not be feasible for git to connect to the remote host and/or run the peer commands, or it may just be preferrable that such processes are not spawned by git itself. Even then it should still be possible to connect the two ends, e.g. over stdio.

The git commands pipe-send-pack and pipe-fetch-pack are wrappers for the respective git commands, which do just that. These commands, along with gfp2gur.awk (which is used by pipe-fetch-pack to update refs fetched from the source), must be installed on the systems where the target repositories live (anywhere in the PATH for a non-interactive command). File descriptor redirection currently requires socat(1), which must also be installed on these systems. You will also need awk and mktemp for pipe-fetch-pack.

git_stdio_client.sh provides (bash) shell functions that setup ssh connections to the sending and receiving hosts and connect the two through a bidirectional pipe. The versions ending in 2 depend on the git commands above; the other versions call socat directly, so the git commands are not necessary, however these implementations are messier and more error prone (but otherwise mostly equivalent in functionality).

Having loaded the functions in your shell, you can then do:

git-pipe-push sending-host:/path/to/source/git/repo receiving-host:/path/to/destination/git/repo

Or for the equivalent of a git fetch:

git-pipe-fetch receiving-host:/path/to/destination/git/repo sending-host:/path/to/source/git/repo

git fetch operations update refs for a (potentially) virtual remote, whose name can be controlled through the --remote option and defaults to the sending-host. If no refs are specified, they default to --all. On the other hand git push operations typically update refs common to the source and the target repo.

Known issues

receive-pack may close the stream too soon, which results in send-pack (the receiving end) complaining like this: fatal: The remote end hung up unexpectedly The socat wrapper may also complain for a broken pipe. As far as I understand, this is harmless. You may verify refs and objects on the receiving end, or you may also repeat the process after setting GIT_TRACE=1 and/or GIT_TRACE_PACKET=1 on the sending-host, in order to check if the transfer completes correctly.

gfp2gur.awk

#!/usr/bin/awk -f
BEGIN {
    if (remote "x" == "x")
	exit
    cmd = gitprefix "update-ref"
}
$1 ~ /^[0-9a-f]+$/ && length($1) == 40 {
    if ($2 == "HEAD")
	sref = "refs/remotes/" remote "/" $2
    else {
	sref = $2
	sub(/heads/, "remotes/" remote, sref)
    }
    print cmd, sref, $1
}

git-pipe-fetch-pack

#!/bin/sh

USAGE='[--write-refs file] [git-fetch-pack options]'
OPTIONS_SPEC=
SUBDIRECTORY_OK=Yes

save () {
    what="$1"
    shift
    for i; do
	if test "$what" = opts && printf %s\\n "$i" | grep -q -- "^--"; then
	    :
	elif test "$what" = args && printf %s\\n "$i" | grep -qv -- "^--"; then
	    :
	elif test "$what" = all; then
	    :
	else
	    continue
	fi
	# escape : for socat
	i=$(printf %s\\n "$i" | sed "s/:/\\\:/g")
	printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
	echo " "
    done
    unset what i
}

. "$(git --exec-path)/git-sh-setup"

write_refs=
i=1
while test ! $i -gt $# ; do
    eval "ival=\$$i"
    eval "jval=\$$((i+1))"
    case "$ival" in
	--write-refs=*)
	    write_refs=$(printf %s\\n "$ival" | sed 's/--write-refs=//')
	    shift	    
	    ;;
	--write-refs)
	    if test -n "$jval" && printf %s\\n "$jval" | grep -qv -- "^-"; then
		write_refs="$jval"
		shift 2
	    else
		usage
	    fi
	    ;;
    esac
    i=$((i+1))
done
unset i ival jval

opts=$(save opts "$@")
args=$(save args "$@")
# fetch-pack apparently needs some refs
if test -z "$args" && printf %s\\n "$opts" | grep -qv -- "--all"; then
    eval "set -- $opts --all . $args"
else
    eval "set -- $opts . $args"
fi
unset opts args

socat=$(which socat)
if test -z "$socat" || test ! -x "$socat"; then
    die "This command requires socat(1)"
fi

gfpcmd="git fetch-pack --upload-pack=\\\"$socat - 5 #\\\" $@"
if test -n "$write_refs"; then
    if touch "$write_refs" && test -w "$write_refs"; then
	:
    else
	die "$write_refs not writeable"
    fi
    gfpcmd="$gfpcmd \>$write_refs"
fi

$socat STDIO SYSTEM:"$gfpcmd",fdin=5

git-pipe-send-pack

#!/bin/sh

USAGE='[git-send-pack options]'
OPTIONS_SPEC=
SUBDIRECTORY_OK=Yes

save () {
    what="$1"
    shift
    for i; do
	if test "$what" = opts && printf %s\\n "$i" | grep -q -- "^--"; then
	    :
	elif test "$what" = args && printf %s\\n "$i" | grep -qv -- "^--"; then
	    :
	elif test "$what" = all; then
	    :
	else
	    continue
	fi
	# escape : for socat
	i=$(printf %s\\n "$i" | sed "s/:/\\\:/g")
	printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/"
	echo " "
    done
    unset what i
}

. "$(git --exec-path)/git-sh-setup"

opts=$(save opts "$@")
args=$(save args "$@")
eval "set -- $opts . $args"
unset opts args

socat=$(which socat)
if test -z "$socat" || test ! -x "$socat"; then
    die "This command requires socat(1)"
fi

gspcmd="git send-pack --receive-pack=\\\"$socat - 5 #\\\" $@"

$socat STDIO SYSTEM:"$gspcmd",fdin=5

git_stdio_client.sh

#!/bin/bash

function git-pipe-push () {
    local shost spath dhost dpath opts=() args=() justafifo rc
    if [ $# -ge 2 ]; then
	eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
  if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
    printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
  }
}' "${@:1:2}")
    fi
    if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
	echo "Usage: ${FUNCNAME[0]} sending-host:[path] receiving-host:[path] [git-send-pack options]" >&2
	[ "$1" = "-h" ] && return 0 || return 1
    else
	shift 2
    fi
    while [ -n "$1" ]; do
	case "$1" in
	    --*)
		opts+=("$1")
		;;
	    *)
		args+=("$1")
	esac
	shift
    done
    opts+=(".")
    opts+=("${args[@]}")
    justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
    mkfifo "$justafifo"
    rc=$?
    if [ $rc -eq 0 ]; then
	trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
    else
	return $rc
    fi
    ssh $shost \
	"cd \"$spath\";"\
	"socat - EXEC:'git send-pack --receive-pack=\\\"socat - 5 #\\\" ${opts[@]//:/\:}',fdin=5"\
      <"$justafifo" | \
    ssh $dhost \
	"git receive-pack \"$dpath\"" \
      >"$justafifo"
    rc=$?
    rm -f "$justafifo"
    return $rc
}

function git-pipe-fetch () {
    local shost spath dhost dpath remote opts=() args=() justafifo rc
    if [ $# -ge 2 ]; then
	eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
  if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
    printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
  }
}' "${@:1:2}")
    fi
    if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
	echo "Usage: ${FUNCNAME[0]} receiving-host:[path] sending-host:[path] [--remote=remote_name] [git-fetch-pack options]" >&2
	[ "$1" = "-h" ] && return 0 || return 1
    else
	shift 2
    fi
    if test -n "$1" && echo "$1" | grep -q "^--remote="; then
	remote="${1#--remote=}"
	shift
    fi
    while [ -n "$1" ]; do
	case "$1" in
	    --*)
		opts+=("$1")
		;;
	    *)
		args+=("$1")
	esac
	shift
    done
    # fetch-pack apparently needs refs
    if test -z "${args[*]}" && echo "${opts[*]}" | grep -qv -- "--all"; then
	opts+=("--all")
    fi
    opts+=(".")
    opts+=("${args[@]}")
    justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
    mkfifo "$justafifo"
    rc=$?
    if [ $rc -eq 0 ]; then
	trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
    else
	return $rc
    fi
    ssh $shost \
	"cd \"$spath\";"\
	"export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
	"socat - SYSTEM:'git fetch-pack --upload-pack=\\\"socat - 5 #\\\" ${opts[@]//:/\:} >\$refstmp',fdin=5;"\
	"awk -v remote=\"${remote:-$dhost}\""\
	"'\$1 ~ /^[0-9a-f]+$/ && length(\$1) == 40 {
		if (\$2 == \"HEAD\") { sref = \"refs/remotes/\" remote \"/\" \$2; }
		else { sref = \$2; sub(/heads/, \"remotes/\" remote, sref); };
		print \"update-ref\", sref, \$1;
	}' \${refstmp} | xargs -L 1 git;"\
	"rm \${refstmp}"\
      <"$justafifo" | \
    ssh $dhost \
	"git upload-pack \"$dpath\"" \
      >"$justafifo"
    rc=$?
    rm -f "$justafifo"
    return $rc
}

function git-pipe-push2 () {
    local shost spath dhost dpath opts=() sudo justafifo rc
    if [ $# -ge 2 ]; then
	eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
  if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
    printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
  }
}' "${@:1:2}")
    fi
    if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
	echo "Usage: ${FUNCNAME[0]} sending-host:[path] receiving-host:[path] [--sudo[=user]] [git-send-pack options]" >&2
	[ "$1" = "-h" ] && return 0 || return 1
    else
	shift 2
    fi
    while [ -n "$1" ]; do
	case "$1" in
	    --sudo*)
		if [ -z "${1#--sudo}" ]; then
		    sudo=root
		else
		    sudo="${1#--sudo=}"
		fi
		;;
	    *)
		opts+=("$1")
		;;
	esac
	shift
    done
    justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
    mkfifo "$justafifo"
    rc=$?
    if [ $rc -eq 0 ]; then
	trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
    else
	return $rc
    fi
    if [ -n "$sudo" ]; then
	ssh $shost \
	    "sudo -u \"$sudo\" -- sh -c '"\
	    "cd \"$spath\";"\
	    "git pipe-send-pack ${opts[@]};"\
	    "'"\
	  <"$justafifo" | \
	ssh $dhost \
	    "sudo -u \"$sudo\" -- sh -c '"\
	    "git receive-pack \"$dpath\"" \
	    "'"\
	  >"$justafifo"
    else
	ssh $shost \
	    "cd \"$spath\";"\
	    "git pipe-send-pack ${opts[@]};"\
	  <"$justafifo" | \
	ssh $dhost \
	    "git receive-pack \"$dpath\"" \
	  >"$justafifo"
    fi
    rc=$?
    rm -f "$justafifo"
    return $rc
}

function git-pipe-fetch2 () {
    local shost spath dhost dpath opts=() remote sudo justafifo rc
    if [ $# -ge 2 ]; then
	eval $(perl -e \
'my @d = qw(s d);
for (my $i=0; $i<=1; $i++) {
  if ($ARGV[$i] =~ /^(\[[0-9A-Fa-f:]+\]|[^:]+):(.*)$/) {
    printf "%1\$shost=\"%2\$s\" %1\$spath=\"%3\$s\" ", $d[$i], $1, $2 || ".";
  }
}' "${@:1:2}")
    fi
    if [ -z "$shost" -o -z "$dhost" -o "$1" = "-h" ]; then
	echo "Usage: ${FUNCNAME[0]} receiving-host:[path] sending-host:[path] [--remote=remote_name] [--sudo[=user]] [git-fetch-pack options]" >&2
	[ "$1" = "-h" ] && return 0 || return 1
    else
	shift 2
    fi
    while [ -n "$1" ]; do
	case "$1" in
	    --remote*)
		if [ -z "${1#--remote}" ]; then
		    echo "missing required argument to --remote" >&2
		    return 1
		else
		    remote="${1#--remote=}"
		fi
		;;
	    --sudo*)
		if [ -z "${1#--sudo}" ]; then
		    sudo=root
		else
		    sudo="${1#--sudo=}"
		fi
		;;
	    *)
		opts+=("$1")
		;;
	esac
	shift
    done
    justafifo=$(mktemp -u /tmp/gitpipe.XXXXXX)
    mkfifo "$justafifo"
    rc=$?
    if [ $rc -eq 0 ]; then
	trap 'rm -f "$justafifo"' HUP INT QUIT TERM KILL
    else
	return $rc
    fi
    if [ -n "$sudo" ]; then
	ssh $shost \
	    "sudo -u \"$sudo\" -- sh -c '"\
	    "cd \"$spath\";"\
	    "export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
	    "git pipe-fetch-pack --write-refs=\${refstmp} ${opts[@]};"\
	    "gfp2gur.awk -v remote=\"${remote:-$dhost}\" \${refstmp} | xargs -L 1 git;"\
	    "rm \${refstmp}"\
	    "'"\
	  <"$justafifo" | \
	ssh $dhost \
	    "sudo -u \"$sudo\" -- sh -c '"\
	    "git upload-pack \"$dpath\"" \
	    "'"\
	  >"$justafifo"
    else
	ssh $shost \
	    "cd \"$spath\";"\
	    "export refstmp=$(mktemp -u /tmp/gitfetchpack.XXXXXXXX);"\
	    "git pipe-fetch-pack --write-refs=\${refstmp} ${opts[@]};"\
	    "gfp2gur.awk -v remote=\"${remote:-$dhost}\" \${refstmp} | xargs -L 1 git;"\
	    "rm \${refstmp}"\
	  <"$justafifo" | \
	ssh $dhost \
	    "git upload-pack \"$dpath\"" \
	  >"$justafifo"
    fi
    rc=$?
    rm -f "$justafifo"
    return $rc
}

git-fetch-pack seems to spit refs in stdout, which we need in order to update-ref

FWIW: Passing -q to git-fetch-pack might help anyway (in other ways), but seems to only quieten the output of git unpack-objects... Also, what about the --no-progress flag?

The problem is not how to make fetch-pack quiet (I think that stuff goes to stderr) but how to get its' output (refs fetched) in the middle of all this redirection hell.

Fixed! What a mess :)