Got this tip from the master Myles Byrne (@quackingduck)
git-sh loads your bash config from ~/.bashrc not ~/.bash_profile
So one way to fix this (this is how my machine is set up) is to copy your ~/.bash_profile to ~/.bashrc and then change ~/.bash_profile to just have this line:
test -f ~/.bashrc && source ~/.bashrc