Get it from http://developer.apple.com. You will not be able to submit apps to any stores using this XCode version, so turn away if that is something you might want to do.
In XCode's Preferences > Downloads you can install command line tools.
[insert screenshot]
sudo xcode-select -switch /Applications/Xcode45-DP1.app/Contents/Developer
Visit http://xquartz.macosforge.org/trac/wiki and download and install 2.7.2+.
You will need to fix the symlink it makes:
ln -s /opt/X11 /usr/X11
brew update
brew upgrade
Try out everything one by one and when one doesn't work brew remove it and then reinstall it. As of this writing, I am using the same brew I already had installed (upgrading to Mountain Lion doesn't remove your brews and they should be able to continue working normally).