(this is worth looking at http://www.stefan-seelmann.de/wiki/rasperrypi-homeserver, which is basically this:
apt-get purge --auto-remove scratch debian-reference-en dillo idle3 python3-tk idle python-pygame python-tk lightdm gnome-themes-standard gnome-icon-theme raspberrypi-artwork gvfs-backends gvfs-fuse desktop-base lxpolkit netsurf-gtk zenity xdg-utils mupdf gtk2-engines alsa-utils lxde lxtask menu-xdg gksu midori xserver-xorg xinit xserver-xorg-video-fbdev libraspberrypi-dev libraspberrypi-doc dbus-x11 libx11-6 libx11-data libx11-xcb1 x11-common x11-utils lxde-icon-theme gconf-service gconf2-common
)
Warning: this will probably break something. If you need sound, don't remove the "alsa" packages
sudo apt-get --yes purge xserver-common x11-xfs-utils x11-xserver-utils xinit libsmbclient blt gvfs gvfs-backends gvfs-daemons gvfs-fuse idle idle-python2.7 idle-python3.2 idle3 libaudio2 libice6 liblightdm-gobject-1-0 libobrender27 libpulse0 libqt4-svg libqt