In order to make text boot the default under systemd (regardless of which distro, really):
systemctl set-default multi-user.target
To change back to booting to the GUI,
systemctl set-default graphical.target
I confirmed those work on my Jessie VM and Slashback confirmed it on Stretch, too.
PS: You don't actually need the X server on your machine to run X clients over ssh. The X server is only needed where the display (monitor) is.