- Make sure VM's OS type is set to Linux 64 bits.
- On the VM's Settings -> Display, make sure "Graphics Controller" is "VMSVGA". 32 MB of "Video Memory" should be enough.
- On Haiku, run "pkgman install vmware_addons", reboot.
- Change resolution as you wish from the Screen preflet.
You should now be able to change modes as you like.
To make sure VirtualBox starts in the resolution you want: