How to remove device that can't be removed from the Bluetooth window?
Sometimes you can't pair a new device (like a new game controller) because there's a conflict with an old device. When you try to remove the device it may keep reporting 'Failed to remove'. If you go into Device Manager > View > Show Hidden Devices, and then expand the Bluetooth section you may see grayed out duplicate entries for your device. Try removing those and see if the troublesome device disappears from the Bluetooth window.
How to set up an XBox controller
I got a USB to Bluetooth dongle, and for an older controller I was able to just plug it in, go to Bluetooth > Add Device > Everything else, put the controller into pairing mode and it would just show up and pair.