To sucessfully render mujoco on headless machines make sure the following dependencies are installed via apt-get:
sudo apt-get --assume-yes install libxrender1 libgl1-mesa-dev libgl1-mesa-glx libglew-dev libosmesa6-dev xpra patchelf libglfw3-dev libglfw3 libglew2.0
You may perform additional steps to build mujoco-py with gpu rendering support on a remote server without any monitor attached.
First, check whether mujoco_py has already been built with headless gpu rendering
$ python3
import mujoco_py
print('gpu' in str(mujoco_py.cymj).split('/')[-1])
If the output is False, you may perform the steps below to build gpu extension
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia-000
$ sudo mkdir -p /usr/lib/nvidia-000
$ python3
import mujoco_py
print('gpu' in str(mujoco_py.cymj).split('/')[-1])
# True
https://github.com/openai/mujoco-py/blob/804443ebc475a0ab053249164a51932baada06eb/README.md