MOD File Builder often has trouble connecting to the PSSE API if the environment on your machine isn't exactly like PSSE/MOD expects.
This document show some common issues with MOD File Builder install and how to fix them.
If you can't get PSSE to open without MOD, you need to debug that first. Here are some PSSE Debug Notes