to ensure the output from a viewport is high quality, scale up its content:
quad viewport content
if your content is 800x400, apply a scale of 4 to it (resulting in 3200x1600) set the viewport to the same size the quad mesh doesnt need to change size - it will scale down the viewport texture. but because its much higher resolution, the quality will be much better