Because Varnish doesn't support SSL, most people choose a setup where Nginx SSL will forward all traffic to Varnish and Varnish will forward will forward the traffic it cannot handle back to nginx. Or worse, bind Varnish on port 80 and direct all traffic into Varnish. This will already degrade performance, because Varnish will purge more because static files are also taking up room in the cache.
Next up, the Nginx configuration of Magento will handle static files.