Take a simple 5-page site that just covers who you are, what you do, some example projects and contact info or something. I bookmark one of those pages so I can read it later when I have more time. I get home, open the bookmark and the page loads with only the relevant HTML, CSS and JS. Done.
Say that site is converted into a single-page, super-scroll site and the content equivalent to the page I bookmarked happens to have been placed at the bottom of this page. Now, when I open the bookmark (assuming there is a 301 set up correctly) I have to wait for the content from all the other pages to load as well before the page is ready. While this is happening I might start reading my content only to have the page grow as images load in, pushing my content down and making me lose my place. There might be a larger amount of CSS and JS as well because previously page-specific code will now need to be loaded every time, regardless of if it is needed for the content the user wants, which could hurt functionality.
It is also widely accepted that users have very short attention spans and will happily give up on a site if the content they want is not presented to them in a matter of seconds. They tend to be put off when given large amounts of content to sift through when they're only after a couple of specific things. How does a single-page site appeal to the 'average' user? They open a site only to be presented with 5 pages of content all at once and a scroll bar with a dauntingly small thumb.
the 'thumb' of a scroll bar:
Not all single-page, scroll to your content pages are like this. There's actually some pretty good implementations that overcome many of the issues you mention. I'll have one to show you in a few days if you're interested. :)