This is what got me working. Staring at C:\ProgramData\ssh\sshd_config I noticed this line was commented!!!
#PubkeyAuthentication yes
No idea how that happened; my box was pre-imaged. I uncommented that, plus ensured that these two lines in the config file were commented out:
#Match Group administrators
# AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
I then restarted with:
Restart-Service -Name sshd, ssh-agent -Force
And now it's working. OMG what a battle. Huge thanks to everyone who's posting here.