See https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html for the basics.
The trouble is running ssh-add -s /long/path/to/some/pkcs11_provider.so
is a pain in the ass.
Also if the Yubikey is removed, ssh-add -l
will still indicate the key is available, even though
it's not, until the provider is removed with ssh-add -e /path/to/pkcs11_provider.so
And if the Yubikey is subsequently re-inserted, the ssh agent won't re-prompt for the pin, and so authentication will fail until the provider is removed and re-added.
Fortunately it's possible to automate these actions based on the insertion/removal of the Yubikey.