When you have access to your private key file, use it for quick login. When you are on the go, use your Yubikey OTP and your password.
Tutorial: https://developers.yubico.com/yubico-pam/YubiKey_and_SSH_via_PAM.html
- Install yubico pam module:
sudo apt-get install libpam-yubico