sign_and_send_pubkey: signing failed for ED25519-SK

TL;DR
Overview
Replicate
Solution

TL;DR

When attempting to SSH with a ed25519-sk resident key if you get the following error:

sign_and_send_pubkey: signing failed for ED25519-SK "" from agent: agent refused operation

It’s probably gnome-keyring-daemon breaking things; and you probably want to disable it.

Overview

Support for security keys (e.g. ed25519-sk and ecdsa-sk) was finally added to Gitlab 14.8 released Feb 22, 2022, something Github has supported since May 2021. Better late than never I guess…

After adding my public key I tried ssh -T [email protected], which failed with:

sign_and_send_pubkey: signing failed for ED25519-SK "" from agent: agent refused operation

I used this same Yubikey successfully with Github on another system. As a sanity check I tried authenticating to Github; given Gitlab’s support was only recently added. However, I encountered the same error message.

A quick search of the error message on Google only turned up "solutions" that involved fixing permissions on the SSH key files. While troubleshooting I happened to run ssh-add without any parameters, which errored with:

FIDO verify-required key .ssh/id_ed25519_sk is not currently supported by ssh-agent

Upgrading to the latest OpenSSH 8.9p1 resolved the ssh-add issue, but the connection error remained. Googling on the above error message led me to gnome-keyring issue #101: gnome-keyring deals poorly with 'verify-required' ssh "SK" keys.

My key was created with -O verify-required, which I always use as it forces the requirement of a PIN. This way if my Yubikey is ever lost or stolen someone can’t simply touch the device and use the SSH key.

[start] of the problem comes because gnome-keyring advertises any *.pub files it finds in ~/.ssh as if they were already added to the agent.

Removing the public key file, or renaming it to have an extension other than .pub will solve the issue.

A quick check confirms that gnome-keyring-daemon is running on the system I’m having issues with:

~$ ps x | grep gnome-keyring-daemon
   4335 ?        Sl     0:00 /usr/bin/gnome-keyring-daemon --daemonize --login

~$ set | grep SSH
SSH_AGENT_LAUNCHER=gnome-keyring
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh

Replicate

Let’s replicate the issue.

Starting with a resident SSH key already on the FIDO authenticator, a YubiKey in this case. There are no SSH keys in .ssh; this is intentional:

~$ ykman fido credentials list
Enter your PIN:
ssh: 0000000000000000000000000000000000000000000000000000000000000000 openssh

~$ ls ~/.ssh
config  known_hosts

Import SSH key from the Yubikey with ssh-add, which as shown matches the one in Gitlab:

~$ ssh-add -K
Enter PIN for authenticator:
Resident identity added: ED25519-SK SHA256:cjIJhSQX9KAX4KbGneSqR9xkzfNHhZR12P9F/AmbAyI

~$ ssh-add -L
[email protected] AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIFM90WLqnOIHpj/Xekvaa6MFcG+mO4KVcK6/FL5bgbAIAAAABHNzaDo=

~$ curl https://gitlab.com/111A5AB1.keys
[email protected] AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIFM90WLqnOIHpj/Xekvaa6MFcG+mO4KVcK6/FL5bgbAIAAAABHNzaDo= Wasabi (gitlab.com)

Connection attempt fails:

~$ ssh -T [email protected]
sign_and_send_pubkey: signing failed for ED25519-SK "" from agent: agent refused operation
[email protected]: Permission denied (publickey,keyboard-interactive).

Let’s clear and try importing to disk with ssh-keygen

~$ ssh-add -D
All identities removed.

~$ ssh-add -L
The agent has no identities.

~$ cd .ssh

~/.ssh$ ssh-keygen -K -N ""
Enter PIN for authenticator:
You may need to touch your authenticator to authorize key download.
Saved ED25519-SK key to id_ed25519_sk_rk

~/.ssh$ ls
config  id_ed25519_sk_rk  id_ed25519_sk_rk.pub  known_hosts

~/.ssh$ ssh-add -L
[email protected] AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIFM90WLqnOIHpj/Xekvaa6MFcG+mO4KVcK6/FL5bgbAIAAAABHNzaDo= ssh:

Connection attempt still fails:

~/.ssh$ ssh -i ./id_ed25519_sk_rk -T [email protected]
sign_and_send_pubkey: signing failed for ED25519-SK "" from agent: agent refused operation
[email protected]: Permission denied (publickey,keyboard-interactive).

As outlined in the issue, the public key is automatically being loaded from disk:

~/.ssh$ ssh-add -D
All identities removed.

~/.ssh$ ssh-add -L
[email protected] AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIFM90WLqnOIHpj/Xekvaa6MFcG+mO4KVcK6/FL5bgbAIAAAABHNzaDo= ssh:

Let’s try again, but removing the public key first… success!

~/.ssh$ rm id_ed25519_sk_rk.pub

~/.ssh$ ls
config  id_ed25519_sk_rk  known_hosts

~/.ssh$ ssh-add -L
The agent has no identities.

$ ssh -i ./id_ed25519_sk_rk -T [email protected]
Enter PIN for ED25519-SK key /home/user/.ssh/id_ed25519_sk:
Confirm user presence for key ED25519-SK SHA256:cjIJhSQX9KAX4KbGneSqR9xkzfNHhZR12P9F/AmbAyI
User presence confirmed
Welcome to GitLab, @111a5ab1!

Solution

While removing the public key and writting the "private" key to disk works with gnome-keyring, it doesn’t solve being able to use ssh-add.

The issue ends with:

So basically: here’s a well-understood problem, but one with no clear solution. Sorry. :/

I disagree.

The solution is just don’t use gnome-keyring and you’re fine. Use ssh-agent instead.

~$ rm ~/.ssh/id_ed25519_sk_rk

~$ ls ~/.ssh
config  known_hosts

~$ killall gnome-keyring-daemon

~$ ps x | grep gnome-keyring-daemon

~$ eval "$(ssh-agent)"
Agent pid 86227

~$ ssh-add -K
Enter PIN for authenticator:
Resident identity added: ED25519-SK SHA256:cjIJhSQX9KAX4KbGneSqR9xkzfNHhZR12P9F/AmbAyI

$ ssh -T [email protected]
Welcome to GitLab, @111A5AB1!

Note: Not shown above is the OpenSSH GUI dialog box prompting with "Enter PIN and confirm user presence for ED25519-SK key SHA:cjIJhSQX9KAX4KbGneSqR9xkzfNHhZR12P9F/AmbAyI" followed by the need to physically touch the key

111a5ab1/gnome-keyring-daemon.adoc

sign_and_send_pubkey: signing failed for ED25519-SK

TL;DR

Overview

Replicate

Solution