gpg-agent has support for the ssh-agent protocol, but first we need to enable it:

echo enable-ssh-support >> $HOME/.gnupg/gpg-agent.conf
gpg-connect-agent reloadagent /bye

This lets you use the authentication keys saved on your Yubikey for ssh authentication. To point at gpg-agent instead of ssh-agent, we need to change the expected socket:

export SSH_AUTH_SOCK=$XDG_RUNTIME_DIR/gnupg/S.gpg-agent.ssh

Make sure our public key is loaded:

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQ...lwumDDLYIeSJLfWjh1p5Rr7hL/0f3z7F3Kds5qUkYkxa12mBbN cardno:000607020192

Notice the cardno comment shows the same serial number that gpg recognizes for the Yubikey. Not only does this confirm that our key has been properly loaded, but it happens to be the correct format for adding to ssh authorized_keys files. Take the exact output of ssh-add -L and append it to $HOME/.ssh/authorized_keys on whichever machines you want to be able to log into with this key. You’ll need to do the same for each Yubikey, since we generated a separate authentication key for each one.

From the terminal where you set SSH_AUTH_SOCK, you should now be able to ssh to a machine where you added the key to authorized_keys. You’ll have to enter your PIN if you haven’t done so in a while, and if you set your key to require a touch for authentication, you’ll also have to touch the key. Then you should see your shell on the remote machine without typing a password.

Unfortunately, the default setup (on Fedora at least) is hardcoded to start ssh-agent when it starts your X session, so you would have to follow these steps on each new terminal window you open. Luckily, there’s a way to override this. Create a file /etc/X11/xinit/xinitrc.d/90-gpg-agent.sh with these contents:

#!/bin/bash

export SSH_AUTH_SOCK=${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh

gpg-connect-agent -q /bye 

This will be sourced by all of the Fedora X startup scripts, regardless of your desktop environment. If SSH_AUTH_SOCK is set, ssh-agent is not launched when starting the session, and SSH_AUTH_SOCK gets inherited by all the child processes in the session. Everything should “just work” after logging out and starting a new X11 session.

If you’re using Wayland, this doesn’t work. You may be able to achieve a similar effect by adding a file in /etc/config/environment.d. I’ll probably update this once Cinnamon supports Wayland and I have to figure it out.