If You’re Not Using SSH Certificates You’re Not Necessarily Doing It Wrong

Update

2022-07-05: A recent post about Tailscale SSH makes similar points to the Smallstep one. It’s not clear to me that it solves any problem you have if you use Kerberos, and it appears that you can’t authenticate to Tailscale with Kerberos, unless that’s a ‘custom provider’ you get with the ‘enterprise’ version.

There’s a post from Smallstep that is often referenced, for instance on Hacker News, If you’re not using SSH certificates you’re doing SSH wrong. Now, while SSH certificates are perfectly fine,For those all-to-familiar with horrors of the ~~Globus~~Grid Community Toolkit, this is not the X.509 certificates used by gsi-openssh.

and the article is accurate, I think, there are things it leaves out when discussing the disadvantages of SSH public keys, and notably it ignores the three-headed dog in the room.See below; elephants aren’t useful.

Smallstep have something to sell you, though that’s a fine enough (as far as I know) self-hostable service and client setup which is free software — kudos.

For what it’s worth, my background for this is in universities, with research facilities, and operating HPC compute clusters (somewhat closed systems). Research computing users, at least, don’t seem to have particular problems with using the public key mechanism (other than when host keys change due to poor system management), whatever its general (de)merits.

The Smallstep post and this article concern OpenSSH. I don’t know if any other implementations, apart from Teleport’s, do certificates. Also, we’re talking about an organization that can run a certificate authority, and so probably runs other services.

So, the three problems argued in the post are:

SSH user experience is terrible. Connected with Trust-On-First-Use host keys which may change, with “weird new credentials”;
Operating SSH at scale is a disaster. Concerned with “Key approvalI don’t know what the approval bit is about.

& distribution” and “scatter[ing] key material across your fleet”;
SSH encourages bad security practices. Connected with re-keying not being done, users ‘exposed to key material’ and being encouraged to use keys across devices, and permanent trust in keys.

Concerning 1: host keys should be managed and distributed as appropriate. Obviously there is a management issue there. Unfortunately OpenSSH doesn’t have a host keys equivalent of AuthorizedKeysCommand to allow looking up host keys in LDAP, say. However, you can, and should, distribute the GlobalKnownHostsFile, perhaps on a network filesystem. It’s certainly annoying when HPC clusters don’t generate ssh_known_hosts for the cluster nodes.Which may reasonably share a host key amongst compute nodes, particularly stateless nodes booted from a network filesystem image.

You should, anyway, manage servers so that the host key is preserved across re-installation, moving the service, etc.

Concerning 2: key distribution is solved by using AuthorizedKeysCommand with AuthorizedKeysFile none, with centralized storage of public keys — typically in LDAP.E.g. using openssh-ldap-authkeys.

Concerning 3: partly see 2 but, more importantly, the authentication (authNObviously access control people don’t like typing. (Authorization is authZ.)

) method below.

Note that internally to a compute cluster, or similar, you want to use host-based authN anyway, not client public keys; see HostbasedAuthentication et al. Also note that within an organization, ~/.ssh/authorized_keys may well be shared on a network filesystem if they’re not in LDAP.

So, public key authN isn’t as bad as presented. However, it’s definitely not the best approach withing an organization and neither, I maintain, is certificate authN.

OpenSSH supports the authN methods hostbased, keyboard-interactive, none, password, publickey, and gssapi-with-mic. The last is the interesting one. While publickey via certificates provides single sign-on (SSO) within the certificate realm, but for SSH only, and hostbased finesses it, gssapi-with-mic uses general SSO provided by Kerberos.GSSAPI is actually more general than just Kerberos support, but at least the GSI mechanism needs a patched OpenSSH; I don’t know about Moonshot now.

Actually, the Smallstep post confuses the meaning of SSO. My definition of it — and I’m not alone — is obtaining some token, say once per day, which provides access to all relevant services for the day with no further interaction for authN. SSH certificates and public keys in LDAP provide that for SSH only — without the once per day bit, at least for public keys normally. The ‘SSO’ from the sort of ‘SSO providers’ the post refers to is uniform sign-on — i.e. the same credentials for each service, but still explicitly authenticating to each. It’s also not private.

So, assuming you have Kerberos — the three-headed dog mentioned above — use it for SSH. You typically will have it, at least through Active Directory to service MS Windows systems. To use it, add the servers to the Kerberos realm,‘Join the domain’ in AD parlance.

turn on GSSAPIAuthentication and GSSAPIKeyExchangeKey exchange to avoid SSH host keys is only in a patch set maintained at least in major GNU/Linux distributions, but rejected ‘upstream’.

in client and server, and be happy. You could also disable publickey. That avoids stated downsides of SSH public keys, as well as something like the additional Smallstep tools or service. You’ll want to keep Hostbased within your HPC cluster.

It’s worth noting that only hostbased and pure publickeyEither with normal keys or fixed certificates.

really play nicely with unattended automation, especially if your SSO demands some sort of multi-factor interaction.Though you could store OATH secrets and use oathtool to get new TGTs periodically, or do FIDO with a TPM.

The canonical solution for a daemon with Kerberos and no MFA is to run with credentials in a keytab under something like kstart.

Although I said this is about OpenSSH, at least PuTTY and libssh support GSSAPI.With key exchange in PuTTY; I don’t know if that’s in libssh.

Other clients, I don’t know.