2019-09-26 Kerberos, NFS, system management, updated

Simple Configuration of GNU/Linux Clients for Kerberized NFS

Table of Contents

This note discusses options for GNU/Linux client configurations to mount Kerberized NFSv4 exports. It documents a setup with minimal admin requirements which seems no to be widely known. It is likely to be useful for sites where clients don’t necessarily have Kerberos host principals — i.e. not the privileged action of ‘joining the domain’ in Active Directory speak — and want to treat NFS similarly to CIFS.

TL;DR: see EL 7 (RHEL 7 and clones) and Ubuntu 16.04 for setup recipes. Those should carry forward to RHEL8 and 9, and more recent Debian-ish systems respectively.

Updates

2022-07-18:

Mention anonymous tickets as a possibility.

2022-12-12

Mention more recent distro releases.

Background

Suppose a large-ish organization, such as a university, wants to provide Kerberos-authenticated NFSv4 storage widely to GNU/Linux clients, similarly to the support for CIFS, which is likely in place, possibly to access he same storage.Presumably there will be restrictions to the local network in the NFS exports.

 Assume that the Kerberos realm involved uses Active Directory (AD) — though the client setup would be the same for any Kerberos realm.I’m no apologist for Microsoft, but AD is fine as a Kerberos server.

 To discuss this, I’ll assume Kerberos knowledge, but Simo’s blog is one possible source of information on principals and keys.

Overall, the aim is:

  • Loose coupling to AD, specifically not ‘joining the domain’ or any other action that requires privileges on AD to configure a client;Experience since the early days of AD suggests avoiding strong coupling, simply using AD Kerberos and maybe LDAP, and especially not allowing AD to pollute the site DNS.

  • No sysadmin interaction on the client to provide credentials at mount time (e.g. for unattended reboots and automounting);

  • General similarity to handling of CIFS, which is assumed to be familiar, particularly security-wise. That’s obviously reasonable if CIFS and NFS export the same filesystem(s).

All the guides I’ve seen for setting up Kerberized NFS require Kerberos admin privilege to create host principals (the relevant part of ‘joining the domain’), or at least host-specific nfs/ service principals.You wouldn’t expect a client of a service to need a service principal, but the GSS infrastructure for Linux does need one (see the gssd material).

 For instance, Red Hat’s instructions involve joining the domain with realmd (though you could use basic Kerberos tools).

That appears to be a blocking problem which would at best require some sort of proxy service for creating client nfs/ principals on demand, given that setting up all clients with administrative privileges is likely not to be practical. Apart from support issues making it impractical and/or undesirable on-site, that would probably rule out off-site use via private networking of some sort.Also there appear to be problems with dual boot systems due to periodic rotation of the host key not being synchronized across operating systems, which may or may not be easily soluble.

 Anyway, such a setup isn’t usually required for client use of CIFS, contrary to the aim above; any client can probably just get a TGT for Kerberized CIFS access (e.g. for smbclient -k, or automounted filesystems).

Context:   This note is based on an original for the University of Manchester,UoM people may be able to guess values for the elided local names, but will at least need local documentation for the nfs/anyhost Kerberos key.

 written after researching a technique to satisfy the aims above. The original instructions for the site simply followed the Red Hat instructions for joining the domain, and still required manually supplying credentials at filesystem mount time.

General Solution

So, When all else fails, read the documentation.

It turns out that a list of principals is used for the mount protocol, and the first one that is available is tried (gssd). The key insight is that the list includes ones that aren’t host-specific, like nfs/anyname. For this method, the only action requiring Kerberos admin privilege is adding a single such service principal with a well-known key to the site realm (AD domain).A Microsoft note shows setting up service principals on AD with setspn; probably ignore the rest of it about NFS setup.

 Assume it is created as nfs/anyhost@DS.EXAMPLE.AC.UK.Presumably you could use an anonymous ticket, but apparently not at my site.

The examples of this scheme below assume a client username fred in a DNS domain example.ac.uk which has an AD service at ds.example.ac.uk for Kerberos realm DS.EXAMPLE.AC.UK, and an NFS service at storage.example.ac.uk exporting a filesystem /nfs4. Permissions on exports are assumed to be set to allow anyone with Kerberos credentials to access them.

For the examples, names and keys were substituted in example transcripts of sessions run at Manchester. The filesystem was a parallel NFS export of the one supplying the shared CIFS per-user MS Windows ‘drives’; it would be similar with Unix home directories intended for NFS export.

The examples use sec=krb5i because sec=krb5 isn’t allowed on this site.That doesn’t follow the aims above, as krb5 provides similar security to CIFS exports of the same filesystems, but doesn’t break anything.

 krb5p is also an option for privacy protection at the expense of performance. Note that the permissions shown by ls, as in the examples, can be misleading, since access is actually controlled by ACLs.

The minimal required steps for the client are:

  1. Configure Kerberos for the realm in krb5.conf(5) as usual;

  2. Install the NFS client package(s);

  3. Get the nfs/ key into the system keytab;

  4. Configure the id mapper;

  5. (Re)start relevant services;

  6. Mount appropriately.

Those don’t address things like non-trivial id mapping (if server and client disagree about names), maintaining user credentials through expiry times, ACLs, and Kerberized home directories — probably the subject of another note or two.

Configuration Examples

EL 7 (RHEL 7 and clones)

This is a demonstration of setting up from scratch on a clean CentOS 7 system. If the client already uses Kerberos authentication, the krb5-workstation package will already be installed and configured, and the authconfig step will be unnecessary. I’d use the Heimdal clients, as for Ubuntu, but they’re only in the EPEL repository, not main EL. (There are incompatibilities between the MIT and Heimdal implementations, even in the base configuration, like the current lack of include/includedir support in Heimdal krb5.conf.)

1 # yum install -y nfs-utils krb5-workstation
    [...]
. # authconfig --krb5kdc=ds.example.ac.uk --krb5adminserver=ds.example.ac.uk --krb5realm=DS.EXAMPLE.AC.UK --update
2 # ktutil
  ktutil:  addent -key -p nfs/anyhost@DS.EXAMPLE.AC.UK -k 5 -e aes256-cts-hmac-sha1-96
  Key for nfs/anyhost@DS.EXAMPLE.AC.UK (hex): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  ktutil:  wkt /etc/krb5.keytab
  ktutil:  q
3 # sed -i 's/# *Domain = .*$/Domain = ds.example.ac.uk/' /etc/idmapd.conf
. # systemctl restart nfs-client.target
4 # mount -o vers=4.0,sec=krb5i storage.example.ac.uk:/nfs4 /mnt
. # ls -l /mnt
  ls: cannot open directory /mnt: Permission denied

Notes, by line number:

  1. Minimal package requirements.

  2. See ktutil(1) and rpc.gssd(8). The actual key is elided here.

  3. Maybe not strictly necessary; see idmapd.conf(5).

  4. vers=4.0 was needed with certain (older?) combinations of server (Isilon) and RHEL7 kernel which didn’t negotiate correctly; it’s not clear which was at fault. Only use it if necessary.

As above, root can’t access the mount without an appropriate Kerberos ticket. A user can get one and access their files, although they don’t have permission for the mount itself. Also the group is shown as nobody with misleading permissions, since the passwd database there doesn’t use AD LDAP, and access is controlled by ACLs at the server end.

$ kinit
Password for fred@DS.EXAMPLE.AC.UK: ...
$ ls /mnt
ls: cannot open directory /mnt: Permission denied
$ ls -ld /mnt
drwxrwx--x. 118005 nobody nobody 3071687 Oct 23 10:03 /mnt
$ ls -ld /mnt/fred
drwxrwx---. 11 fred nobody 314 Sep 13 15:14 /mnt/fred
$ ls -l /mnt/fred/desktop.ini
-rwxrwx---. 1 fred nobody 402 Feb 19  2019 /mnt/fred/desktop.ini
$

Ubuntu 16.04

As with RHEL, this is with a clean minimal installation, and Kerberos may already be configured in practice. The Heimdal clients are actually more convenient than the MIT Kerberos ones: see the notes below.

1 # apt install -y nfs-common krb5-user
  [...]
2 # dpkg-reconfigure krb5-config
. # sed -i "/NEED_GSSD=/s/$/yes/" /etc/default/nfs-common
. # sed -i "s/# *Domain = .*$/Domain = ds.example.ac.uk/" /etc/idmapd.conf
3 # ktutil
  ktutil:  addent -key -p nfs/anyhost@DS.EXAMPLE.AC.UK -k 5 -e aes256-cts-hmac-sha1-96
  Key for nfs/anyhost@DS.EXAMPLE.AC.UK (hex): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  ktutil:  wkt /etc/krb5.keytab
  ktutil:  q
. # systemctl restart nfs-client.target
4 # mount -o sec=krb5i storage.example.ac.uk:/nfs4 /mnt

Notes, by line number:

  1. Configure with DS.EXAMPLE.AC.UK. For Heimdal clients, install heimdal-clients, not krb5-user;

  2. Configure with

    DS.EXAMPLE.AC.UK
Yes
ds.example.ac.uk
ds.example.ac.uk

  3. With Heimdal just use

    # ktutil add --hex -p nfs/anyhost@DS.EXAMPLE.AC.UK -V 5 \
  -e aes256-cts-hmac-sha1-96 \
  --password=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

  4. No vers=4.0 was required with this kernel and the same server configuration as with EL7.

In contrast to the EL7 case, there was some problem seen with ls below, and reported in syslog, with the Nobody-Group id mapping; that is despite similar entries in /etc/idmapd.conf and similar versions of the client software. Increasing the idmap verbosity didn’t help, and I haven’t tried seriously to debug it. Using sssd (with EL, but not Debian/Ubuntu by default) seems the most likely difference. As seen in this example, Heimdal’s kinit is more flexible than MIT’s:

# sudo -s -u fred kinit fred ls -l /p/fred/desktop.ini
fred@DS.MAN.AC.UK's Password:
-rwxrwx--- 1 fred 4294967294 402 Feb 19  2019 /p/fred/desktop.ini

Other Systems

Similar setups to the above, with some variations between them, have been tested on several likely distributions, both with MIT and Heimdal, and systemd-filed or not, where there’s the choice. I expect other Unixes like the BSDs will work straightforwardly, and also SunOS/Illumos which I’ve used before (though only SunOS 5.10, and I forget how that was configured).Kerberized NFS has probably seen most serious production use on SunOS.

MS Windows I don’t know about, but there is apparently a native client as well as UMich’s client.

MacOS should work, but has been reported to be messy in some way I don’t know.

Automounting

Automounting works fine in this setup (at least with autofs.I don’t know about AMD, although it looks dead now.

 For instance, suppose the server exports a filesystem with per-user directories as in the example above. They might home directories, although you probably wouldn’t have these names for homes.Kerberized homes have particular credential management issues to beware of. For instance, SSH authorized_keys may need to be stored elsewhere.

 Then, if /etc/auto.master.d/nfs4.autofs contains

/nfs4 /etc/auto.nfs4

/etc/auto.nfs4 might contain

* -fstype=nfs4,sec=krb5i storage.example.ac.uk:/nfs4/&

so that a user with a Kerberos ticket can access /nfsv4/$USER, though if the export has permissions as in the example, its contents can’t be listed.With an expired ticket, you probably see a ‘stale file handle’ error.

Other Notes

There doesn’t appear to be a security problem — at least relative to use of CIFS — with the well-known key, but I haven’t checked with an expert.

It might be more convenient to distribute a keytab (which should be portable enough between implementations) and use that simply with rpc.gssd’s -k option to avoid the ktutil step. However, according to a Debian bug report (not checked), gssd configured arguments aren’t propagated on systemd-filed systems.

StatefulI.e. with / on a local filesystem, not a read-only shared image.

 Debian-like distributions could usefully be configured with a site-specific config-package encapsulating the steps above. Unfortunately RPM-based distributions can’t use the same technique.

Destroying user credentials (kdestroy -A) doesn’t kill access to the mount in a timely way, but on a scale of hours. I haven’t researched the mechanism.

In case you know the name of the file server, but not the relevant generally-accessible export to mount, you might be able to divine it by looking for likely names in the export list with showmount(8):

$ /sbin/showmount -e storage.example.ac.uk | grep everybody
/nfs4                                                    (everybody)

Beware that Kerberized mounts need the appropriate DNS name (probably the CNAME) for the file server, in case it has several; it must match the Kerberos service principal. For instance, my site advertises per-user CIFS shares from a ‘short-form’ addressLong and short forms, albeit now not big-endian, are possibly the last remnants of the early UK SRCNET, whose NRS name system pre-dated DNS.

 (….man.ac.uk), but the same underlying filesystem needs the ‘long-form’ CNAME (….manchester.ac.uk) for Kerberized mounts of the NFS export.

Appendix A: Original Testing Environment

I originally tested client setups with a Unix NFS and Kerberos KDC/admin server in a small Vagrant cluster in the absence of resources to use an AD instance and simulator of the site Isilon server. (It took a couple of hours, mainly due to not being familiar with MIT Kerberos admin since Heimdal isn’t in base RHEL.) This was the procedure.

Set up server (I used a CentOS 7 image) in DNS domain .test with a KDC and necessary principals, including nfs/server.test and vagrant in domain TEST. server exports NFS as:

# cat /etc/exports
/srv/nfs *.test(rw,no_root_squash,sec=krb5)

On a CentOS 7 client in the cluster

# yum install nfs-utils krb5-workstation
# authconfig --krb5kdc=server.test --krb5adminserver=server.test --krb5realm=TEST --update
# kadmin ank -randkey nfs/ignored.test  # requires root/admin p/w
# kadmin ktadd nfs/ignored.test
# sed -i 's/#Domain = local.domain.edu/Domain = test/' /etc/idmapd.conf
# systemctl start nfs-client.target
# mount server:/srv/nfs /mnt
# su - vagrant
$ ls -l /mnt
ls: cannot open directory /mnt: Permission denied
$ kinit
Password for vagrant@TEST:
$ ls -l /mnt
total 0
drwx------. 2 vagrant vagrant 27 Jun 11 11:36 a
drwxr-xr-x. 2 root    root     6 Jun 11 08:50 b
$ ls -l /mnt/a/
total 0
-rw-r--r--. 1 vagrant vagrant 0 Jun 11 11:36 vagrant-owned
$ sudo ls -l /mnt/a/
ls: cannot open directory /mnt/a/: Permission denied

References