2019-09-26
Simple Configuration of GNU/Linux Clients for Kerberized NFS
This note discusses options for GNU/Linux client configurations to mount Kerberized NFSv4 exports. It documents a setup with minimal admin requirements which seems no to be widely known. It is likely to be useful for sites where clients don’t necessarily have Kerberos host principals — i.e. not the privileged action of ‘joining the domain’ in Active Directory speak — and want to treat NFS similarly to CIFS.
TL;DR: see EL 7 (RHEL 7 and clones) and Ubuntu 16.04 for setup recipes. Those should carry forward to RHEL8 and 9, and more recent Debian-ish systems respectively.
Updates
- 2022-07-18:
-
Mention anonymous tickets as a possibility.
- 2022-12-12
-
Mention more recent distro releases.
Background
Suppose a large-ish organization, such as a university, wants to provide
Kerberos-authenticated NFSv4 storage widely to GNU/Linux clients, similarly to
the support for CIFS, which is likely in place, possibly to access he same
storage.Presumably there will be restrictions to the local network
in the NFS exports.
Assume that the Kerberos realm involved uses Active
Directory (AD) — though the client setup would be the same for any Kerberos
realm.I’m no apologist for Microsoft, but AD is fine as a Kerberos
server.
To discuss this, I’ll assume Kerberos knowledge, but
Simo’s blog is one possible source
of information on principals and keys.
Overall, the aim is:
Loose coupling to AD, specifically not ‘joining the domain’ or any other action that requires privileges on AD to configure a client;Experience since the early days of AD suggests avoiding strong coupling, simply using AD Kerberos and maybe LDAP, and especially not allowing AD to pollute the site DNS.
No sysadmin interaction on the client to provide credentials at mount time (e.g. for unattended reboots and automounting);
General similarity to handling of CIFS, which is assumed to be familiar, particularly security-wise. That’s obviously reasonable if CIFS and NFS export the same filesystem(s).
All the guides I’ve seen for setting up Kerberized NFS require Kerberos admin
privilege to create host principals (the relevant part of ‘joining the
domain’), or at least host-specific nfs/
service
principals.You wouldn’t expect a client of a service to need a
service principal, but the GSS infrastructure for Linux does need one (see
the gssd material).
For instance,
Red
Hat’s instructions involve joining the domain with
realmd (though you could use basic Kerberos tools).
That appears to be a blocking problem which would at best require some sort of
proxy service for creating client nfs/
principals on demand, given
that setting up all clients with administrative privileges is likely not to be
practical. Apart from support issues making it impractical and/or undesirable
on-site, that would probably rule out off-site use via private networking of
some sort.Also there appear to be problems with dual boot systems
due to periodic rotation of the host key not being synchronized across
operating systems, which may or may not be easily soluble.
Anyway, such a
setup isn’t usually required for client use of CIFS, contrary to the aim
above; any client can probably just get a TGT for Kerberized CIFS access
(e.g. for smbclient -k
, or automounted filesystems).
Context: This note is based on an original for the University of
Manchester,UoM people may be able to guess values for the elided
local names, but will at least need local documentation for the
nfs/anyhost
Kerberos key.
written after researching a technique
to satisfy the aims above. The original instructions for the site simply
followed the Red Hat instructions for joining the domain, and still required
manually supplying credentials at filesystem mount time.
General Solution
So, When all else fails, read the documentation.
It turns out that a list of principals is used for the mount protocol, and the
first one that is available is tried (gssd). The key insight
is that the list includes ones that aren’t host-specific, like
nfs/
anyname. For this method, the only action requiring
Kerberos admin privilege is adding a single such service principal with a
well-known key to the site realm (AD domain).A
Microsoft
note shows setting up service principals on AD with setspn
;
probably ignore the rest of it about NFS setup.
Assume it is created as
nfs/anyhost@DS.EXAMPLE.AC.UK
.Presumably you could use an
anonymous ticket, but apparently not at my site.
The examples of this scheme below assume a client username fred
in a
DNS domain example.ac.uk
which has an AD service at
ds.example.ac.uk
for Kerberos realm DS.EXAMPLE.AC.UK
, and an
NFS service at storage.example.ac.uk
exporting a filesystem
/nfs4
. Permissions on exports are assumed to be set to allow anyone
with Kerberos credentials to access them.
For the examples, names and keys were substituted in example transcripts of sessions run at Manchester. The filesystem was a parallel NFS export of the one supplying the shared CIFS per-user MS Windows ‘drives’; it would be similar with Unix home directories intended for NFS export.
The examples use sec=krb5i
because sec=krb5
isn’t allowed on
this site.That doesn’t follow the aims above, as krb5
provides similar security to CIFS exports of the same filesystems, but
doesn’t break anything.
krb5p
is also an option for
privacy protection at the expense of performance. Note that the permissions
shown by ls
, as in the examples, can be misleading, since access is
actually controlled by ACLs.
The minimal required steps for the client are:
Configure Kerberos for the realm in
krb5.conf(5)
as usual;Install the NFS client package(s);
Get the
nfs/
key into the system keytab;Configure the id mapper;
(Re)start relevant services;
Mount appropriately.
Those don’t address things like non-trivial id mapping (if server and client disagree about names), maintaining user credentials through expiry times, ACLs, and Kerberized home directories — probably the subject of another note or two.
Configuration Examples
EL 7 (RHEL 7 and clones)
This is a demonstration of setting up from scratch on a clean CentOS 7
system. If the client already uses Kerberos authentication, the
krb5-workstation
package will already be installed and configured,
and the authconfig
step will be unnecessary. I’d use the Heimdal
clients, as for Ubuntu, but they’re only in the EPEL repository, not main EL.
(There are incompatibilities between the MIT and Heimdal implementations, even
in the base configuration, like the
current lack of
include
/includedir
support in Heimdal
krb5.conf
.)
1 # yum install -y nfs-utils krb5-workstation
[...]
. # authconfig --krb5kdc=ds.example.ac.uk --krb5adminserver=ds.example.ac.uk --krb5realm=DS.EXAMPLE.AC.UK --update
2 # ktutil
ktutil: addent -key -p nfs/anyhost@DS.EXAMPLE.AC.UK -k 5 -e aes256-cts-hmac-sha1-96
Key for nfs/anyhost@DS.EXAMPLE.AC.UK (hex): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ktutil: wkt /etc/krb5.keytab
ktutil: q
3 # sed -i 's/# *Domain = .*$/Domain = ds.example.ac.uk/' /etc/idmapd.conf
. # systemctl restart nfs-client.target
4 # mount -o vers=4.0,sec=krb5i storage.example.ac.uk:/nfs4 /mnt
. # ls -l /mnt
ls: cannot open directory /mnt: Permission denied
Notes, by line number:
Minimal package requirements.
See ktutil(1) and rpc.gssd(8). The actual key is elided here.
Maybe not strictly necessary; see idmapd.conf(5).
vers=4.0
was needed with certain (older?) combinations of server (Isilon) and RHEL7 kernel which didn’t negotiate correctly; it’s not clear which was at fault. Only use it if necessary.
As above, root can’t access the mount without an appropriate Kerberos ticket.
A user can get one and access their files, although they don’t have permission
for the mount itself. Also the group is shown as nobody
with
misleading permissions, since the passwd database there doesn’t use AD LDAP,
and access is controlled by ACLs at the server end.
$ kinit
Password for fred@DS.EXAMPLE.AC.UK: ...
$ ls /mnt
ls: cannot open directory /mnt: Permission denied
$ ls -ld /mnt
drwxrwx--x. 118005 nobody nobody 3071687 Oct 23 10:03 /mnt
$ ls -ld /mnt/fred
drwxrwx---. 11 fred nobody 314 Sep 13 15:14 /mnt/fred
$ ls -l /mnt/fred/desktop.ini
-rwxrwx---. 1 fred nobody 402 Feb 19 2019 /mnt/fred/desktop.ini
$
Ubuntu 16.04
As with RHEL, this is with a clean minimal installation, and Kerberos may already be configured in practice. The Heimdal clients are actually more convenient than the MIT Kerberos ones: see the notes below.
1 # apt install -y nfs-common krb5-user
[...]
2 # dpkg-reconfigure krb5-config
. # sed -i "/NEED_GSSD=/s/$/yes/" /etc/default/nfs-common
. # sed -i "s/# *Domain = .*$/Domain = ds.example.ac.uk/" /etc/idmapd.conf
3 # ktutil
ktutil: addent -key -p nfs/anyhost@DS.EXAMPLE.AC.UK -k 5 -e aes256-cts-hmac-sha1-96
Key for nfs/anyhost@DS.EXAMPLE.AC.UK (hex): xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ktutil: wkt /etc/krb5.keytab
ktutil: q
. # systemctl restart nfs-client.target
4 # mount -o sec=krb5i storage.example.ac.uk:/nfs4 /mnt
Notes, by line number:
Configure with
DS.EXAMPLE.AC.UK
. For Heimdal clients, installheimdal-clients
, notkrb5-user
;Configure with
DS.EXAMPLE.AC.UK Yes ds.example.ac.uk ds.example.ac.uk
With Heimdal just use
# ktutil add --hex -p nfs/anyhost@DS.EXAMPLE.AC.UK -V 5 \ -e aes256-cts-hmac-sha1-96 \ --password=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
No
vers=4.0
was required with this kernel and the same server configuration as with EL7.
In contrast to the EL7 case, there was some problem seen with
ls
below, and reported in syslog, with the
Nobody-Group
id mapping; that is despite similar entries in
/etc/idmapd.conf
and similar versions of the client software.
Increasing the idmap
verbosity didn’t help, and I haven’t tried
seriously to debug it. Using sssd
(with EL, but not
Debian/Ubuntu by default) seems the most likely difference. As seen in
this example, Heimdal’s kinit
is more flexible than MIT’s:
# sudo -s -u fred kinit fred ls -l /p/fred/desktop.ini
fred@DS.MAN.AC.UK's Password:
-rwxrwx--- 1 fred 4294967294 402 Feb 19 2019 /p/fred/desktop.ini
Other Systems
Similar setups to the above, with some variations between them, have
been tested on several likely distributions, both with MIT and Heimdal, and
systemd
-filed or not, where there’s the choice. I expect other Unixes like
the BSDs will work straightforwardly, and also SunOS/Illumos which I’ve used
before (though only SunOS 5.10, and I forget how that was
configured).Kerberized NFS has probably seen most serious production
use on SunOS.
MS Windows I don’t know about, but there is apparently a native client as well as UMich’s client.
MacOS should work, but has been reported to be messy in some way I don’t know.
Automounting
Automounting works fine in this setup (at least with
autofs
.I don’t know about AMD,
although it looks dead now.
For instance, suppose the server exports a
filesystem with per-user directories as in the example above. They might home
directories, although you probably wouldn’t have these names for
homes.Kerberized homes have particular credential management issues
to beware of. For instance, SSH authorized_keys
may need to be
stored elsewhere.
Then, if /etc/auto.master.d/nfs4.autofs
contains
/nfs4 /etc/auto.nfs4
/etc/auto.nfs4
might contain
* -fstype=nfs4,sec=krb5i storage.example.ac.uk:/nfs4/&
so that a user with a Kerberos ticket can access /nfsv4/$USER
,
though if the export has permissions as in the example, its contents can’t be
listed.With an expired ticket, you probably see a ‘stale file
handle’ error.
Other Notes
There doesn’t appear to be a security problem — at least relative to use of CIFS — with the well-known key, but I haven’t checked with an expert.
It might be more convenient to distribute a keytab (which should be
portable enough between implementations) and use that simply with
rpc.gssd
’s -k
option to avoid the
ktutil
step. However, according to a
Debian
bug report (not checked), gssd
configured arguments aren’t
propagated on systemd
-filed systems.
StatefulI.e. with /
on a local filesystem, not a read-only
shared image.
Debian-like distributions could usefully be configured with
a site-specific
config-package
encapsulating the steps above. Unfortunately RPM-based distributions can’t use
the same technique.
Destroying user credentials (kdestroy -A
) doesn’t kill access to the
mount in a timely way, but on a scale of hours. I haven’t researched the
mechanism.
In case you know the name of the file server, but not the relevant
generally-accessible export to mount, you might be able to divine it by
looking for likely names in the export list with showmount(8)
:
$ /sbin/showmount -e storage.example.ac.uk | grep everybody
/nfs4 (everybody)
Beware that Kerberized mounts need the appropriate DNS name (probably the
CNAME) for the file server, in case it has several; it must match the Kerberos
service principal. For instance, my site advertises per-user CIFS shares from
a ‘short-form’ addressLong and short forms, albeit now
not big-endian, are possibly the last remnants of the early UK
SRCNET, whose NRS name
system pre-dated DNS.
(….man.ac.uk
), but the same underlying
filesystem needs the ‘long-form’ CNAME (….manchester.ac.uk
) for
Kerberized mounts of the NFS export.
Appendix A: Original Testing Environment
I originally tested client setups with a Unix NFS and Kerberos KDC/admin server in a small Vagrant cluster in the absence of resources to use an AD instance and simulator of the site Isilon server. (It took a couple of hours, mainly due to not being familiar with MIT Kerberos admin since Heimdal isn’t in base RHEL.) This was the procedure.
Set up server
(I used a CentOS 7 image) in DNS domain
.test
with a KDC and necessary principals, including
nfs/server.test
and vagrant
in domain TEST
.
server
exports NFS as:
# cat /etc/exports
/srv/nfs *.test(rw,no_root_squash,sec=krb5)
On a CentOS 7 client in the cluster
# yum install nfs-utils krb5-workstation
# authconfig --krb5kdc=server.test --krb5adminserver=server.test --krb5realm=TEST --update
# kadmin ank -randkey nfs/ignored.test # requires root/admin p/w
# kadmin ktadd nfs/ignored.test
# sed -i 's/#Domain = local.domain.edu/Domain = test/' /etc/idmapd.conf
# systemctl start nfs-client.target
# mount server:/srv/nfs /mnt
# su - vagrant
$ ls -l /mnt
ls: cannot open directory /mnt: Permission denied
$ kinit
Password for vagrant@TEST:
$ ls -l /mnt
total 0
drwx------. 2 vagrant vagrant 27 Jun 11 11:36 a
drwxr-xr-x. 2 root root 6 Jun 11 08:50 b
$ ls -l /mnt/a/
total 0
-rw-r--r--. 1 vagrant vagrant 0 Jun 11 11:36 vagrant-owned
$ sudo ls -l /mnt/a/
ls: cannot open directory /mnt/a/: Permission denied