Why won't my NFS shares mount at boot?

April 29, 2021 systemd

This is part 2 of "Retrying with NGINX & systemd" - it turns out that I had to fix the underlying issue in order to have my NFS shares reliably mount at boot. Had the mount units accepted the same 'solution' as the nginx service I'd probably never fixed it. :)

This was a interesting trip down the systemd documentation and was actually quite simple once I understood it (after extensive research + trial and error). It all came down to not having a proper definition of when the network was up and running.

Solution

TL;DR - I use systemd-networkd and any service that relies on network connectivity should depend on that target (which they typically do through network-online.target). I have multiple vlans on my main interface but only one that provides routing (enp1s0.10), so I had to tell systemd-networkd-wait-online.service which interface that had to be up for anything to work.

# systemctl edit --full systemd-networkd-wait-online.service
[Unit]
Description=Wait for Network to be Configured
Documentation=man:systemd-networkd-wait-online.service(8)
DefaultDependencies=no
Conflicts=shutdown.target
Requires=systemd-networkd.service
After=systemd-networkd.service
Before=network-online.target shutdown.target

[Service]
Type=oneshot
# Here's the important part                             <-- here -->
ExecStart=/usr/lib/systemd/systemd-networkd-wait-online -i enp1s0.10
RemainAfterExit=yes

[Install]
WantedBy=network-online.target

Don't forget to enable it as well:

systemctl enable --now systemd-networkd-wait-online.service

`systemd-fstab-generator`

Another interesting thing I discovered was the systemd-fstab-generator, which apparently translates entries from my /etc/fstab into systemd units. Sometimes you have to ask it (by force) to update your mount units after changing your fstab.

A typical mount unit looks like this, given the fstab line in the comment:

  # fstab line:
  # 192.168.0.10:/export/things /srv/nfs/things nfs4 rw,bg
  # ---
  # Resulting srv-nfs-things.mount
  # ---
  # Automatically generated by systemd-fstab-generator

  [Unit]
  Documentation=man:fstab(5) man:systemd-fstab-generator(8)
  SourcePath=/etc/fstab
  Requires=network-online.target
  After=network-online.target

  [Mount]
  Where=/srv/nfs/things
  What=192.168.0.10:/export/things
  Type=nfs4
  TimeoutSec=infinity
  Options=x-systemd.mount-timeout=infinity,retry=10000,nofail,rw,bg,fg

These didn't properly update for me after changing the fstab line. So I moved the mount files to /tmp and ran /usr/lib/systemd/system-generators/systemd-fstab-generator

…and nothing. What the..?

So apparently this program implicitly writes it's output to /tmp. It doesn't take any args (from what the manual page states on my machine), but apparently you can tell it where to output it's files as one of three args. The other two args can/must be empty strings.

# /usr/lib/systemd/system-generators/systemd-fstab-generator /etc/systemd/system '' ''

I've read the man-page for systemd-fstab-generator again and there are no mentions of this, which is confusing. Most other documentation for systemd has been pretty good.

Oh well, it's working now. And my nginx doesn't need any retries at boot anymore!