Debian Bug report logs - #860264
nfs-kernel-server: NFS starts before DNS works, "exportfs: Failed to resolve ..."

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Greg Wooledge <wooledg@eeg.ccf.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: nfs-kernel-server: NFS starts before DNS works, "exportfs: Failed to resolve ..."

Date: Thu, 13 Apr 2017 14:24:34 -0400

Package: nfs-kernel-server
Version: 1:1.3.4-2.1
Severity: normal

The NFS server starts before DNS is working, causing exportfs to fail:

wooledg:~$ sudo systemctl status nfs-kernel-server
[sudo] password for wooledg:
* nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor prese
  Drop-In: /etc/systemd/system/nfs-server.service.d
           `-waitlonger.conf
   Active: active (exited) since Wed 2017-04-12 16:39:29 EDT; 6min ago
 Main PID: 513 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Apr 12 16:39:29 wooledg systemd[1]: Starting NFS server and services...
Apr 12 16:39:29 wooledg exportfs[510]: exportfs: Failed to resolve ebase
Apr 12 16:39:29 wooledg exportfs[510]: exportfs: Failed to resolve ebase
Apr 12 16:39:29 wooledg exportfs[510]: exportfs: Failed to resolve ebase-fla
Apr 12 16:39:29 wooledg systemd[1]: Started NFS server and services.

I attempted to work around this by creating a drop-in unit thing:

wooledg:~$ cat /etc/systemd/system/nfs-server.service.d/waitlonger.conf
[Unit]
Wants= network-online.target
After= network-online.target

But this didn't help.  With or without the drop-in thing, the exportfs
command fails at boot time.  If I login and manually run "exportfs -a"
or restart nfs-kernel-server (or nfs-server), then it's fine.

For now, I'm willing to entertain any kludge workarounds.  This is just
a desktop workstation.  But anyone experiencing this on a server may
want a more robust fix.

-- Package-specific info:
-- rpcinfo --
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  40904  mountd
    100005    1   tcp  50855  mountd
    100005    2   udp  53578  mountd
    100005    2   tcp  40253  mountd
    100005    3   udp  33327  mountd
    100005    3   tcp  47879  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp  56937  nlockmgr
    100021    3   udp  56937  nlockmgr
    100021    4   udp  56937  nlockmgr
    100021    1   tcp  41093  nlockmgr
    100021    3   tcp  41093  nlockmgr
    100021    4   tcp  41093  nlockmgr
-- /etc/default/nfs-kernel-server --
RPCNFSDCOUNT=8
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""
-- /etc/exports --
/home	ebase(ro,sync,no_subtree_check) ebase-fla(ro,sync,no_subtree_check)
-- /proc/fs/nfs/exports --
# Version 1.1
# Path Client(Flags) # IPs
/home	ebase.eeg.ccf.org(ro,root_squash,sync,wdelay,no_subtree_check,uuid=57cd3d98:68de46b4:b8aded3c:8e2f7c10,sec=1)
/home	ebase-fla.eeg.ccf.org(ro,root_squash,sync,wdelay,no_subtree_check,uuid=57cd3d98:68de46b4:b8aded3c:8e2f7c10,sec=1)
/	ebase.eeg.ccf.org(ro,root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=57cd3d98:68de46b4:b8aded3c:8e2f7c10,sec=390003:390004:390005:1)
/	ebase-fla.eeg.ccf.org(ro,root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=57cd3d98:68de46b4:b8aded3c:8e2f7c10,sec=390003:390004:390005:1)

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages nfs-kernel-server depends on:
ii  init-system-helpers  1.47
ii  keyutils             1.5.9-9
ii  libblkid1            2.29.2-1
ii  libc6                2.24-9
ii  libcap2              1:2.25-1
ii  libsqlite3-0         3.16.2-3
ii  libtirpc1            0.2.5-1.1
ii  libwrap0             7.6.q-26
ii  lsb-base             9.20161125
ii  netbase              5.4
ii  nfs-common           1:1.3.4-2.1
ii  ucf                  3.0036

nfs-kernel-server recommends no packages.

nfs-kernel-server suggests no packages.

-- debconf-show failed

Message #10 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Greg Wooledge <wooledg@eeg.ccf.org>

To: 860264@bugs.debian.org

Subject: Re: Bug#860264: Acknowledgement (nfs-kernel-server: NFS starts before DNS works, "exportfs: Failed to resolve ...")

Date: Mon, 17 Apr 2017 09:19:59 -0400

I believe I was able to get this working by making a change to the
/etc/network/interfaces file:

# allow-hotplug eth0
auto eth0
iface eth0 inet dhcp

I replaced the allow-hotplug with auto.  Now, after booting, it appears
to be working:

wooledg:~$ sudo systemctl status nfs-server
* nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor prese
  Drop-In: /etc/systemd/system/nfs-server.service.d
           `-waitlonger.conf
   Active: active (exited) since Mon 2017-04-17 08:57:17 EDT; 2min 17s ago
  Process: 555 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/
  Process: 552 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS
 Main PID: 555 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/nfs-server.service

Apr 17 08:57:17 wooledg systemd[1]: Starting NFS server and services...
Apr 17 08:57:17 wooledg systemd[1]: Started NFS server and services.

wooledg:~$ /sbin/showmount -e
Export list for wooledg:
/home ebase-fla.eeg.ccf.org,ebase.eeg.ccf.org

I only tested one time, though.  There might still be a race condition.

Inspiration for this came from
https://unix.stackexchange.com/questions/209832/debian-systemd-network-online-target-not-working
which shows an Ubuntu unit file (not present on my stretch installation).
This unit uses the "ifquery --list --exclude lo --allow auto" command to
try to determine which interfaces it should wait for.  This command did
not work for me, since my interface had been created by the installer with
"allow-hotplug" instead of "auto".

So, on the theory that maybe systemd's "network-online.target" is doing
something similar to this, I tried the change.  It seemed to work.

Systemd brags about how great its documentation is, but it doesn't bother
explaining what network-online.target actually does, or how to make it
work.

Digging around some more (for the benefit of anyone who may be reading
this bug report and trying to learn, as I am):

wooledg:~$ systemctl list-dependencies network-online.target
network-online.target
* `-networking.service

wooledg:~$ cat /lib/systemd/system/networking.service
[...]
[Service]
Type=oneshot
EnvironmentFile=-/etc/default/networking
ExecStartPre=-/bin/sh -c '[ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle'
ExecStart=/sbin/ifup -a --read-environment
ExecStop=/sbin/ifdown -a --read-environment --exclude=lo
RemainAfterExit=true
TimeoutStartSec=5min

wooledg:~$ man ifquery
[...]
       -l, --list
              For  ifquery,  list all the interfaces which match the specified
              class.  If no class specified, prints all the interfaces  listed
              as auto.
[...]

Again, the reference to "auto".  Apparently "allow-hotplug" breaks all
kinds of things.

Message #15 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Gabriel Filion <gabster@lelutin.ca>

To: 860264@bugs.debian.org

Subject: Re: Bug#860264: Acknowledgement (nfs-kernel-server: NFS starts before DNS works, "exportfs: Failed to resolve ...")

Date: Sat, 1 Sep 2018 03:16:34 -0400

[Message part 1 (text/plain, inline)]

Hi!

I'm currently experiencing the exact same issue with an NFS mount for
/home, and another nfs mount (from another server) that gets mounted
under /srv/something: at boot, nfs mount times out, and afterwards when
the boot is complete, if I ssh in and execute "mount -a" as root the
mount points become available.

On Mon, 17 Apr 2017 09:19:59 -0400 Greg Wooledge <wooledg@eeg.ccf.org>
wrote:
> wooledg:~$ systemctl list-dependencies network-online.target
> network-online.target
> * `-networking.service
> 
> wooledg:~$ cat /lib/systemd/system/networking.service
> [...]
> [Service]
> Type=oneshot
> EnvironmentFile=-/etc/default/networking
> ExecStartPre=-/bin/sh -c '[ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle'
> ExecStart=/sbin/ifup -a --read-environment

This seems to be problematic.. unfortunately if I understand correctly
"allow-hotplug" is meant to make it possible to have the network
interface be configured dynamically whenever it appears. So it might be
useful for network interface dongles ... but if I try to reason why
debian's installer configures interfaces as "allow-hotplug" by default,
I would guess that it might also be useful for laptops that have wifi
on/off physical switches to disable/enable the interface.

I'm wondering why physical interfaces (previously known as ethN) would
require this instead of using "auto", though..

Also just to add at least one data point that's more closely relevant to
the paragraph above: the unit file for network-online.target is still
the same in sid now, so the issue still exists!

I'm not sure what would be the correct solution for this to avoid
requiring ppl to manually change the definition of their interface when
they want to use a mount point at boot that requires networking :\

[signature.asc (application/pgp-signature, attachment)]

Message #20 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Gabriel Filion <gabster@lelutin.ca>

To: 860264@bugs.debian.org

Subject: Re: Bug#860264: Acknowledgement (nfs-kernel-server: NFS starts before DNS works, "exportfs: Failed to resolve ...")

Date: Tue, 27 Nov 2018 18:33:03 -0500

[Message part 1 (text/plain, inline)]

On Sat, 1 Sep 2018 03:16:34 -0400 Gabriel Filion <gabster@lelutin.ca> wrote:
> > wooledg:~$ cat /lib/systemd/system/networking.service
> > [...]
> > [Service]
> > Type=oneshot
> > EnvironmentFile=-/etc/default/networking
> > ExecStartPre=-/bin/sh -c '[ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle'
> > ExecStart=/sbin/ifup -a --read-environment
> 
> This seems to be problematic.. unfortunately if I understand correctly
> "allow-hotplug" is meant to make it possible to have the network
> interface be configured dynamically whenever it appears. So it might be
> useful for network interface dongles ... but if I try to reason why
> debian's installer configures interfaces as "allow-hotplug" by default,
> I would guess that it might also be useful for laptops that have wifi
> on/off physical switches to disable/enable the interface.
> 
> I'm wondering why physical interfaces (previously known as ethN) would
> require this instead of using "auto", though..

I've just tested changing "allow-hotplug enp0s25" to "auto enp0s25" as
suggested by Greg as a workaround.

Then I've rebooted the machine multiple times and I ended up seeing the
problem again. So I believe that this workaround does not fix the issue
after all.

the bug lies somewhere else :\

is it possible that "udevadm settle" is the command that gets stuck
sometimes?

[signature.asc (application/pgp-signature, attachment)]

Message #25 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Vincent McIntyre <vincent.mcintyre@csiro.au>

To: <860264@bugs.debian.org>

Subject: ifquery behaviour

Date: Sat, 15 Dec 2018 19:39:56 +1100

# grep auto /etc/network/interfaces
auto lo
# grep hotplug /etc/network/interfaces
allow-hotplug enp0s31f6

# ifquery --read-environment --list --exclude=lo  # as in unit file
#
# ifquery --read-environment --list --exclude=lo --allow auto
#
# ifquery --read-environment --list --exclude=lo --allow hotplug
enp0s31f6

But if I convert enp0s31f6 to auto
# grep /etc/network/interfaces
auto lo
auto enp0s31f6

# ifquery --read-environment --list --exclude=lo --allow hotplug
#  (as expected)
# ifquery --read-environment --list --exclude=lo --allow auto
enp0s31f6
# ifquery --read-environment --list --exclude=lo
enp0s31f6

So with the interface set to auto, the ifquery in the unit file
picks it up. With the interface set to allow-hotplug, it doesn't.

I think this is part of the puzzle, but there is more to it.
I think that there are some issues arising from how ifupdown
and systemd (fail to) communicate about when exactly the network
interface is fully up and working (particularly in the dhcp case).
If you look at the work that has gone into ifupdown since the
stretch release, it seems like there are some issues there that
have now, hopefully, been resolved.

Kind regards
Vince

Message #30 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupre <anarcat@debian.org>

To: Greg Wooledge <wooledg@eeg.ccf.org>, Gabriel Filion <gabster@lelutin.ca>, Vincent McIntyre <vincent.mcintyre@csiro.au>

Cc: 860264@bugs.debian.org

Subject: similar bug with sshfs, maybe in systemd/ifupdown

Date: Tue, 29 Jan 2019 21:26:52 -0500

[Message part 1 (text/plain, inline)]

I still need to reproduce this in a normal Debian install, but I have
had similar problems than this with sshfs, in a OSMC (Debian derivative)
install:

https://discourse.osmc.tv/t/how-to-sshfs-tutorial/77852/11

There's an easy workaround for sshfs: the `delay_connect` option delays
network operations until the mountpoint is accessed. It would still be
nice to fix this for sshfs as it will bite novice users pretty badly.

All this leads me to believe this is a bug in systemd, and not NFS. I
wonder if we should reassign this bug to the systemd (or ifupdown?)
package to get more visibility into it (and keep nfs/sshfs as affected).

We'd need an easier way to reproduce this however. Has anyone worked on
getting some virtual machine images up to try and orchestrate a
reproducer for this? That would be ideal but a step-by-step set of
minimal instructions starting from a clean install would be an
acceptable compromise.

Once we have that, we can try to reproduce in buster as well and figure
out if the bug is actually fixed there or not.

Thanks for the bug report!

A.

-- 
Never be deceived that the rich will allow you to vote away their wealth.
                        - Lucy Parsons

[signature.asc (application/pgp-signature, inline)]

Message #35 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Gabriel Filion <gabster@lelutin.ca>

To: Antoine Beaupre <anarcat@debian.org>, Greg Wooledge <wooledg@eeg.ccf.org>, Vincent McIntyre <vincent.mcintyre@csiro.au>

Cc: 860264@bugs.debian.org

Subject: Re: similar bug with sshfs, maybe in systemd/ifupdown

Date: Tue, 29 Jan 2019 21:32:24 -0500

[Message part 1 (text/plain, inline)]

Hi there,

On 2019-01-29 9:26 p.m., Antoine Beaupre wrote:
> We'd need an easier way to reproduce this however. Has anyone worked on
> getting some virtual machine images up to try and orchestrate a
> reproducer for this? That would be ideal but a step-by-step set of
> minimal instructions starting from a clean install would be an
> acceptable compromise.

The bug is hard to reproduce since it's a run condition that might not
happen sometimes.

The simplest way to reproduce is to have an nfs (or maybe sshfs if it's
possible to reproduce with this) server, then on a buster machine to add
the nfs mount as a line to the fstab file, then reboot.

when the mount does not work, it is quite apparent: the boot process
hangs for some time before NFS decides to timeout.

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 860264@bugs.debian.org (full text, mbox, reply):

From: Antoine Beaupré <anarcat@debian.org>

To: Gabriel Filion <gabster@lelutin.ca>, Greg Wooledge <wooledg@eeg.ccf.org>, Vincent McIntyre <vincent.mcintyre@csiro.au>

Cc: 860264@bugs.debian.org

Subject: Re: similar bug with sshfs, maybe in systemd/ifupdown

Date: Tue, 29 Jan 2019 22:00:02 -0500

On 2019-01-29 21:32:24, Gabriel Filion wrote:
> Hi there,
>
> On 2019-01-29 9:26 p.m., Antoine Beaupre wrote:
>> We'd need an easier way to reproduce this however. Has anyone worked on
>> getting some virtual machine images up to try and orchestrate a
>> reproducer for this? That would be ideal but a step-by-step set of
>> minimal instructions starting from a clean install would be an
>> acceptable compromise.
>
> The bug is hard to reproduce since it's a run condition that might not
> happen sometimes.

It's pretty reliable on the vero 4k+, for what that's worth. But it's a
slower ARM machine...

> The simplest way to reproduce is to have an nfs (or maybe sshfs if it's
> possible to reproduce with this) server, then on a buster machine to add
> the nfs mount as a line to the fstab file, then reboot.
>
> when the mount does not work, it is quite apparent: the boot process
> hangs for some time before NFS decides to timeout.

understood. do you think it would be possible to setup a vagrant box
with this somehow?

a

-- 
From the age of uniformity, from the age of solitude, from the age of
Big Brother, from the age of doublethink - greetings!
                        - Winston Smith, 1984

Message #45 received at 860264-done@bugs.debian.org (full text, mbox, reply):

From: carnil@debian.org

To: 860264-done@bugs.debian.org

Cc: 860264-submitter@bugs.debian.org

Subject: Closing this bug (BTS maintenance for src:linux bugs)

Date: Sat, 14 Sep 2024 09:51:51 +0200 (CEST)

Hi

[This reply and bug closer is sent for doing BTS maintenance for
src:nfs-utils bugs]

This bug was reported against a very old nfs-utils version without much
followups/triaging itself.

If you can reproduce it with the current version in unstable/testing or
stable at least, please reopen the bug,
https://www.debian.org/Bugs/server-control for details.

Regards,
Salvatore

Send a report that this bug log contains spam.