Debian Bug report logs - #905894
nss-mdns: Forward resolution spends 10 seconds looking for .local SOA record

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Eduard Bloch <edi@gmx.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Hostname resolution getting stuck for many seconds

Date: Tue, 13 Mar 2018 20:52:28 +0100

Package: libnss-mdns
Version: 0.10-8
Severity: normal

Hi,

a few days ago I realized that something wrong is going on. The route
started freezing for no apparent reason, also ssh logins (which probably
include reverse hostname resolution) were stuck and even could timeout.

I figure that this might have to do something with unconventional
network interfaces being registered here.
Example with route:

Kernel-IP-Routentabelle
Ziel            Router          Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
<stuck for a while>
192.168.10.6    0.0.0.0         255.255.255.255 UH    0      0        0 tap4

I checked the process with strace and saw something weird. My DNS was
accessed, was queried, it returned data, then the connection was closed
and then it started doing something with your library and THAT is where
it didn't continue anymore. So I removed your lib and it's all fine
again.

Details:

recvfrom(4, "3W\205\3\0\1\0\0\0\1\0\0\0016\00210\003168\003192\7in-add"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [28->16]) = 93
close(4)                                = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=291488, ...}) = 0
mmap(NULL, 291488, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f59aec84000
close(4)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_mdns4.so.2", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\f\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st_mode=S_IFREG|0644, st_size=10160, ...}) = 0
mmap(NULL, 2105360, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f59ad724000
mprotect(0x7f59ad726000, 2093056, PROT_NONE) = 0
mmap(0x7f59ad925000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1000) = 0x7f59ad925000
close(4)                                = 0
mprotect(0x7f59ad925000, 4096, PROT_READ) = 0
munmap(0x7f59aec84000, 291488)          = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 4
fcntl(4, F_GETFD)                       = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
connect(4, {sa_family=AF_UNIX, sun_path="/var/run/avahi-daemon/socket"}, 110) = 0
fcntl(4, F_GETFL)                       = 0x2 (flags O_RDWR)
fstat(4, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
write(4, "RESOLVE-ADDRESS 192.168.10.6\n", 29) = 29
read(4, ^C0x557ea9f505b0, 4096)           = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.15.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

-- 
Eine Rechtsprechung, die an der grundsätzlichen Kriminalisierung von
Cannabis festhält, geht an medizinischen Erkenntnissen vorbei.
		-- Ellis Huber

Message #10 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Eduard Bloch <edi@gmx.de>, 892854@bugs.debian.org

Subject: Re: Bug#892854: Hostname resolution getting stuck for many seconds

Date: Tue, 13 Mar 2018 23:36:11 +0000

On Tue, 13 Mar 2018 at 20:52:28 +0100, Eduard Bloch wrote:
> I checked the process with strace and saw something weird. My DNS was
> accessed, was queried, it returned data, then the connection was closed
> and then it started doing something with your library and THAT is where
> it didn't continue anymore.

What's in the hosts: line in your /etc/nsswitch.conf?

New installations of nss-mdns will set it up with

    mdns4_minimal [NOTFOUND=return]

just before either resolve or dns, whichever is seen first. That means
only names in the .local domain are resolved this way. For example, on
a machine with systemd-resolved and various other non-standard modules,
I have:

hosts:          files mymachines gw_name myhostname mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns

However, we don't edit nsswitch.conf if there is already a mention
of mdns (because that way there'd be a risk of overwriting sysadmin
configuration or otherwise breaking stuff), so in particular old
installations that already had nss-mdns installed before 2013 can have
a no-longer-recommended configuration left over.

From your strace showing libnss_mdns4.so.2 being loaded, I think you
have mdns4 instead of mdns4_minimal. mdns{,4,6} tries to look up more
names in mDNS, and because of how mDNS works, will cause an arbitrary
delay for each name that cannot be resolved by any means (hosts file,
DNS, mDNS, whatever). I don't think there's much we can do about that
without making it impossible to choose the "non-minimal" behaviour.

If you purge and reinstall libnss-mdns, it should come back with a
better configuration that doesn't cause arbitrary delays (unless you
specifically ask to resolve a .local name that doesn't exist).

    smcv

Message #15 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Eduard Bloch <edi@gmx.de>

To: Simon McVittie <smcv@debian.org>

Cc: 892854@bugs.debian.org

Subject: Re: Bug#892854: Hostname resolution getting stuck for many seconds

Date: Wed, 14 Mar 2018 20:01:07 +0100

Hallo,
* Simon McVittie [Tue, Mar 13 2018, 11:36:11PM]:
> On Tue, 13 Mar 2018 at 20:52:28 +0100, Eduard Bloch wrote:
> > I checked the process with strace and saw something weird. My DNS was
> > accessed, was queried, it returned data, then the connection was closed
> > and then it started doing something with your library and THAT is where
> > it didn't continue anymore.
> 
> What's in the hosts: line in your /etc/nsswitch.conf?

hosts:          files myhostname mdns4_minimal [NOTFOUND=return] dns mdns4

> New installations of nss-mdns will set it up with
> 
>     mdns4_minimal [NOTFOUND=return]
> 
> just before either resolve or dns, whichever is seen first. That means
> only names in the .local domain are resolved this way. For example, on
> a machine with systemd-resolved and various other non-standard modules,
> I have:
> 
> hosts:          files mymachines gw_name myhostname mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns
> 
> However, we don't edit nsswitch.conf if there is already a mention
> of mdns (because that way there'd be a risk of overwriting sysadmin
> configuration or otherwise breaking stuff), so in particular old
> installations that already had nss-mdns installed before 2013 can have
> a no-longer-recommended configuration left over.
> From your strace showing libnss_mdns4.so.2 being loaded, I think you
> have mdns4 instead of mdns4_minimal. mdns{,4,6} tries to look up more
> names in mDNS, and because of how mDNS works, will cause an arbitrary
> delay for each name that cannot be resolved by any means (hosts file,
> DNS, mDNS, whatever). I don't think there's much we can do about that
> without making it impossible to choose the "non-minimal" behaviour.

ETOOMUCHINFORMATION

I cannot remember having touched this file ever on this system. And I
guess that if you have a record of all versions installed in the past
then it should be easy to detect a pristine copy of that file and
update/replace it as needed.

$ md5sum /etc/nsswitch.conf 
60686f928ec8782f409b742e49314427  /etc/nsswitch.conf

> If you purge and reinstall libnss-mdns, it should come back with a
> better configuration that doesn't cause arbitrary delays (unless you
> specifically ask to resolve a .local name that doesn't exist).

Ok, I did just that (had to purge another multiarch version as well) and
now it works, just as you described.

Still, I think this problem deserves a better solution. Because it's not
really obvious for the user what is going on in case of such trouble.

Best regards,
Eduard.

Message #20 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Eduard Bloch <edi@gmx.de>

Cc: 892854@bugs.debian.org

Subject: Re: Bug#892854: Hostname resolution getting stuck for many seconds

Date: Wed, 14 Mar 2018 19:50:00 +0000

On Wed, 14 Mar 2018 at 20:01:07 +0100, Eduard Bloch wrote:
> I cannot remember having touched [nsswitch.conf] on this system. And I
> guess that if you have a record of all versions installed in the past
> then it should be easy to detect a pristine copy of that file and
> update/replace it as needed.

Unfortunately, no, we can't. Each line of nsswitch.conf is collaboratively
maintained by glibc and every relevant nsswitch plugin (libnss-*)
by editing it in-place. So the expected contents of the hosts line
depend which other nsswitch hosts plugins you have, and there is no
single history.

In a more opinionated distribution, the distro's core maintainers might
maintain a single nsswitch.conf containing all the supported plugins in
some reasonable order; but this is Debian, where all configurations are
considered valid, even the ones that make little sense.

> Still, I think this problem deserves a better solution. Because it's not
> really obvious for the user what is going on in case of such trouble.

I'm open to suggestions for how we can improve things for pre-2013
installations, within the constraints that I can't change the past,
and I don't want to violate Policy by overriding sysadmin configuration.

Post-2013 installations should be OK already (the purge/reinstall cycle
that I suggested gives you the equivalent of a post-2013 installation).

    smcv

Message #25 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Matthijs van Duin <matthijsvanduin@gmail.com>

To: 892854@bugs.debian.org

Subject: Re: Hostname resolution getting stuck for many seconds

Date: Mon, 6 Aug 2018 13:41:27 +0200

[Message part 1 (text/plain, inline)]

This problem seems to be cause by the local_soa() call in src/util.c that I
presume was introduced as part of this "feature" listed in NEWS:
* nss-mdns now implements [standard
  heuristics](https://support.apple.com/en-us/HT201275) for
  detecting `.local` unicast resolution and will automatically
  disable resolution when a local server responds to `.local` requests

The 10-second delay caused by this detection is extremely annoying and
renders libnss-mdns effectively unusable for me. Please offer a way to
disable this!

[Message part 2 (text/html, inline)]

Message #30 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Matthijs van Duin <matthijsvanduin@gmail.com>

To: 892854@bugs.debian.org

Cc: Eduard Bloch <edi@gmx.de>, Simon McVittie <smcv@debian.org>

Subject: Re: Hostname resolution getting stuck for many seconds

Date: Mon, 6 Aug 2018 14:05:02 +0200

The following patch fixes the problem for me:

diff --git c/src/util.c w/src/util.c
index aaebd62..594ed42 100644
--- c/src/util.c
+++ w/src/util.c
@@ -62,7 +62,7 @@ int verify_name_allowed_with_soa(const char* name, FILE* mdns_allow_file) {
     case VERIFY_NAME_RESULT_ALLOWED:
         return 1;
     case VERIFY_NAME_RESULT_ALLOWED_IF_NO_LOCAL_SOA:
-        return !local_soa();
+        return 1;
     default:
         return 0;
     }

Message #35 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Matthijs van Duin <matthijsvanduin@gmail.com>

Cc: 892854@bugs.debian.org, Eduard Bloch <edi@gmx.de>

Subject: Re: Hostname resolution getting stuck for many seconds

Date: Fri, 10 Aug 2018 17:50:06 +0100

On Mon, 06 Aug 2018 at 13:41:27 +0200, Matthijs van Duin wrote:
> This problem seems to be cause by the local_soa() call in src/util.c

Which nss-mdns version are you using?

What's in the hosts: line in your /etc/nsswitch.conf?

I suspect that you and Eduard might be experiencing different issues
that have the same high-level symptom (delays in name resolution).
If that's the case, then I'll need to clone this bug for the two separate
issues. I'll also retitle the original bug to make it clearer that
it's about a specific source of delays.

On Mon, 06 Aug 2018 at 14:05:02 +0200, Matthijs van Duin wrote:
> The following patch fixes the problem for me:
> 
> diff --git c/src/util.c w/src/util.c
> index aaebd62..594ed42 100644
> --- c/src/util.c
> +++ w/src/util.c
> @@ -62,7 +62,7 @@ int verify_name_allowed_with_soa(const char* name, FILE* mdns_allow_file) {
>      case VERIFY_NAME_RESULT_ALLOWED:
>          return 1;
>      case VERIFY_NAME_RESULT_ALLOWED_IF_NO_LOCAL_SOA:
> -        return !local_soa();
> +        return 1;
>      default:
>          return 0;
>      }

This effectively disables the code that tries to avoid interfering with
a .local domain if one exists in normal (non-multicast) DNS, so it's
likely to solve problems for some people but cause problems for others.

If you try to resolve a SOA record through your normal DNS resolver,
does that hang for a long time? These commands' output would be useful
information:

time host -t SOA debian.org.
time host -t SOA local.

If you happen to have macOS or Windows on the same LAN, it would also
be interesting to know whether they suffer similar delays for mDNS
resolution: the code disabled by the patch above is a standard heuristic
used in multiple (perhaps all) mDNS implementations, for example
<https://support.apple.com/en-us/HT201275> in macOS.

Thanks,
    smcv

Message #40 received at 892854@bugs.debian.org (full text, mbox, reply):

From: Matthijs van Duin <matthijsvanduin@gmail.com>

To: Simon McVittie <smcv@debian.org>

Cc: 892854@bugs.debian.org, Eduard Bloch <edi@gmx.de>

Subject: Re: Hostname resolution getting stuck for many seconds

Date: Sat, 11 Aug 2018 04:35:25 +0200

On 10 August 2018 at 18:50, Simon McVittie <smcv@debian.org> wrote:
> Which nss-mdns version are you using?

0.14.1-1

> What's in the hosts: line in your /etc/nsswitch.conf?

I currently have:
hosts:          files mymachines mdns_minimal resolve [!UNAVAIL=return] dns myhostname

but I also tried some variations. The delay simply occurs once it hits
the mdns_minimal module.

> I suspect that you and Eduard might be experiencing different issues
> that have the same high-level symptom (delays in name resolution).

Oops, I think you're right.  While quickly scanning through the bug
report it seemed to match, but I somehow managed to overlook that it
was resolved for the original poster, I thought the issue was still
open for him.  My sincere apologies!


> This effectively disables the code that tries to avoid interfering with
> a .local domain if one exists in normal (non-multicast) DNS, so it's
> likely to solve problems for some people but cause problems for others.

Yes I didn't mean to imply this is a valid solution, it simply
confirms that this particular code is indeed responsible for the delay
for me (and provides a workaround for me, since I personally don't
care about this feature).


> time host -t SOA debian.org.
> time host -t SOA local.

The first works, the second hangs for 10 seconds until
;; connection timed out; no servers could be reached

However, I'm using systemd-resolved and /etc/resolv.conf points to its
stub resolver.  If I ask the upstream dns server instead, I do get a
prompt answer.  So, it would seem that the bug actually lies in
systemd-resolved.

Matthijs

Message #53 received at 905894@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Matthijs van Duin <matthijsvanduin@gmail.com>

Cc: 905894@bugs.debian.org, systemd@packages.debian.org

Subject: Re: Bug #905894: nss-mdns: Forward resolution takes 10 seconds looking for SOA record

Date: Sat, 11 Aug 2018 11:40:49 +0100

On Sat, 11 Aug 2018 at 04:35:25 +0200, Matthijs van Duin wrote:
> > I suspect that you and Eduard might be experiencing different issues
> > that have the same high-level symptom
> 
> Oops, I think you're right.

I've cloned the bug and retitled both clones. Please send any further
correspondence about this issue to #905894, reserving #892854 for the
mdns{,4,6} module's effect on reverse resolution.

> > time host -t SOA debian.org.
> > time host -t SOA local.
> 
> The first works, the second hangs for 10 seconds until
> ;; connection timed out; no servers could be reached
> 
> However, I'm using systemd-resolved and /etc/resolv.conf points to its
> stub resolver.  If I ask the upstream dns server instead, I do get a
> prompt answer.  So, it would seem that the bug actually lies in
> systemd-resolved.

That sounds plausible. What versions of systemd and libnss-resolve do
you have?

Context for systemd maintainers: libnss-mdns contains a heuristic borrowed
from macOS (the de facto reference implementation of mDNS) which tries
to avoid interfering with a legacy unicast .local domain (which used to
be a popular choice for LANs that used an unregistered TLD) by disabling
itself if it detects unicast .local. It does this by resolving the SOA
record for local.; if it gets an answer, then there is a unicast .local,
so .local is not available for its RFC use as the mDNS domain. However,
systemd-resolved's stub resolver seems to cause a 10 second delay before
failing to resolve .local in this case.

I wonder whether this is systemd-resolved's support for LLMNR and mDNS
getting in the way? It should probably only try to resolve SOA records
via traditional unicast DNS, since a SOA record would be meaningless in
LLMNR or mDNS.

> I currently have:
> hosts:          files mymachines mdns_minimal resolve [!UNAVAIL=return] dns myhostname

FYI, the recommendation is for the mdns-related entry to be

    ... mdns4_minimal [NOTFOUND=return] ...

which makes mdns4_minimal authoritative for .local (missing .local names
are not looked up in systemd-resolved or DNS). Replace mdns4_minimal
with mdns_minimal if IPv6 .local is more important to you than quick
name resolution in legacy programs that call getaddrinfo() with AF_INET6
followed by getaddrinfo() with AF_INET (programs that correctly make a
single getaddrinfo() call with AF_UNSPEC are equally fast either way).

However, changing that entry to the recommended one is not going to
help you here.

    smcv

Send a report that this bug log contains spam.