Debian Bug report logs - #773525
Randomly excludes available connections [when there are too many?]

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Pietro Battiston <me@pietrobattiston.it>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Randomly excludes available connections [when there are too many?]

Date: Fri, 19 Dec 2014 15:32:08 +0100

Package: network-manager
Version: 0.9.10.0-4
Severity: grave

Copypasted from a shell:

pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | wc; done
    127     637   12827
    127     637   12827
    127     627   12827
    126     628   12726
    127     629   12573
    127     634   12828
    127     629   12827
    127     630   12319
    127     631   12827
    127     627   12827


I am clearly not changing my list of available connections (so quick!). So what
is happening is that network-manager is dropping some of my registered
connections, in a random way. Initially I though "it is unable to handle more
than 127", but then I saw that sometimes it only lists 126. The output of
"nmcli c" is otherwise almost sane (see below).

Some connections show up more frequently, some less. Consider for instance:

pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | grep -i condividi; done |
sort
Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
802-3-ethernet   --
Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
802-3-ethernet   --
Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
802-3-ethernet   --
Condividi                             cab18fac-376f-42cf-9349-d9b4170dacce
802-3-ethernet   --
condividi con dnsmasq                 d54c2a91-f654-4d52-bdbd-cf9d8efdd9e5
802-3-ethernet   --
condividi                              daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --
condividi                           daa501b5-87eb-4a3f-a6d5-4fa73d042a66
802-3-ethernet   --

So
- one connection, "Condividi", is shown 3 times out of 10
- one connection, "condividi con dnsmasq", is shown 1 time only
- one connection, "condividi", is shown most of the times... with different
space paddings!!!!

Different runs of the above command will clearly give different results, but
some connections are indeed more rare - i.e. "condividi con dnsmasq" usually
shows in none of the 10 tries - and some exhibit (even more than 2) different
space paddings. The wireless connection I need to use at work shows up more or
less once every 100 tries. Which, by the way, is very annoying: it usually
disables autoconnect, makes it virtually impossible to use gnome-control-center
(or the GNOME 3 drop-down menu) to connect, and makes it very hard and time
consuming also with "nmcli c up" (nm-connection-editor is also affected).

This is why the bug is grave: in particular, if a remote server relies on
network-manager in order to connect at startup... you'd better hope it is not
too remote.

The bug was present also with 0.9.10.0-3. My rough guess is that the
problematic upgrade was located between July and September.

P.S: just for info: _most_ of the connections are dropped:
pietro@debiousci:~$ ls /etc/NetworkManager/system-connections | wc
    431     891    8594



-- System Information:
Debian Release: 8.0
  APT prefers testing
  APT policy: (650, 'testing'), (600, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages network-manager depends on:
ii  adduser                3.113+nmu3
ii  dbus                   1.8.12-1
ii  init-system-helpers    1.22
ii  isc-dhcp-client        4.3.1-5
ii  libc6                  2.19-13
ii  libdbus-1-3            1.8.12-1
ii  libdbus-glib-1-2       0.102-1
ii  libgcrypt20            1.6.2-4+b1
ii  libglib2.0-0           2.42.1-1
ii  libgnutls-deb0-28      3.3.8-5
ii  libgudev-1.0-0         215-7
ii  libmm-glib0            1.4.0-1
ii  libndp0                1.4-2
ii  libnewt0.52            0.52.17-1+b1
ii  libnl-3-200            3.2.24-2
ii  libnl-genl-3-200       3.2.24-2
ii  libnl-route-3-200      3.2.24-2
ii  libnm-glib4            0.9.10.0-3
ii  libnm-util2            0.9.10.0-3
ii  libpam-systemd         215-7
ii  libpolkit-gobject-1-0  0.105-8
ii  libreadline6           6.3-8+b1
ii  libsoup2.4-1           2.48.0-1
ii  libsystemd0            215-7
ii  libteamdctl0           1.12-1
ii  libuuid1               2.25.2-3
ii  lsb-base               4.1+Debian13+nmu1
ii  policykit-1            0.105-8
ii  udev                   215-7
ii  wpasupplicant          2.3-1

Versions of packages network-manager recommends:
ii  crda            3.13-1
ii  dnsmasq-base    2.72-2
ii  iptables        1.4.21-2+b1
ii  iputils-arping  3:20121221-5+b2
ii  modemmanager    1.4.0-1
ii  ppp             2.4.6-3

Versions of packages network-manager suggests:
ii  avahi-autoipd  0.6.31-4+b1
pn  libteam-utils  <none>

-- no debconf information

Message #10 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org

Subject: Re: [Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 19 Dec 2014 15:45:31 +0100

[Message part 1 (text/plain, inline)]

Control: severity -1 important
Control: tags -1 moreinfo

Am 19.12.2014 um 15:32 schrieb Pietro Battiston:
> Package: network-manager
> Version: 0.9.10.0-4
> Severity: grave
> 
> Copypasted from a shell:
> 
> pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | wc; done
>     127     637   12827
>     127     637   12827
>     127     627   12827
>     126     628   12726
>     127     629   12573
>     127     634   12828
>     127     629   12827
>     127     630   12319
>     127     631   12827
>     127     627   12827
> 
> 
> I am clearly not changing my list of available connections (so quick!). So what
> is happening is that network-manager is dropping some of my registered
> connections, in a random way. Initially I though "it is unable to handle more
> than 127", but then I saw that sometimes it only lists 126. The output of
> "nmcli c" is otherwise almost sane (see below).
> 
> Some connections show up more frequently, some less. Consider for instance:
> 
> pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | grep -i condividi; done |
> sort
> Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
> 802-3-ethernet   --
> Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
> 802-3-ethernet   --
> Condividi                             8a006f93-95ac-4683-a683-56413cbb95bb
> 802-3-ethernet   --
> Condividi                             cab18fac-376f-42cf-9349-d9b4170dacce
> 802-3-ethernet   --
> condividi con dnsmasq                 d54c2a91-f654-4d52-bdbd-cf9d8efdd9e5
> 802-3-ethernet   --
> condividi                              daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                             daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> condividi                           daa501b5-87eb-4a3f-a6d5-4fa73d042a66
> 802-3-ethernet   --
> 
> So
> - one connection, "Condividi", is shown 3 times out of 10
> - one connection, "condividi con dnsmasq", is shown 1 time only
> - one connection, "condividi", is shown most of the times... with different
> space paddings!!!!
> 
> Different runs of the above command will clearly give different results, but
> some connections are indeed more rare - i.e. "condividi con dnsmasq" usually
> shows in none of the 10 tries - and some exhibit (even more than 2) different
> space paddings. The wireless connection I need to use at work shows up more or
> less once every 100 tries. Which, by the way, is very annoying: it usually
> disables autoconnect, makes it virtually impossible to use gnome-control-center
> (or the GNOME 3 drop-down menu) to connect, and makes it very hard and time
> consuming also with "nmcli c up" (nm-connection-editor is also affected).
> 
> This is why the bug is grave: in particular, if a remote server relies on
> network-manager in order to connect at startup... you'd better hope it is not
> too remote.

Not really, grave means, the package is unusable, which it clearly i'snt.

> The bug was present also with 0.9.10.0-3. My rough guess is that the
> problematic upgrade was located between July and September.
> 

Can you pinpoint the exact version please.


> pietro@debiousci:~$ ls /etc/NetworkManager/system-connections | wc
>     431     891    8594

How did you generate 431 connections? Did you copy files around?


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #19 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Pietro Battiston <me@pietrobattiston.it>

To: Michael Biebl <biebl@debian.org>

Cc: 773525@bugs.debian.org

Subject: Re: [Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 19 Dec 2014 16:15:54 +0100

Il giorno ven, 19/12/2014 alle 15.45 +0100, Michael Biebl ha scritto:
> [...]
> Am 19.12.2014 um 15:32 schrieb Pietro Battiston:
> [...]
> > This is why the bug is grave: in particular, if a remote server relies on
> > network-manager in order to connect at startup... you'd better hope it is not
> > too remote.
> 
> Not really, grave means, the package is unusable, which it clearly i'snt.
> 

It is hardly usable for me - it would be totally unusable for me if at
work I did not (usually) have also a cable connection, which usually
works (or maybe I just created enough copies of it? don't know...).

Except that your question below seems to imply nobody except me has many
(> 127, presumably?) connections. So, OK, if it's just me I understand.
My best guess is that computers on which many connections got remembered
along the years are statistically not the same ones which are updated to
testing, but I have no data to test this hypothesis.


> > The bug was present also with 0.9.10.0-3. My rough guess is that the
> > problematic upgrade was located between July and September.
> > 
> 
> Can you pinpoint the exact version please.
> 

(I'm not forgetting your request, but I don't think I will manage to do
it today)


> > pietro@debiousci:~$ ls /etc/NetworkManager/system-connections | wc
> >     431     891    8594
> 
> How did you generate 431 connections? Did you copy files around?
> 

Yeah, I enjoy copying files around!

No.

Looking at them, I certainly have some doubles, i.e. because one
connection did not show up so I recreated it thinking it had completely
disappeared. But mostly (>300) they are "real" connections. The oldest
ones are from 2011, which means ~ 1.5 per week... given that I travel
quite a bit, I hack quite a bit, and when I am without Internet and see
some open wireless I often try to connect, this is not a particularly
strange number.

Pietro

Message #24 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Pietro Battiston <me@pietrobattiston.it>

Cc: 773525@bugs.debian.org

Subject: Re: [Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 19 Dec 2014 16:24:53 +0100

[Message part 1 (text/plain, inline)]

Am 19.12.2014 um 16:15 schrieb Pietro Battiston:
> Il giorno ven, 19/12/2014 alle 15.45 +0100, Michael Biebl ha scritto:
>> Am 19.12.2014 um 15:32 schrieb Pietro Battiston:

>>> pietro@debiousci:~$ ls /etc/NetworkManager/system-connections | wc
>>>     431     891    8594
>>
>> How did you generate 431 connections? Did you copy files around?
>>
> 
> Yeah, I enjoy copying files around!
> 
> No.
> 
> Looking at them, I certainly have some doubles, i.e. because one
> connection did not show up so I recreated it thinking it had completely
> disappeared. 

The reason I was asking is, that by manually copying and creating files
in /etc/NetworkManager/system-connections, you might end up with
connection files, with non-unique UUIDs, not being valid/properly
formatted etc. which *might* confuse NetworkManager.

It might probably also be useful to get a debug log from NetworkManager. See
https://wiki.gnome.org/Projects/NetworkManager/Debugging


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #35 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Simon McVittie <smcv@debian.org>, Dan Williams <dcbw@redhat.com>, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: [Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Thu, 16 Jul 2015 19:16:20 +0200

[Message part 1 (text/plain, inline)]

Am 19.12.2014 um 15:32 schrieb Pietro Battiston:
> Package: network-manager
> Version: 0.9.10.0-4
> Severity: grave
> 
> Copypasted from a shell:
> 
> pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | wc; done
>     127     637   12827
>     127     637   12827
>     127     627   12827
>     126     628   12726
>     127     629   12573
>     127     634   12828
>     127     629   12827
>     127     630   12319
>     127     631   12827
>     127     627   12827
> 
> 
> I am clearly not changing my list of available connections (so quick!). So what
> is happening is that network-manager is dropping some of my registered
> connections, in a random way. Initially I though "it is unable to handle more
> than 127", but then I saw that sometimes it only lists 126. The output of
> "nmcli c" is otherwise almost sane (see below).

Seeing the latest upstream release, it noticed the following commit [1]:

> dbus: increase 'max_replies_per_connection' limit in D-Bus configuration
> D-Bus default limit of replies per connection has been lowered to 128 due to
> CVE-2014-3638, see:
> http://cgit.freedesktop.org/dbus/dbus/commit/?id=5bc7f9519ebc6117ba300c704794b36b87c2194b
> https://bugs.freedesktop.org/show_bug.cgi?id=81053
> 
> The limit seems to be too low and causes problems in libnm-glib, that will not
> return all NetworkManager connection profiles if there are too many of them
> (roughly more than the limit). As a consequence, libnm-glib based clients will
> not work properly.

This looks like it could be the reason for the problem you are seeing.

That said, I'm not sure if individual daemon packages overriding the
dbus policy in that regard is a good idea (and a proper fix).

I've CCed the upstream maintainers of NM and dbus. Maybe we can find a
solution for this.

Michael


[1]
http://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=2c299ba65c51e9c407090dc83929d692c74ee3f2
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Message #40 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Dan Williams <dcbw@redhat.com>

To: Michael Biebl <biebl@debian.org>

Cc: Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Simon McVittie <smcv@debian.org>, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: [Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Thu, 16 Jul 2015 13:56:03 -0500

On Thu, 2015-07-16 at 19:16 +0200, Michael Biebl wrote:
> Am 19.12.2014 um 15:32 schrieb Pietro Battiston:
> > Package: network-manager
> > Version: 0.9.10.0-4
> > Severity: grave
> > 
> > Copypasted from a shell:
> > 
> > pietro@debiousci:~$ for i in `seq 1 10`; do nmcli c | wc; done
> >     127     637   12827
> >     127     637   12827
> >     127     627   12827
> >     126     628   12726
> >     127     629   12573
> >     127     634   12828
> >     127     629   12827
> >     127     630   12319
> >     127     631   12827
> >     127     627   12827
> > 
> > 
> > I am clearly not changing my list of available connections (so quick!). So what
> > is happening is that network-manager is dropping some of my registered
> > connections, in a random way. Initially I though "it is unable to handle more
> > than 127", but then I saw that sometimes it only lists 126. The output of
> > "nmcli c" is otherwise almost sane (see below).
> 
> Seeing the latest upstream release, it noticed the following commit [1]:
> 
> > dbus: increase 'max_replies_per_connection' limit in D-Bus configuration
> > D-Bus default limit of replies per connection has been lowered to 128 due to
> > CVE-2014-3638, see:
> > http://cgit.freedesktop.org/dbus/dbus/commit/?id=5bc7f9519ebc6117ba300c704794b36b87c2194b
> > https://bugs.freedesktop.org/show_bug.cgi?id=81053
> > 
> > The limit seems to be too low and causes problems in libnm-glib, that will not
> > return all NetworkManager connection profiles if there are too many of them
> > (roughly more than the limit). As a consequence, libnm-glib based clients will
> > not work properly.
> 
> This looks like it could be the reason for the problem you are seeing.
> 
> That said, I'm not sure if individual daemon packages overriding the
> dbus policy in that regard is a good idea (and a proper fix).
> 
> I've CCed the upstream maintainers of NM and dbus. Maybe we can find a
> solution for this.

There really isn't any solution that I can think of, except serializing
the requests in the client libraries.  Unfortunately, that's not a great
way to go about it and it simply complicates the code on the client
side.  For example, say you have perhaps 40 saved connections and you're
in a large city and so can see 30 or 40 access points.  When some client
starts up, it may request all of these initially, and also request
properties on the objects when some of the initial replies come back.
Obviously all of this happens asynchronously because blocking in a UI is
bad.  It's quite easy to run over 128 pending replies.

Attempting to serialize these in libnm-glib (or libnm for that matter)
would likely result in a bunch of code that's just a wrapper over
dbus-glib and one more layer of suck-itude.

We do have the 'libnm' library with NM 1.0+ that uses GDBus all the way
through, so if GDBus somehow manages to avoid this problem then great.
Otherwise, we'll have the same problem in libnm too...

Simon, any thoughts here?

Dan

Message #45 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Dan Williams <dcbw@redhat.com>, Michael Biebl <biebl@debian.org>

Cc: Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Thu, 16 Jul 2015 23:56:32 +0100

On 16/07/15 19:56, Dan Williams wrote:
> There really isn't any solution that I can think of, except serializing
> the requests in the client libraries.  Unfortunately, that's not a great
> way to go about it and it simply complicates the code on the client
> side.

In the medium to long term I think the only way is to have some sort of
queue of requests, or improve NM's D-Bus API so that things can be
batched (for instance getting properties of more than one AP per
round-trip, which would also make it faster).

> It's quite easy to run over 128 pending replies.

How many do you need? Is the answer in fact "arbitrarily many"?

With some appropriate benchmarks we might be able to increase the limit
by an order of magnitude or two, but I'm concerned that going back to
the 1K that NetworkManager historically used might be too many.

The problem is that if you have an unbounded number of requests
in-flight, the system dbus-daemon uses an unbounded amount of memory to
track them; and the system dbus-daemon is a shared resource acting on
behalf of various trusted and untrusted processes, so that would be bad.
The reasons it tracks them at all are so that unsolicited "replies" can
be rejected; so that if a client can call a method on a service, the
service allowed to reply; and so that if a service falls off the bus
without replying to all method calls it should have handled, dbus-daemon
can synthesize error replies immediately, rather than leaving the
clients to time out.

At the moment dbus-daemon also doesn't use clever enough data
structures, resulting in a CPU-consumption denial of service attack:
with a lot of pending replies and a lot of connections allowed, an
attacker can make dbus-daemon use a lot of CPU time. We dropped the
pending reply limit from 8K to 128 because there was a practical
denial-of-service attack with 8K (CVE-2014-3638,
https://bugs.freedesktop.org/show_bug.cgi?id=81053): simple method calls
could be made to take more than 25 seconds (the default timeout).

https://bugs.freedesktop.org/show_bug.cgi?id=83938 has some attempts at
better data structures, but it's significant code churn, so we're
unlikely to land anything like that in a dbus stable-branch. I'd be
happy to review a cleaned up implementation, but that isn't going to
help for distributions that aren't tracking bleeding-edge dbus.

kdbus' hard-coded limit was also 128 pending replies per sender, last
time I looked (although I think it was just coincidence that Alban
suggested the same number after experimenting with denial of service
attacks). It needs to track requested replies for basically the same
reasons as dbus-daemon, and also has to be fairly conservative with its
limits because it's allocating kernel memory. So moving to kdbus is
probably not going to save us from this.

> We do have the 'libnm' library with NM 1.0+ that uses GDBus all the way
> through, so if GDBus somehow manages to avoid this problem then great.
> Otherwise, we'll have the same problem in libnm too...

It sounds as though this is really an issue with the high-level design
of NM's D-Bus API, rather than the specifics of how the client library
is implemented, so I don't think GDBus is going to help.

    S

Message #50 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Dan Williams <dcbw@redhat.com>

To: Simon McVittie <smcv@debian.org>

Cc: Michael Biebl <biebl@debian.org>, Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Thu, 16 Jul 2015 18:13:51 -0500

On Thu, 2015-07-16 at 23:56 +0100, Simon McVittie wrote:
> On 16/07/15 19:56, Dan Williams wrote:
> > There really isn't any solution that I can think of, except serializing
> > the requests in the client libraries.  Unfortunately, that's not a great
> > way to go about it and it simply complicates the code on the client
> > side.
> 
> In the medium to long term I think the only way is to have some sort of
> queue of requests, or improve NM's D-Bus API so that things can be
> batched (for instance getting properties of more than one AP per
> round-trip, which would also make it faster).
> 
> > It's quite easy to run over 128 pending replies.
> 
> How many do you need? Is the answer in fact "arbitrarily many"?
> 
> With some appropriate benchmarks we might be able to increase the limit
> by an order of magnitude or two, but I'm concerned that going back to
> the 1K that NetworkManager historically used might be too many.
> 
> The problem is that if you have an unbounded number of requests
> in-flight, the system dbus-daemon uses an unbounded amount of memory to
> track them; and the system dbus-daemon is a shared resource acting on
> behalf of various trusted and untrusted processes, so that would be bad.
> The reasons it tracks them at all are so that unsolicited "replies" can
> be rejected; so that if a client can call a method on a service, the
> service allowed to reply; and so that if a service falls off the bus
> without replying to all method calls it should have handled, dbus-daemon
> can synthesize error replies immediately, rather than leaving the
> clients to time out.
> 
> At the moment dbus-daemon also doesn't use clever enough data
> structures, resulting in a CPU-consumption denial of service attack:
> with a lot of pending replies and a lot of connections allowed, an
> attacker can make dbus-daemon use a lot of CPU time. We dropped the
> pending reply limit from 8K to 128 because there was a practical
> denial-of-service attack with 8K (CVE-2014-3638,
> https://bugs.freedesktop.org/show_bug.cgi?id=81053): simple method calls
> could be made to take more than 25 seconds (the default timeout).
> 
> https://bugs.freedesktop.org/show_bug.cgi?id=83938 has some attempts at
> better data structures, but it's significant code churn, so we're
> unlikely to land anything like that in a dbus stable-branch. I'd be
> happy to review a cleaned up implementation, but that isn't going to
> help for distributions that aren't tracking bleeding-edge dbus.
> 
> kdbus' hard-coded limit was also 128 pending replies per sender, last
> time I looked (although I think it was just coincidence that Alban
> suggested the same number after experimenting with denial of service
> attacks). It needs to track requested replies for basically the same
> reasons as dbus-daemon, and also has to be fairly conservative with its
> limits because it's allocating kernel memory. So moving to kdbus is
> probably not going to save us from this.
> 
> > We do have the 'libnm' library with NM 1.0+ that uses GDBus all the way
> > through, so if GDBus somehow manages to avoid this problem then great.
> > Otherwise, we'll have the same problem in libnm too...
> 
> It sounds as though this is really an issue with the high-level design
> of NM's D-Bus API, rather than the specifics of how the client library
> is implemented, so I don't think GDBus is going to help.

I'd take issue with that assertion actually.  The object model of NM is
such that the number of objects can be determined by either (a)
configuration, eg number of saved network connections or (b) external
network properties like number of access points, etc.  I don't think
that's really unreasonable, especially given how D-Bus wants objects +
properties + interfaces.  I'd rather not go all libpurple-style and
stuff all properties into one interface so we can save 2 GetAll() calls
per object.

eg, think of it as each contact in Telepathy being an object, and I can
easily see somebody having 100+ contacts/friends etc.  Each of those
contacts has properties to be retrieved too.  So would a client just try
to get 50 of those at a time, wait for the reply, then get the next 50?
And then fire off GetAll() on each of those in 50-item batches?  And
then for multiple interfaces?

Also, how is a client supposed to know how big the limit is?  Or is a
client just supposed to use a sufficiently low number that it is known
to work in all cases?

The solution here is likely to transition the libnm implementation over
to the ObjectManager interface's GetManagedObjects() method for the
initial setup, to get everything in one call.  A lot of data, but likely
faster than doing it piece-by-piece.  That's only going to happen with
NM 1.2 and later though when the daemon is converted to GDBus.  Which
might well leave a lot of people out in the cold if they start using
kdbus with Debian Jessie or something where NM is still at 0.9.10.  Any
chance we could get dbus-glib to implement GetManagedObjects()?

Dan

Message #55 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Dan Williams <dcbw@redhat.com>

Cc: Michael Biebl <biebl@debian.org>, Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 17 Jul 2015 00:35:13 +0100

On 17/07/15 00:13, Dan Williams wrote:
> eg, think of it as each contact in Telepathy being an object

Telepathy contacts aren't "real D-Bus objects", for precisely this reason.

Telepathy does a lot of batching information into one method call; some
of it is basically a precursor of the ObjectManager interface, and some
of it (contacts and their attributes) is more specialized. The initial
motivating use-case was that batching the "get contact's name" method
call into (effectively) "get n contacts' names" made it orders of
magnitude faster to join a busy IRC channel like #ubuntu. Recent-ish
Telepathy can also batch arbitrarily many attributes of contacts into
one method call, with a filtering mechanism so you don't have to get the
attributes you don't care about.

If Telepathy didn't already exist and I had to invent it now, I'd mostly
be using ObjectManager. Contacts would probably be strings, because
practicality beats purity, and some uses of Telepathy (like joining
#ubuntu) involve a silly number of contacts; or they might be
object-paths that in practice you mainly use as opaque tokens in
array-based APIs, and rarely or never actually call methods on.

> Also, how is a client supposed to know how big the limit is?  Or is a
> client just supposed to use a sufficiently low number that it is known
> to work in all cases?

My usual rule of thumb is that if you have a finite number of orthogonal
method calls ("get name", "get icon", "get hat colour") you can probably
get away with parallelizing them, but if you have an potentially
unbounded number of method calls ("get Dan's icon", "get Michael's
icon", ...) you should delay each one until the previous one finished.

Note that the system bus has a much lower limit than the session bus,
because if you carry out a DoS attack on the session bus you're only
hurting yourself, whereas a DoS attack on the system bus hurts system
services and other users. (Not true in kdbus-land, though, because
kernel resources; I think kdbus imposes the same lower limit on user buses.)

> The solution here is likely to transition the libnm implementation over
> to the ObjectManager interface's GetManagedObjects() method for the
> initial setup, to get everything in one call.  A lot of data, but likely
> faster than doing it piece-by-piece.

That's what I'd recommend; ObjectManager is precisely for situations
where clients are interested in "most" children of a parent object.

> Any
> chance we could get dbus-glib to implement GetManagedObjects()?

It is entirely possible to "roll your own" GetManagedObjects; indeed,
Telepathy reimplements the entire service-side Properties interface
(because dbus-glib was historically even worse than it is now).

I'm a little reluctant to add features to dbus-glib, because it's a dead
end; but if it helps NM, I'd make an exception for reviewing
ObjectManager support.

    S

Message #60 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Pietro Battiston <me@pietrobattiston.it>

To: Simon McVittie <smcv@debian.org>, Dan Williams <dcbw@redhat.com>

Cc: Michael Biebl <biebl@debian.org>, 773525@bugs.debian.org, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 17 Jul 2015 08:53:09 +0200

Sorry if this is trivial to all the participants in the discussion - it
was not stated clearly so I thought it may be useful information.

"nmcli c" (1.0.2-2) currently appears to be fixed, in the sense that it
consistently outputs the same list of my 456 connections (see also the
duplicate bug).

nm-connection-editor (from network-manager-gnome 1.0.2-1) and gnome
-control-center (from gnome-system-tools 3.0.0-4) are still broken:
they both show only part of the connections, and not always the same
ones.

Pietro

Message #65 received at 773525@bugs.debian.org (full text, mbox, reply):

From: Dan Williams <dcbw@redhat.com>

To: Simon McVittie <smcv@debian.org>

Cc: Michael Biebl <biebl@debian.org>, Pietro Battiston <me@pietrobattiston.it>, 773525@bugs.debian.org, Jiří Klimeš <jklimes@redhat.com>

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Wed, 12 Aug 2015 11:58:36 -0500

On Fri, 2015-07-17 at 00:35 +0100, Simon McVittie wrote:
> On 17/07/15 00:13, Dan Williams wrote:
> > eg, think of it as each contact in Telepathy being an object

<snip>

> > The solution here is likely to transition the libnm implementation over
> > to the ObjectManager interface's GetManagedObjects() method for the
> > initial setup, to get everything in one call.  A lot of data, but likely
> > faster than doing it piece-by-piece.
> 
> That's what I'd recommend; ObjectManager is precisely for situations
> where clients are interested in "most" children of a parent object.

That's the path I'd like to pursue, I did an initial implementation of
the OM interface for NM git master (what will become 1.2) based on our
gdbus conversion (which is now merged! hurrah!!!) and filed a bug to
track its ongoing work:

https://bugzilla.gnome.org/show_bug.cgi?id=753566

> > Any
> > chance we could get dbus-glib to implement GetManagedObjects()?
> 
> It is entirely possible to "roll your own" GetManagedObjects; indeed,
> Telepathy reimplements the entire service-side Properties interface
> (because dbus-glib was historically even worse than it is now).
> 
> I'm a little reluctant to add features to dbus-glib, because it's a dead
> end; but if it helps NM, I'd make an exception for reviewing
> ObjectManager support.

Since we've just merged our gdbus conversion, there is less need for
dbus-glib changes now.  Unless of course somebody tries to use kdbus
with NM <= 1.0.x, since the ObjectManager work isn't trivially
backportable because it depends on gdbus.  But if it comes to that,
we'll either say "upgrade" or we'll have to investigate solutions for
dbus-glib-based 1.0.x.  We will not be touching NM 0.9.10 for any of
these issues though.

Dan

Message #70 received at 773525-done@bugs.debian.org (full text, mbox, reply):

From: Michael Biebl <biebl@debian.org>

To: 773525-done@bugs.debian.org

Subject: Re: Bug#773525: Randomly excludes available connections [when there are too many?]

Date: Fri, 3 Jun 2016 11:25:14 +0200

[Message part 1 (text/plain, inline)]

Version: 1.2.0-1

On Wed, 12 Aug 2015 11:58:36 -0500 Dan Williams <dcbw@redhat.com> wrote:
> On Fri, 2015-07-17 at 00:35 +0100, Simon McVittie wrote:
> > On 17/07/15 00:13, Dan Williams wrote:
> > > eg, think of it as each contact in Telepathy being an object
> 
> <snip>
> 
> > > The solution here is likely to transition the libnm implementation over
> > > to the ObjectManager interface's GetManagedObjects() method for the
> > > initial setup, to get everything in one call.  A lot of data, but likely
> > > faster than doing it piece-by-piece.
> > 
> > That's what I'd recommend; ObjectManager is precisely for situations
> > where clients are interested in "most" children of a parent object.
> 
> That's the path I'd like to pursue, I did an initial implementation of
> the OM interface for NM git master (what will become 1.2) based on our
> gdbus conversion (which is now merged! hurrah!!!) and filed a bug to
> track its ongoing work:
> 
> https://bugzilla.gnome.org/show_bug.cgi?id=753566


Version 1.2 is in unstable/testing now, Closing the bug accordingly.

Regards,
Michael
-- 
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.