Debian Bug report logs - #553211
/sbin/dhclient3: dhclient kills root filesystem on network

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: "Michal Suchanek <hramrach@centrum.cz>" <michal.suchanek@ruk.cuni.cz>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Thu, 29 Oct 2009 18:24:22 +0100

Package: dhcp3-client
Version: 3.1.1-6+lenny3
Severity: normal
File: /sbin/dhclient3


I am building debian-live images which boot form the network.

For this to work the network is first initialized and the root
filesystem mounted in in initramfs and then dhclient is started as
normal from initscripts to manage lease renewal.

The most common boot failure (other than the root filesystem not being
on the specified server) is dhclient fumbling the network setup.

dhclient runs external programs and scritps while the network is not
fully configured. This cannot work because the programs are on an
external server and they have to be fetched from there over the network.

This requires that the IP address, routing (default route) and
resolv.conf are in a working state every time dhclient attempts to run
anything.

A working state is either the state before dhclient started or the state
after it is finished setting up the network.

I wrote a script that moves dhcp3 enter hooks to exit hooks which seemed
to resolve most issues but this time I built a new image and tried to
boot it in a network different from the server subnet and it would fail
with default route unset.


-- System Information:
Debian Release: 5.0.3
  APT prefers stable
  APT policy: (900, 'stable'), (500, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.32-rc3 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages dhcp3-client depends on:
ii  debconf [debconf-2.0]     1.5.24         Debian configuration management sy
ii  debianutils               2.30           Miscellaneous utilities specific t
ii  dhcp3-common              3.1.1-6+lenny3 common files used by all the dhcp3
ii  libc6                     2.10.1-2       GNU C Library: Shared libraries

dhcp3-client recommends no packages.

Versions of packages dhcp3-client suggests:
pn  avahi-autoipd                 <none>     (no description available)
pn  resolvconf                    <none>     (no description available)

-- debconf information excluded

Message #10 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Andrew Pollock <apollock@debian.org>

To: "Michal Suchanek <hramrach@centrum.cz>" <michal.suchanek@ruk.cuni.cz>, 553211@bugs.debian.org

Subject: Re: Bug#553211: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Wed, 4 Nov 2009 17:17:35 +1000

tags 553211 + moreinfo
thanks

On Thu, Oct 29, 2009 at 06:24:22PM +0100, Michal Suchanek <hramrach@centrum.cz> wrote:
> Package: dhcp3-client
> Version: 3.1.1-6+lenny3
> Severity: normal
> File: /sbin/dhclient3
> 
> 
> I am building debian-live images which boot form the network.
> 
> For this to work the network is first initialized and the root
> filesystem mounted in in initramfs and then dhclient is started as
> normal from initscripts to manage lease renewal.
> 
> The most common boot failure (other than the root filesystem not being
> on the specified server) is dhclient fumbling the network setup.
> 
> dhclient runs external programs and scritps while the network is not
> fully configured. This cannot work because the programs are on an
> external server and they have to be fetched from there over the network.
> 
> This requires that the IP address, routing (default route) and
> resolv.conf are in a working state every time dhclient attempts to run
> anything.
> 
> A working state is either the state before dhclient started or the state
> after it is finished setting up the network.
> 
> I wrote a script that moves dhcp3 enter hooks to exit hooks which seemed
> to resolve most issues but this time I built a new image and tried to
> boot it in a network different from the server subnet and it would fail
> with default route unset.
> 

I don't understand what your bugreport is about. What exactly is at fault
and what do you need changed?

If you just need basic networking configured from within your initramfs,
maybe look at ipconfig from klibc-utils

Message #17 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Michal Suchanek <michal.suchanek@ruk.cuni.cz>

To: Andrew Pollock <apollock@debian.org>

Cc: "553211@bugs.debian.org" <553211@bugs.debian.org>

Subject: Re: Bug#553211: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Wed, 04 Nov 2009 11:47:14 +0100

Andrew Pollock wrote:
> tags 553211 + moreinfo
> thanks
> 
> On Thu, Oct 29, 2009 at 06:24:22PM +0100, Michal Suchanek <hramrach@centrum.cz> wrote:
>> Package: dhcp3-client
>> Version: 3.1.1-6+lenny3
>> Severity: normal
>> File: /sbin/dhclient3
>>
>>
>> I am building debian-live images which boot form the network.
>>
>> For this to work the network is first initialized and the root
>> filesystem mounted in in initramfs and then dhclient is started as
>> normal from initscripts to manage lease renewal.
>>
>> The most common boot failure (other than the root filesystem not being
>> on the specified server) is dhclient fumbling the network setup.
>>
>> dhclient runs external programs and scritps while the network is not
>> fully configured. This cannot work because the programs are on an
>> external server and they have to be fetched from there over the network.
>>
>> This requires that the IP address, routing (default route) and
>> resolv.conf are in a working state every time dhclient attempts to run
>> anything.
>>
>> A working state is either the state before dhclient started or the state
>> after it is finished setting up the network.
>>
>> I wrote a script that moves dhcp3 enter hooks to exit hooks which seemed
>> to resolve most issues but this time I built a new image and tried to
>> boot it in a network different from the server subnet and it would fail
>> with default route unset.
>>
> 
> I don't understand what your bugreport is about. What exactly is at fault
> and what do you need changed?
> 
> If you just need basic networking configured from within your initramfs,
> maybe look at ipconfig from klibc-utils


That works flawlessly.

However, when dhclient starts from the normal networking initscripts it 
kills the network if

 - dhclient enter hooks exist

or

 - the client is not on the same subnet as the server that provides the 
root filesystem

I can disable dhclient but then I won't get lease renewal. This could 
work when I have control over the dhcp server and configure it to 
support this behaviour but in general this is not how dhcp should work.

Thanks

Michal

Message #22 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Ferenc Wagner <wferi@niif.hu>

To: 553211@bugs.debian.org

Subject: Re: Bug#553211: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Fri, 16 Apr 2010 20:30:20 +0200

Hi,

This is an old grief of mine, too.  To state the problem differently:
when dhclient acquires a new lease, it flushes all network config first,
which brings an NFS rooted setup down.  Even if a second later the exact
same network config would be set up, since the lease isn't really new,
initramfs acquired it a couple of seconds earlier.  It just isn't
transferred from the initramfs to the running system.

A generic solution would be much needed here, see for example
http://thread.gmane.org/gmane.linux.raid/26191/focus=1290: others also
hack around this by various ways...
-- 
Regards,
Feri.

Message #27 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Michal Suchanek <hramrach@centrum.cz>

To: Debian Bug Tracking System <553211@bugs.debian.org>

Subject: Re: Bug#553211: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Mon, 7 Jun 2010 23:34:20 +0200

[Message part 1 (text/plain, inline)]

This patch seems to resolve the issue with reconfiguring the network.

I am not sure it it might cause other problems. The script might rely
on the routes through the interface going away which would no loner be
the case after applying the patch I guess.

[dhclient-script.patch (text/x-diff, attachment)]

Message #32 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Michal Suchanek <michal.suchanek@ruk.cuni.cz>

To: Debian Bug Tracking System <553211@bugs.debian.org>

Subject: /sbin/dhclient: What is unclear about this issue?

Date: Tue, 17 May 2011 15:46:58 +0000

Package: isc-dhcp-client
Version: 4.1.1-P1-15+squeeze2
Severity: normal
File: /sbin/dhclient


This report is marked "moreinfo".

What is unclear about this issue.

To reproduce:

1) netboot a box over eth0 eg. using one of the squeeze debian-live images
2) run "dhclint eth0"

Result: box dies
Expected: box keeps renewing dhcp leases as required by dhcp protocol

Thanks

Michal

-- System Information:
Debian Release: 6.0.1
  APT prefers stable
  APT policy: (990, 'stable'), (900, 'testing'), (800, 'unstable'), (500, 'stable-updates'), (110, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.39-rc7-overlayfs.v9-20110517103355-55cc666-i686+ (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages isc-dhcp-client depends on:
ii  debianutils         3.4                  Miscellaneous utilities specific t
ii  iproute             20100519-3           networking and traffic control too
ii  isc-dhcp-common     4.1.1-P1-15+squeeze2 common files used by all the isc-d
ii  libc6               2.11.2-10            Embedded GNU C Library: Shared lib

isc-dhcp-client recommends no packages.

Versions of packages isc-dhcp-client suggests:
pn  avahi-autoipd                 <none>     (no description available)
pn  resolvconf                    <none>     (no description available)

-- no debconf information

Message #37 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Michal Suchanek <michal.suchanek@ruk.cuni.cz>

To: Debian Bug Tracking System <553211@bugs.debian.org>

Subject: /sbin/dhclient: Patch.

Date: Wed, 18 May 2011 16:18:11 +0200

[Message part 1 (text/plain, inline)]

Package: isc-dhcp-client
Version: 4.1.1-P1-15+squeeze2
Severity: normal
File: /sbin/dhclient
Tags: patch


Sending patch which works around this issue.

There si no way to update routing tables atomically together with the ip
address so there is always a possibility that the network goes down
while routing is updated.

However, the time when network is down should be minimized.
Specifically the preinit should *not* reset networking, dhclient uses
raw socket to obtain IP address so it should work regardless of current
networking setup so long as the interface is up.

This patch relies on kernel cache to hold the "ifconfig" and "route"
binaries while the network is down and on the network filesystem driver
to retry any transactions that might get lost while networking is
reenacted by dhclient.

Thanks

Michal



-- System Information:
Debian Release: 6.0.1
  APT prefers stable
  APT policy: (900, 'stable'), (510, 'unstable'), (500, 'testing'), (200, 'experimental'), (111, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.38-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages isc-dhcp-client depends on:
ii  debianutils         3.4                  Miscellaneous utilities specific t
ii  iproute             20100519-3           networking and traffic control too
ii  isc-dhcp-common     4.1.1-P1-15+squeeze2 common files used by all the isc-d
ii  libc6               2.11.2-10            Embedded GNU C Library: Shared lib

isc-dhcp-client recommends no packages.

Versions of packages isc-dhcp-client suggests:
pn  avahi-autoipd                 <none>     (no description available)
pn  resolvconf                    <none>     (no description available)

-- Configuration Files:
/etc/dhcp/dhclient.conf changed [not included]

-- no debconf information

[dhclient-script.patch (text/plain, attachment)]

Message #42 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Daniel Baumann <daniel.baumann@progress-technologies.net>

To: 553211@bugs.debian.org

Cc: 553211-submitter@bugs.debian.org

Subject: Re: /sbin/dhclient3: dhclient kills root filesystem on network

Date: Sat, 22 Oct 2011 07:43:35 +0200

severity 553211 important
tag 553211 - moreinfo
tag 553211 patch
thanks

Andrew,

please comment on the last messages of Michal. This is a major problem
for debian-live that basically makes netboot setups near to impossible.

Regards,
Daniel

-- 
Address:        Daniel Baumann, Donnerbuehlweg 3, CH-3012 Bern
Email:          daniel.baumann@progress-technologies.net
Internet:       http://people.progress-technologies.net/~daniel.baumann/

Message #56 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Andrew Pollock <apollock@debian.org>

To: Michal Suchanek <michal.suchanek@ruk.cuni.cz>, 553211@bugs.debian.org

Subject: Re: Bug#553211: /sbin/dhclient: Patch.

Date: Sat, 19 Nov 2011 21:05:54 -0800

[Message part 1 (text/plain, inline)]

On Wed, May 18, 2011 at 04:18:11PM +0200, Michal Suchanek wrote:
> Package: isc-dhcp-client
> Version: 4.1.1-P1-15+squeeze2
> Severity: normal
> File: /sbin/dhclient
> Tags: patch
> 
> 
> Sending patch which works around this issue.

Thankyou for the patch. Sorry I've only just gotten around to looking at it.

[snip]

> 
> This patch relies on kernel cache to hold the "ifconfig" and "route"
> binaries while the network is down and on the network filesystem driver
> to retry any transactions that might get lost while networking is
> reenacted by dhclient.

So if I'm reading your patch correctly, it looks like just the act of doing
an ifconfig up of an already up interface is enough to cause an NFS-mounted
root filesystem to lose the plot?

Unfortunately since you wrote your patch, we've gone and changed
dhclient-script to use iproute2 instead of ifconfig and friends, so I'd like
to know if the "ip link set dev blah up" we're now doing in place of the
ifconfig call also causes the problem.

If so, it won't be a problem for me to take the intent of your patch and
adjust it for the way dhclient-script looks today.
 
regards

Andrew

[signature.asc (application/pgp-signature, inline)]

Message #61 received at 553211@bugs.debian.org (full text, mbox, reply):

From: Michal Suchanek <michal.suchanek@ruk.cuni.cz>

To: Andrew Pollock <apollock@debian.org>

Cc: "553211@bugs.debian.org" <553211@bugs.debian.org>

Subject: Re: Bug#553211: /sbin/dhclient: Patch.

Date: Mon, 21 Nov 2011 16:05:13 +0100

Excerpts from Andrew Pollock's message of Sun Nov 20 06:05:54 +0100 2011:
> On Wed, May 18, 2011 at 04:18:11PM +0200, Michal Suchanek wrote:

> 
> > 
> > This patch relies on kernel cache to hold the "ifconfig" and "route"
> > binaries while the network is down and on the network filesystem driver
> > to retry any transactions that might get lost while networking is
> > reenacted by dhclient.
> 
> So if I'm reading your patch correctly, it looks like just the act of doing
> an ifconfig up of an already up interface is enough to cause an NFS-mounted
> root filesystem to lose the plot?

No. The script does "ifconfig $interface inet 0 up" which clears the
internet address in the interface which obviously does break networking.

> 
> Unfortunately since you wrote your patch, we've gone and changed
> dhclient-script to use iproute2 instead of ifconfig and friends, so I'd like
> to know if the "ip link set dev blah up" we're now doing in place of the
> ifconfig call also causes the problem.

So long as it merely brings up the interface but does not change the
address it should not cause problems.

Thanks

Michal

Message #66 received at 553211-done@bugs.debian.org (full text, mbox, reply):

From: Andrew Pollock <apollock@debian.org>

To: Michal Suchanek <michal.suchanek@ruk.cuni.cz>

Cc: "553211@bugs.debian.org" <553211-done@bugs.debian.org>

Subject: Re: Bug#553211: /sbin/dhclient: Patch.

Date: Mon, 21 Nov 2011 22:28:59 -0800

[Message part 1 (text/plain, inline)]

Version: 4.2.2-1

On Mon, Nov 21, 2011 at 04:05:13PM +0100, Michal Suchanek wrote:
> Excerpts from Andrew Pollock's message of Sun Nov 20 06:05:54 +0100 2011:
> > On Wed, May 18, 2011 at 04:18:11PM +0200, Michal Suchanek wrote:
> 
> > 
> > > 
> > > This patch relies on kernel cache to hold the "ifconfig" and "route"
> > > binaries while the network is down and on the network filesystem driver
> > > to retry any transactions that might get lost while networking is
> > > reenacted by dhclient.
> > 
> > So if I'm reading your patch correctly, it looks like just the act of doing
> > an ifconfig up of an already up interface is enough to cause an NFS-mounted
> > root filesystem to lose the plot?
> 
> No. The script does "ifconfig $interface inet 0 up" which clears the
> internet address in the interface which obviously does break networking.

Ah yes, I overlooked the zero, sorry.
 
> > 
> > Unfortunately since you wrote your patch, we've gone and changed
> > dhclient-script to use iproute2 instead of ifconfig and friends, so I'd like
> > to know if the "ip link set dev blah up" we're now doing in place of the
> > ifconfig call also causes the problem.
> 
> So long as it merely brings up the interface but does not change the
> address it should not cause problems.

Yes, that's all it does now. I'll mark this as fixed in 4.2.2-1

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.