Debian Bug report logs - #968519
configure_networking(): race condition with RockPro64: calls ipconfig before NIC is detected

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Forest <forestix@sonic.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: dropbear-initramfs: race condition prevents launch at boot

Date: Sun, 16 Aug 2020 12:31:38 -0700

Package: dropbear-initramfs
Version: 2020.80-1
Severity: important

Dear Maintainer,

About four times out of five, dropbear fails to launch at boot on the
RockPro64 single-board computer. When it fails, I see this message
repeated several times, scattered between various other messages on the
serial console:

  ipconfig: no devices to configure

Finally, this message appears:

  /scripts/init-premount/dropbear: .: line 279:
  can't open '/run/net-*.conf': No such file or directory

The system then proceeds to the luks unlock prompt at the serial console
only, with dropbear not running, so the boot process will not continue
unless I connect a serial console and enter the password that way.

I have tried adding all my ethernet-related driver modules to
/etc/initramfs-tools/modules and rebuilding initramfs, but it
didn't help. I have tried using ip= kernel command line arguments
for a static address instead of DHCP, but that didn't help either.

However, editing /usr/share/initramfs-tools/scripts/init-premount/dropbear
does get it working. Making that script wait for a /run/net-*.conf file to
appear before it calls configure_networking appears to be a solution.
Even just sleeping for a few seconds before the configure_networking call
seems to work, though I don't know how consistently.

Based on this workaround and the fact that the aforementioned error messages
are interleaved with other boot messages, it looks to me like dropbear's
init-premount script is being run in parallel with driver and network setup,
and often executing before the rk3399's onboard ethernet has a chance to
finish initializing.


-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: arm64 (aarch64)

Kernel: Linux 5.7.0-2-arm64 (SMP w/6 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not
set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages dropbear-initramfs depends on:
ii  busybox          1:1.30.1-5
ii  dropbear-bin     2020.80-1
ii  initramfs-tools  0.137
ii  udev             246-2

Versions of packages dropbear-initramfs recommends:
ii  cryptsetup-initramfs  2:2.3.3-1

dropbear-initramfs suggests no packages.

-- Configuration Files:
/etc/dropbear-initramfs/config changed:
DROPBEAR_OPTIONS="-p 222"


-- no debconf information

Message #10 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Guilhem Moulin <guilhem@debian.org>

To: Forest <forestix@sonic.net>, 968519@bugs.debian.org

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Sun, 16 Aug 2020 23:10:34 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Sun, 16 Aug 2020 at 12:31:38 -0700, Forest wrote:
> When it fails, I see this message repeated several times, scattered
> between various other messages on the serial console:
> 
> ipconfig: no devices to configure
> 
> Finally, this message appears:
> 
> /scripts/init-premount/dropbear: .: line 279:
> can't open '/run/net-*.conf': No such file or directory

This comes from initramfs-tools's configure_networking()
https://salsa.debian.org/kernel-team/initramfs-tools/-/blob/v0.137/scripts/functions#L279

> However, editing /usr/share/initramfs-tools/scripts/init-premount/dropbear
> does get it working. Making that script wait for a /run/net-*.conf file to
> appear before it calls configure_networking appears to be a solution.

No idea if that's a valid workaround, but I don't think that's something
to implement in dropbear-initramfs: presumably NFS rootfs are affected
as well on your hardware, in which case it has nothing do with
dropbear-initramfs (nor cryptsetup/LUKS).

> Based on this workaround and the fact that the aforementioned error messages
> are interleaved with other boot messages, it looks to me like dropbear's
> init-premount script is being run in parallel with driver and network setup,
> and often executing before the rk3399's onboard ethernet has a chance to
> finish initializing.

configure_networking() is run in the backround, yup (and the dropbear
server is started afterwards if successful) see #806884.  But that's done
at init-premount stage, after udev and the required modules are loaded.
Is your network interface brought up by a script in init-premount or
local-top?

Please also try the attached patch (with and without delay).  If this
alone doesn't help BUT the addition of some delay *before* running
configure_networking() fixes the race condition, then this bug should
probably reassigned to initramfs-tools.

Cheers,
-- 
Guilhem.

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Forest <forestix@sonic.net>

To: Guilhem Moulin <guilhem@debian.org>

Cc: 968519@bugs.debian.org

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Sun, 16 Aug 2020 17:54:01 -0700

On Sun, 16 Aug 2020 23:10:34 +0200, Guilhem Moulin wrote:

>configure_networking() is run in the backround, yup (and the dropbear
>server is started afterwards if successful) see #806884.  But that's done
>at init-premount stage, after udev and the required modules are loaded.
>Is your network interface brought up by a script in init-premount or
>local-top?

It doesn't appear so.

Both of these dirs are empty:
/etc/initramfs-tools/scripts/{init-premount,local-top}

/usr/share/initramfs-tools/scripts/init-premount contains only the
"dropbear" script from dropbear-initramfs.

/usr/share/initramfs-tools/scripts/local-top contains only "cryptopensc" and
"cryptroot", neither of which look like they bring up a network interface.

>Please also try the attached patch (with and without delay).

Was there supposed to be an attachment? The only one I received is your pgp
signature. I don't see a patch in the bugs.debian.org copy of this thread,
either.

Message #20 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Guilhem Moulin <guilhem@debian.org>

To: Forest <forestix@sonic.net>

Cc: 968519@bugs.debian.org

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Mon, 17 Aug 2020 10:44:47 +0200

[Message part 1 (text/plain, inline)]

On Sun, 16 Aug 2020 at 17:54:01 -0700, Forest wrote:
>>Please also try the attached patch (with and without delay).
> 
> Was there supposed to be an attachment? The only one I received is your pgp
> signature. I don't see a patch in the bugs.debian.org copy of this thread,
> either.

Ooops typical, should be better now :-)

-- 
Guilhem.

[dropbear-initramfs.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Forest <forestix@sonic.net>

To: Guilhem Moulin <guilhem@debian.org>

Cc: 968519@bugs.debian.org

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Tue, 18 Aug 2020 23:27:42 -0700

On Sun, 16 Aug 2020 23:10:34 +0200, Guilhem Moulin wrote:

>Please also try the attached patch (with and without delay).  If this
>alone doesn't help BUT the addition of some delay *before* running
>configure_networking() fixes the race condition, then this bug should
>probably reassigned to initramfs-tools.

As you suspected, that patch alone does not fix it, but adding a delay at
the point you marked does.  (At least, it worked 8 times in a row.)

Message #30 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Guilhem Moulin <guilhem@debian.org>

To: 968519@bugs.debian.org

Cc: Forest <forestix@sonic.net>

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Wed, 19 Aug 2020 14:22:01 +0200

[Message part 1 (text/plain, inline)]

Control: tag -1 - moreinfo
Control: reassign -1 initramfs-tools-core 0.137
Control: retitle -1 configure_networking(): race condition with RockPro64: calls ipconfig before NIC is detected

On Tue, 18 Aug 2020 at 23:27:42 -0700, Forest wrote:
> On Sun, 16 Aug 2020 23:10:34 +0200, Guilhem Moulin wrote:
>>Please also try the attached patch (with and without delay).  If this
>>alone doesn't help BUT the addition of some delay *before* running
>>configure_networking() fixes the race condition, then this bug should
>>probably reassigned to initramfs-tools.
> 
> As you suspected, that patch alone does not fix it, but adding a delay at
> the point you marked does.  (At least, it worked 8 times in a row.)

Thanks for the follow up, reassigning accordingly.

-- 
Guilhem.

[signature.asc (application/pgp-signature, inline)]

Message #43 received at 968519@bugs.debian.org (full text, mbox, reply):

From: Forest <forestix@sonic.net>

To: 968519@bugs.debian.org

Subject: Re: Bug#968519: dropbear-initramfs: race condition prevents launch at boot

Date: Mon, 31 Aug 2020 12:06:33 -0700

Thanks, Guilhem.

Is anything more needed from me to get this fixed?

Message #50 received at 965935-close@bugs.debian.org (full text, mbox, reply):

From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>

To: 965935-close@bugs.debian.org

Subject: Bug#965935: fixed in initramfs-tools 0.140

Date: Sat, 13 Mar 2021 18:49:40 +0000

Source: initramfs-tools
Source-Version: 0.140
Done: Bastian Blank <waldi@debian.org>

We believe that the bug you reported is fixed in the latest version of
initramfs-tools, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 965935@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bastian Blank <waldi@debian.org> (supplier of updated initramfs-tools package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sat, 13 Mar 2021 19:13:18 +0100
Source: initramfs-tools
Architecture: source
Version: 0.140
Distribution: unstable
Urgency: medium
Maintainer: Debian kernel team <debian-kernel@lists.debian.org>
Changed-By: Bastian Blank <waldi@debian.org>
Closes: 965935
Changes:
 initramfs-tools (0.140) unstable; urgency=medium
 .
   [ Norbert Lange ]
   * mkinitramfs: use temporary file for uncompressed cpio
 .
   [ Michael Prokop ]
   * mkinitramfs: report actually requested compress util if it's not present
 .
   [ Ben Hutchings ]
   * init: Mount /dev without the noexec option
 .
   [ dann frazier ]
   * configure_networking(): Wait for specified network device (Closes: #965935)
 .
   [ Bastian Blank ]
   * Include all PCI controller modules
   * Include all reset controller modules
Checksums-Sha1:
 ca9b2b7c1bcd523e6b0d01610a8ffc1f9469d3c9 1599 initramfs-tools_0.140.dsc
 503b9ba5619bbb5fc1fefc54148b438517306a2f 95044 initramfs-tools_0.140.tar.xz
 98067441d1545346eb097dbc0e8065c3bee85dfd 5816 initramfs-tools_0.140_source.buildinfo
Checksums-Sha256:
 5537ff004f5e7fc9be4790a3248664806fa1ae4e5fe05ea47df821be8953aef4 1599 initramfs-tools_0.140.dsc
 17ec1b0e5e1c6f8254f2184be780e50c8274f71cdf482e5c46d50f955370685f 95044 initramfs-tools_0.140.tar.xz
 82eb6659251b86789c099f0c8f935eeac45c3158ad9a019583071baaf7e012cc 5816 initramfs-tools_0.140_source.buildinfo
Files:
 02ccd954ee0094babe87b535f98f2149 1599 utils optional initramfs-tools_0.140.dsc
 fd1364cfa57fa3580a70c6acf53b64ae 95044 utils optional initramfs-tools_0.140.tar.xz
 162c5689195e92895b38ef34766eb053 5816 utils optional initramfs-tools_0.140_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCgAdFiEER3HMN63jdS1rqjxLbZOIhYpp/lEFAmBNAiAACgkQbZOIhYpp
/lFf9AgAkzB4sVAsdAwPFul7jOh+Ocw4eZfHE4K10cAdz2p0tNq5bMipbwCBwLjh
ZvxYmFAcEDQBBCZkdQyWZPrVoiwM73IgU8MTP28MYfO/I2T4/N4cLO1KdUVpbg23
fridBQVl9xmVonmskgbjtQfvP4g0GJvEBcskexwpuTFEuIhFwJ3G2IXFYSkLkO3X
2zUv8jtZ4GpQ674kgK3+aco+IqsfCtD+BSvnpSkttuZdU/GjFsY1puFFceWbx8JU
zozPKEvPEH8mvtk3KenZbCO4jalF3xbhmmU00yHsHXeUlSKotpPyM5DsSaOy9S2D
5slscGKgtoTlxkqaPkcvlsj8JrKivA==
=/SaL
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.