Debian Bug report logs - #1006800
debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Date: Sat, 05 Mar 2022 12:18:12 +0100

Package: debian-installer
Version: 20210731+deb11u2
Severity: normal

Dear maintainer(s) of debian-installer,

I've been working on reproducible live ISO images for some time now [1] and the
images can be generated in a reproducible manner.
As the next step, I want to test the functionality of the live ISO images and
recently I started working with Philip Hands to get them being tested in
openQA.

I have noticed that the officially released version debian-installer [2][3]
will not work for bookworm and sid, because the kernel version in the debian-
installer does not match the current kernel version. You recently fixed this in
git [4].

Could you release a new version of debian-installer for bookworm and sid?
Or do you recommend a different (release) strategy?
I'm aware of the daily images [5], but they are currently not being
snapshotted, which makes it impossible to reproduce an image after the older
images have been removed from [5].

With kind regards,
Roland Clobus

[1] https://wiki.debian.org/ReproducibleInstalls/LiveImages
[2]
https://snapshot.debian.org/archive/debian/20220305T025031Z/dists/sid/main/installer-
amd64/20210731/
[3] https://deb.debian.org/debian/dists/sid/main/installer-amd64/current/
[4] https://salsa.debian.org/installer-team/debian-
installer/-/blob/f810235e642e7ed266cc6a41b8fccd864180714a
[5] https://d-i.debian.org/daily-images/amd64/daily/


-- System Information:
Debian Release: bookworm/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'testing-debug'), (50, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.16.0-3-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Message #10 received at 1006800@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Roland Clobus <rclobus@rclobus.nl>, 1006800@bugs.debian.org

Subject: Re: Bug#1006800: debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Date: Sat, 5 Mar 2022 12:40:20 +0100

[Message part 1 (text/plain, inline)]

Hi,

Roland Clobus <rclobus@rclobus.nl> (2022-03-05):
> I have noticed that the officially released version debian-installer
> [2][3] will not work for bookworm and sid, because the kernel version
> in the debian- installer does not match the current kernel version.
> You recently fixed this in git [4].
> 
> Could you release a new version of debian-installer for bookworm and
> sid?

We could, and should, release a new d-i and possibly an Alpha 1 at some
point, but I don't have a specific timeline for that.

> Or do you recommend a different (release) strategy?

A new debian-installer upload (prelude to the aforementioned Alpha 1) is
only going to help until src:linux gets a new ABI bump, so that's only
going to be temporary anyway.

> I'm aware of the daily images [5], but they are currently not being
> snapshotted, which makes it impossible to reproduce an image after the
> older images have been removed from [5].

If you're using a specific build, you could mirror it on your side, and
then have a way to point at the mirrored copy so that you wouldn't
depend on d-i.d.o's contents (that's an approach seen in various
projects, e.g.  time-based snapshots and tagged snapshots in Tails, even
if that's for Debian as a whole, not d-i)?

How long do you need to go back / how long do you need to keep a given
build? Maybe we could just keep (some) builds for a longer while there,
but that's at 90 days already.


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 1006800@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>

To: Cyril Brulebois <kibi@debian.org>, 1006800@bugs.debian.org, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Re: Bug#1006800: debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Date: Sat, 5 Mar 2022 14:26:15 +0100

[Message part 1 (text/plain, inline)]

+mailing list rb-general

Hi,

On 05/03/2022 12:40, Cyril Brulebois wrote:
> Roland Clobus <rclobus@rclobus.nl> (2022-03-05):
>> I have noticed that the officially released version debian-installer
>> [2][3] will not work for bookworm and sid, because the kernel version
>> in the debian- installer does not match the current kernel version.
>> You recently fixed this in git [4].
>>
>> Could you release a new version of debian-installer for bookworm and
>> sid?
> 
> We could, and should, release a new d-i and possibly an Alpha 1 at some
> point, but I don't have a specific timeline for that.

Understood. I assume that an Alpha 1 release will be made somewhere near 
the release date of bookworm.

>> Or do you recommend a different (release) strategy?
> 
> A new debian-installer upload (prelude to the aforementioned Alpha 1) is
> only going to help until src:linux gets a new ABI bump, so that's only
> going to be temporary anyway.

Indeed. A new debian-installer upload would need to happen in lock-step 
with every new ABI in src:linux, to guarantee a consistent state of d-i.
This could mean quite some work on your side.

>> I'm aware of the daily images [5], but they are currently not being
>> snapshotted, which makes it impossible to reproduce an image after the
>> older images have been removed from [5].
> 
> If you're using a specific build, you could mirror it on your side, and
> then have a way to point at the mirrored copy so that you wouldn't
> depend on d-i.d.o's contents (that's an approach seen in various
> projects, e.g.  time-based snapshots and tagged snapshots in Tails, even
> if that's for Debian as a whole, not d-i)?

I do not know how much work it is to release a new version of 
debian-installer. Currently the state of the official repository 
(deb.debian.org) is a non-working installer for bookworm and sid.

I'm looking at possible solutions here (that's why I've added the 
rb-general mailing list):
* (Manually) do official releases of debian-installer more often
  (as I wrote, openQA will soon have some tests that detect when the 
kernel version got out-of-sync)
* Automatically release git snapshots to deb.d.o instead of d-i.d.o
* Extend snapshot.d.o and/or snapshot.notset.fr to cover d-i.d.o in 
addition to deb.d.o
* No changes, and accept that older images cannot be recreated (this 
option is not preferred by me)
* Other ...

> How long do you need to go back / how long do you need to keep a given
> build? Maybe we could just keep (some) builds for a longer while there,
> but that's at 90 days already.

Looking at https://d-i.debian.org/daily-images/amd64/, the current 
history I can see is about 15 days.

While investigating reproducible issues I personally tend to pick some 
timestamp and work on that for a longer period of time. 90 days would 
suffice completely for my purpose.

With kind regards,
Roland Clobus

[OpenPGP_signature (application/pgp-signature, attachment)]

Message #20 received at 1006800@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Roland Clobus <rclobus@rclobus.nl>

Cc: 1006800@bugs.debian.org, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Re: Bug#1006800: debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Date: Sat, 5 Mar 2022 18:45:51 +0100

[Message part 1 (text/plain, inline)]

Roland Clobus <rclobus@rclobus.nl> (2022-03-05):
> On 05/03/2022 12:40, Cyril Brulebois wrote:
> > We could, and should, release a new d-i and possibly an Alpha 1 at some
> > point, but I don't have a specific timeline for that.
> 
> Understood. I assume that an Alpha 1 release will be made somewhere
> near the release date of bookworm.

In the past I've tried to have an Alpha 1 released after a few months
into the new release cycle, then aim for something like a release every
1-2 months.

But the archive can disagree from time to time, and lately, I'm rather
busy with other things…

> Indeed. A new debian-installer upload would need to happen in
> lock-step with every new ABI in src:linux, to guarantee a consistent
> state of d-i. This could mean quite some work on your side.

Uploading more often could be doable; the current approach has been to
upload whenever we were getting close to wanting a new release of the
installer, following up with 1-2 more uploads if things didn't work out
immediately.

> I'm looking at possible solutions here (that's why I've added the
> rb-general mailing list):
> * (Manually) do official releases of debian-installer more often

Doable, as stated above.

>   (as I wrote, openQA will soon have some tests that detect when the
>   kernel version got out-of-sync)

Well, I do follow kernel uploads and ABI bumps closely, so that's a
problem that's already solved (git says since 2013).

FWIW we're seeing mutiple hours to multiple days of delay for some
architectures (we need the *-signed packages to get processed).

> * Automatically release git snapshots to deb.d.o instead of d-i.d.o

Uploading untested stuff to the archive, automatically? No, thanks.

> * Extend snapshot.d.o and/or snapshot.notset.fr to cover d-i.d.o in
>   addition to deb.d.o

Maybe avoidable if we go for more frequent uploads.

> * No changes, and accept that older images cannot be recreated (this
>   option is not preferred by me)
> * Other ...
> 
> > How long do you need to go back / how long do you need to keep a given
> > build? Maybe we could just keep (some) builds for a longer while there,
> > but that's at 90 days already.
> 
> Looking at https://d-i.debian.org/daily-images/amd64/, the current
> history I can see is about 15 days.

Right, ISTR that was around 30 days, glanced at the graph before hitting
the sack and didn't check thoroughly: we indeed clean more frequently
than showed on the graph.

> While investigating reproducible issues I personally tend to pick some
> timestamp and work on that for a longer period of time. 90 days would
> suffice completely for my purpose.

At the moment, daily-images is 42G, 6.8G of it being amd64. If we
estimate a ×6 in size for just that architecture, that's nearly doubling
that size, and it's only covering that one architecture. The machine is
shared across various services, but has 121G free at the moment, so we
could probably try something like that (maybe some middleground like 45
days for starters) if that helped in the very short term? Or would you
need to look at more architectures immediately?


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 1006800@bugs.debian.org (full text, mbox, reply):

From: Roland Clobus <rclobus@rclobus.nl>

To: Cyril Brulebois <kibi@debian.org>

Cc: 1006800@bugs.debian.org

Subject: Re: Bug#1006800: debian-installer: kernel mismatch for bookworm and sid installer. New release needed?

Date: Sun, 27 Mar 2022 12:44:24 +0200

[Message part 1 (text/plain, inline)]

Removed: rb-general@lists.reproducible-builds.org

Hello Cyril,

On 05/03/2022 18:45, Cyril Brulebois wrote:
> Roland Clobus <rclobus@rclobus.nl> (2022-03-05):
>> On 05/03/2022 12:40, Cyril Brulebois wrote:
>>> We could, and should, release a new d-i and possibly an Alpha 1 at some
>>> point, but I don't have a specific timeline for that.
>>
>> Understood. I assume that an Alpha 1 release will be made somewhere
>> near the release date of bookworm.
> 
> In the past I've tried to have an Alpha 1 released after a few months
> into the new release cycle, then aim for something like a release every
> 1-2 months.
> 
> But the archive can disagree from time to time, and lately, I'm rather
> busy with other things…

Understood. So I've focussed on building the daily image myself, using 
the git version.
This will
1) allow me to generate installer snapshots for testing and unstable 
that have their correct kernel version (because you diligently fix that 
in the git repo)
2) save you the time of doing new releases for testing and unstable.

After some hick-ups, I've got a working version now.
(See my aborted MR27: 
https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/27)

I'm currently preparing a 'rebuild' script for live-build, that will 
reproducibly re-generate an image, including the installer that matches 
that specific point in time.

If you are interested, it might be usable for the daily-build script as 
well, because it will not use the timestamp for SOURCE_DATE_EPOCH of 
midnight (the time the script is started), but the timestamp of the last 
completed snapshot of the archive.

>>> How long do you need to go back / how long do you need to keep a given
>>> build?

I can go back as far as I want right now. There is no need any more for 
the d-i.debian.org snapshots when I recreate the installer.

With kind regards,
Roland

[OpenPGP_signature (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.