Debian Bug report logs - #724931
ISO loopback support for Debian installer

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Grub2-Multiboot-ISO-USB-Stick needs loop.ko and iso-scan

Date: Sun, 29 Sep 2013 19:59:23 +0200

Package: debian-installer
Severity: wishlist

Dear Maintainer,

The module 'loop.ko' is not shipped with the Debian testing netinstall ISO.
It should reside in /lib/modules/3.10-2-amd64/kernel/drivers/block/.
Because it is missing, it is impossible to mount ISO images during the 
install and thus preventing installation from ISO, if the module is not 
manually imported.
Please include the kernel module loop.ko in all installation ISOs.

It would be awesome, if you could also add iso-scan to the install ISOs, 
so that one can use the 'findiso=' boot option.
This would make it possible to install from the ISO without having to 
mount it manually during the installation.

Thanks,
Andreas


-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (700, 'testing'), (600, 'unstable'), (70, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.10-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Message #10 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: joey@kitenet.net, debian-boot@lists.debian.org, 724931@bugs.debian.org

Subject: Re: loop-mounted ISO images

Date: Mon, 30 Sep 2013 13:10:01 +0200

[Now follows a somewhat lengthy text. If you are bored, jump directly to 
the conclusion.]
On 30.09.2013 01:58, wrote ian_bruce@fastmail.net:
> you wrote:
>
>> Please include the kernel module loop.ko in all installation ISOs.
>>
>> It would be awesome, if you could also add iso-scan to the install
>> ISOs, so that one can use the 'findiso=' boot option. This would make
>> it possible to install from the ISO without having to mount it
>> manually during the installation.
>
> I refer you to my recent posts (including patch) on exactly this
> subject:
>
> http://lists.debian.org/debian-boot/2013/09/msg00292.html
>
> http://lists.debian.org/debian-boot/2013/09/msg00509.html
>
> -- which have so far been completely ignored.
>
> As for the iso-scan package, if you search the source code for the
> string "findiso", you will not find it.
>
> http://packages.debian.org/wheezy/iso-scan
>
> You can confirm this by looking through the "hd-media" images directly:
>
>      $ zcat initrd.gz | strings | grep -i findiso
>      $
>
> http://ftp.us.debian.org/debian/dists/wheezy/main/installer-amd64/current/images/hd-media/
>
> So if this functionality was present at one time, it is not now.
>
> Here is some relevant prior discussion:
>
> http://lists.debian.org/debian-boot/2009/10/threads.html#00064
>
> An unnecessarily complex solution was eventually adopted:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=564441
>
> You will note that my patch does exactly what was asked for then, and
> what you are requesting now. I will post this information to bug
> #724931; hopefully we can get my patch, or something like it, applied to
> the standard installer ISOs, and not just the "hd-media" images, which
> apparently are the only ones to include the iso-scan package, but do not
> even allow specification of the ISO image file via a boot parameter.
>
>> When installing from the Debian netinstall ISO using Grub2's loopback
>> method, and manually mounting the ISO to /cdrom, the installation
>> process gets stuck at apt-setup
>
> I suggest to you that if you copy the kernel and initrd files out of the
> ISO image, into the USB flashdrive(?) filesystem, you do not have to use
> GRUB, you can just use Syslinux, as the ISO usually does. This means
> that you can potentially use the existing Syslinux menu system, if this
> is also copied out of the ISO image.
>
> Could you tell me what exactly is the significance of
> "Grub2-Multiboot-ISO-USB"? Is there some package or utility by that
> name?
>
>
> -- Ian Bruce


Hi Ian,

thank you for this information on bug #724931. I read your posts and it 
seems that the developers are aware of the problem, but choose not to act:

On Thu, 5 Sep 2013 11:44:54 -0400
Joey Hess <joey@kitenet.net> wrote:
> iso-scan is part of the Debian installer[1].
>
> However, it is only included in the hd-media initrd. There is no
> reason to include it on the regular CD initrd, because isohybrid
> allows mounting the USB stick directly.

How can isohybrid replace the findiso option? At least for me it makes a 
huge difference, whether I can use my 32 GB stick for ONE netinst.iso or 
ONE HUNDRED. Aside from that it is an unnecessary complexity to download 
the hd-media initrd, which is why I never did that, but instead only 
downloaded the loop.ko.
By the way, I think that it is reason enough to include findiso on the 
regular CD, when several persons request it. One always has to balance 
gain and cost and the only cost that I can see, is that the ISO will be 
larger. I don't know how big findiso is, but probably less then 100 kB. 
The loop.ko file is 37 kB, so together this gives 137 kB. Since the 
netinst.iso is about 270 MB, it would grow by about 0.05 %. Who will be 
hurt by that? On the other hand according to many post etc. on the 
subject, which I read in the course of the last two years or so, I 
imagine that a lot of people would like it. I certainly would have 
installed Debian earlier, if this option had been included on the 
netinst.iso.

On 30.09.2013 01:58, wrote ian_bruce@fastmail.net:
> As for the iso-scan package, if you search the source code for the
> string "findiso", you will not find it.

I don't know about the hd-media initrd, but there is a debian live ISO at:
http://live.debian.net/
There the option 'findiso=$iso' works as advertised and indeed:
$ zcat initrd.img | strings | grep -i findiso
	if [ -n "$FINDISO" ] && [ "${TORAM}" ]
		if is_mountpoint /root/lib/live/mount/findiso
			umount /root/lib/live/mount/findiso
			rmdir --ignore-fail-on-non-empty /root/lib/live/mount/findiso \
			findiso=*)
				FINDISO="${_PARAMETER#findiso=}"
				export FINDISO
		if [ -n "${FINDISO}" ]
			if [ -f ${mountpoint}/${FINDISO} ]
				mkdir -p /live/findiso
				mount -t ${fstype} -o ro,noatime "${devname}" /live/findiso
				loopdevname=$(setup_loop "/live/findiso/${FINDISO}" "loop" 
"/sys/block/loop*" 0 "")


> I suggest to you that if you copy the kernel and initrd files out of the
> ISO image, into the USB flashdrive(?) filesystem, you do not have to use
> GRUB, you can just use Syslinux, as the ISO usually does. This means
> that you can potentially use the existing Syslinux menu system, if this
> is also copied out of the ISO image.

The reason I use Grub2 is that updating an ISO is very easy: Just 
replace the old one with the new version (and eventually change the name 
in the grub.cfg). If I extracted the kernel and initrd, I would have to 
do this on every update as well. Luckily, most of the ISOs I use come 
with a grub.cfg, so I can copy the entries from there and adapt them, 
e.g. with a findiso option or options for my locales.

> Could you tell me what exactly is the significance of
> "Grub2-Multiboot-ISO-USB"? Is there some package or utility by that
> name?
This should just be a compact description of what I did:
 - I installed Grub2 to the USB stick via grub-install.
 - Then I copied some ISOs to the $USB/ISO folder.
 - After that I adapted $USB/boot/grub/grub.cfg to get a Multiboot USB 
stick, that boots directly from the ISOs.

I had tried various tools for this in the past (not yours, though) and 
was not quite satisfied with them. The above mentioned method seems to 
be the most efficient one for me.



Conclusion:
 - I see no reason NOT to include findiso and loop.ko on every 
installer CD.
 - Hopefully Joey can give us an update of his thoughts on the matter.

Best regards,
Andreas

Message #15 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: joey@kitenet.net, debian-boot@lists.debian.org, 724931@bugs.debian.org

Subject: Re: loop-mounted ISO images

Date: Mon, 30 Sep 2013 05:52:22 -0700

[Message part 1 (text/plain, inline)]

On Mon, 30 Sep 2013 13:10:01 +0200
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:

> <joey@kitenet.net> wrote:
>
>> iso-scan is part of the Debian installer.
>>
>> However, it is only included in the hd-media initrd. There is no
>> reason to include it on the regular CD initrd, because isohybrid
>> allows mounting the USB stick directly.
> 
> How can isohybrid replace the findiso option? At least for me it makes
> a huge difference, whether I can use my 32 GB stick for ONE
> netinst.iso or ONE HUNDRED.

I said the same thing, in my post to the mailing list:

    This is unnecessarily destructive, and makes it hard to install
    multiple distributions on the same flashdrive, or to use it for
    other purposes. The smallest flashdrive you can currently buy is
    8GB; it makes no sense that you would have to have a different one
    for every live ISO you might want to use.

> Aside from that it is an unnecessary complexity to download the
> hd-media initrd, which is why I never did that, but instead only
> downloaded the loop.ko.

Even if you did download the hd-media initrd, it still wouldn't allow
you to use a boot parameter to specify the pathname of the ISO image
file you want to use. It appears that it just grabs anything it can find
that looks like a Debian ISO, and asks the user to confirm it.

Note that "loop.ko" is included on the ISO (but not the initrd), in the
form of /pool/main/l/linux/loop-modules-*.udeb packages.

> By the way, I think that it is reason enough to include findiso on the
> regular CD, when several persons request it. One always has to balance
> gain and cost and the only cost that I can see, is that the ISO will
> be larger. I don't know how big findiso is, but probably less then 100
> kB. The loop.ko file is 37 kB, so together this gives 137 kB. Since
> the netinst.iso is about 270 MB, it would grow by about 0.05 %. Who
> will be hurt by that?

My patch, which seems to do everything that's necessary, is a few
hundred BYTES (uncompressed). So including that and the "loop.ko" module
in the initrd will increase the size of the ISO image by about one part
in ten thousand, which certainly doesn't seem worth worrying about.

> On the other hand according to many post etc. on the subject, which I
> read in the course of the last two years or so, I imagine that a lot
> of people would like it. I certainly would have installed Debian
> earlier, if this option had been included on the netinst.iso.

As you point out next, it appears that this functionality is included on
the Debian live CD. What possible rationale could there be for having it
there, but not on the installer image?

I actually didn't think to investigate the live CD. It never occurred to
me that one might have it, and the other not.

> <ian_bruce@fastmail.net> wrote:
>
>> As for the iso-scan package, if you search the source code for the
>> string "findiso", you will not find it.
> 
> I don't know about the hd-media initrd, but there is a debian live ISO
> at: http://live.debian.net/
>
> There the option 'findiso=$iso' works as advertised

The hd-media initrd is no different from the ones on the standard
installer ISO, in this regard:

    $ zcat debian-7.1.0-amd64-i386-netinst/install.{386,amd}/{,gtk/}initrd.gz |
          strings | grep -i findiso
    $ 

    $ zcat debian-7.1.0-amd64-hd-media/{,gtk/}initrd.gz |
          strings | grep -i findiso
    $ 

My "loopmount=" patch is again attached to this message, for the sake of
the bug report.


-- Ian Bruce

[debian-iso-loopmount.diff (text/x-patch, attachment)]

Message #20 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: joey@kitenet.net, 724931@bugs.debian.org

Subject: Patch works great

Date: Mon, 30 Sep 2013 21:54:15 +0200

tags 724931 patch
thanks

Hi,

On 30.09.2013 14:52, wrote ian_bruce@fastmail.net:
> Note that "loop.ko" is included on the ISO (but not the initrd), in the
> form of /pool/main/l/linux/loop-modules-*.udeb packages.

Thank you for mentioning this. I didn't look there.

I have applied your patch to the debian-testing-amd64-netinst.iso and 
also included the loop.ko. This increased the size from 230.7 MB to 
232.3 MB, i.e. by 0.7 %.
The patch itself is very straightforward, short, and most importantly 
works very well. Thank you!

The only missing thing is 'LOOPFS=' for kfreebsd and hurd.

Therefore I would suggest to forget about the findiso option and simply 
apply this patch to all the Debian isos. (For comptability one could 
rename 'loopmount=' into 'findiso='.)


Best regards,
Andreas

Message #25 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: joey@kitenet.net, 724931@bugs.debian.org

Subject: Re: Patch works great

Date: Mon, 30 Sep 2013 14:16:31 -0700

On Mon, 30 Sep 2013 21:54:15 +0200
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:

> I have applied your patch to the debian-testing-amd64-netinst.iso and
> also included the loop.ko. This increased the size from 230.7 MB to
> 232.3 MB, i.e. by 0.7 %.

This must be because of differences in compression, or something else in
the build method. The combined size of the patch code and the loop.ko
module is about 37KB (uncompressed), not 1.6MB. In other words, it's
completely negligible.

> The patch itself is very straightforward, short, and most importantly
> works very well. Thank you!

I'm glad you found it satisfactory.

> The only missing thing is 'LOOPFS=' for kfreebsd and hurd.

Yes, I pointed that out in my original post to the mailing list. I will
forward that to the bug report, for completeness.

> Therefore I would suggest to forget about the findiso option and
> simply apply this patch to all the Debian isos.

I certainly have no objection to that, although I think it would be a
good thing if the install ISOs and the live ISOs both used the same
loopmount mechanism.

> (For comptability one could rename 'loopmount=' into 'findiso='.)

It might be better not to do that, in case there are any slight
differences in function, which is to say, "incompatibilities".

I think that "loopmount" is more descriptive of what is happening than
"findiso", anyway.


-- Ian Bruce

Message #30 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: 724931@bugs.debian.org

Subject: Fw: PATCH: specify pathname for ISO loop-mount

Date: Mon, 30 Sep 2013 15:03:11 -0700

This message describes the motivation for the
"debian-iso-loopmount.diff" patch, which is reproduced above.
It was originally posted here:

http://lists.debian.org/debian-boot/2013/09/msg00509.html


Begin forwarded message:

Date: Sat, 28 Sep 2013 03:08:48 -0700
From: <ian_bruce@fastmail.net>
To: debian-boot@lists.debian.org
Subject: PATCH: specify pathname for ISO loop-mount


This patch introduces a boot parameter, "loopmount=", which allows the
specification of a full pathname (relative to the root directory of some
block device) of an ISO image file which should be loop-mounted as the
Debian installer root filesystem.

The main purpose of this feature is to facilitate the creation of USB
flashdrives which can contain multiple Linux live distributions,
selectable at boot time via a Syslinux menu.

http://sourceforge.net/projects/multibootusb/

The current method of creating a Debian installer flashdrive overwrites
the existing partition table and filesystem:

    *Warning*

    The procedures described in this section will destroy anything
    already on the device!

http://www.debian.org/releases/stable/amd64/ch04s03.html.en

This is unnecessarily destructive, and makes it hard to install multiple
distributions on the same flashdrive, or to use it for other purposes.
The smallest flashdrive you can currently buy is 8GB; it makes no sense
that you would have to have a different one for every live ISO you might
want to use.

The effect of this patch is that you can just copy the Debian installer
ISO into the existing filesystem on the flashdrive, and boot it with
GRUB or Syslinux, using the "loopmount=" boot parameter to specify the
ISO pathname.

It will be apparent from reading the patch that if this parameter is not
specified, everything functions EXACTLY as before; the new code will not
be run. The changes amount to no more than forty lines of new code, in
one file.

I suppose that this patch applies to the "cdrom-detect" udeb, although I
just patched the initrd directly. A dependency on "loop-modules" will
need to be added, to make sure that the "loop.ko" kernel module gets
copied into the initrd.

One issue that probably should be addressed is what to do if the
specified ISO is not actually present; the boot process should fail
gracefully, but I'm not sure how best to accomplish that.

Minor changes (see the "LOOPFS" variable) would be needed to make it
work for FreeBSD and Hurd; I've only tried it with Linux.

Comments are welcome, but apart from the above, I don't think there's
much wrong with this patch; testing shows that it seems to work
perfectly.


-- Ian Bruce

Message #35 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: joey@kitenet.net, 724931@bugs.debian.org

Subject: Problem with apt-setup

Date: Tue, 01 Oct 2013 11:29:46 +0200

[Message part 1 (text/plain, inline)]

Hi,

The size difference in the ISO was due to an additional vmlinuz (2.5 MB) 
in /install.amd/gtk.
If have created a shell script to patch an existing ISO and attached it 
to this mail.
This script has to be executed with root privileges!
It expects to find a debian-testing-amd64-netinst.iso mounted to /mnt 
and the patch file debian-iso-loopmount.diff in the directory where it 
is executed.
It will create a patched ISO named 
debian-testing-amd64-netinst-loopmount.iso in the directory where it is 
executed.
The size of this patched ISO is 229,8 MB, i.e. less than the original 
size, which is probably due to better compression.

On 30.09.2013 23:16, wrote ian_bruce@fastmail.net:
> I certainly have no objection to that, although I think it would be a
> good thing if the install ISOs and the live ISOs both used the same
> loopmount mechanism.

This is why I suggested the renaming of 'loopmount=' into 'findiso=', 
but I agree, that this is bad, because of possible different behaviour. 
I think the live ISO should switch to 'loopmount=', because it is a very 
small patch, except, of course, if there are any features 'findiso=' has 
that are not provided by 'loopmount='.

While testing the patched ISO, I noticed a problem very similar to bug 
#724933: apt-setup is not working correctly. The error message was:

main-menu[524]: INFO: Menu item 'apt-setup-udeb' selected
apt-setup: umount: can't umount /target/media/cdrom0: Invalid argument
apt-setup: Using CD-ROM mount point: /media/cdrom/
apt-setup: Identifying..
apt-setup: [ae8b99d023e59c58db825bbd443912e2-2]
apt-setup: Scanning disc for index files..
apt-setup: Found 0 package indexes, 0 source indexes, 0 translation 
indexes and 0 signatures
apt-setup: W: Failed to mount '/dev/sr0' to '/media/cdrom'
apt-setup: E: Unable to locate any package files, perhaps this is not a 
Debian Disc or the wrong architecture?
kernel: [1392,717219] ISO 9660 Extensions: Microsoft Joliet Level 3
kernel: [1392,717334] ISO 9660 Extensions: RRIP_1991A

After continuing in the menu, the following line is added:
apt-setup: warning: /usr/lib/apt-setup/generators/40cdrom returned error 
code 1; discarding output

apt-setup tries to create a list entry in /etc/sources.list for the CD. 
I think this entry should be removed at the end of installation. Is that 
correct? (It didn't on my last installation.)
Assuming the entry is only valid during installation, apt-setup should 
simply look in /cdrom (or create a symlink from /cdrom to 
/target/media/cdrom0) and not try to (u)mount anything.
I consider this a bug in apt-setup that should be remedied whether or 
not the loopmount patch is included.

This bug is no big problem for the netinstall ISO, because it does not 
load any more packages from the ISO, but for a full ISO this is a 
critical problem.
Have you ever tried the patch with a full image?

Best regards,
Andreas

[Apply-Patch.sh (application/x-shellscript, attachment)]

Message #40 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: 724931@bugs.debian.org

Cc: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, debian-boot@lists.debian.org, Frédéric Boiteux <fboiteux@free.fr>, Frans Pop <elendil@planet.nl>, Krzysztof Mazurczyk <k.mazurczyk@update.poznan.pl>

Subject: PATCH: improved ISO loopmount option

Date: Fri, 4 Oct 2013 02:27:31 -0700

[Message part 1 (text/plain, inline)]

This is an improved version of the patch I originally posted to this bug
report. It applies mainly to the "cdrom-detect" udeb, although I suggest
that the ISO volume label (such as "Debian 7.1.0 M-A 1") be included
somewhere in the initrd; currently it does not appear to be. As I said
previously, a dependency on "loop-modules" will have to be added, so
that the "loop.ko" kernel module gets copied into the initrd.

This patch adds support for a boot parameter, "loopmount=", which
specifies an ISO image file to be loaded from another filesystem, and
used to boot from. For example:

    loopmount=KINGSTON:/linux/debian-7.1.0-amd64-i386-netinst.iso

This directs the initrd to look for a block device filesystem with the
label "KINGSTON" (as reported by /sbin/blkid), and to use the file
"debian-7.1.0-amd64-i386-netinst.iso", in the subdirectory "/linux" (the
leading "/" is not actually necessary) as a boot image.

In addition to the filesystem labels found in /dev/disk/by-label/, the
block device can be identified by UUID, as found in /dev/disk/by-uuid/.

If the filesystem label/UUID (and ":") is not specified, then all
available partitions and CD devices will be searched for the specified
ISO image file, in the specified directory.

If the "loopmount" parameter is not used, then the initrd will search
for a direct ISO filesystem in the same way as previously, although any
block device with the expected volume label ("Debian 7.1.0 M-A 1") will
be tried first.

It is expected that this facility will be most useful with USB
flashdrives, although it is in no way limited to those. It can be
applied to any block device with a vfat, ext{2,3,4}, or iso9660
filesystem; others could easily be added.

Testing seems to show that the patch works well; the relevant portion of
the installation log is reproduced below.


Oct  4 06:02:19 cdrom-detect: Searching for Debian installation media...
Oct  4 06:02:19 cdrom-detect: Devices: '/dev/sr0'
Oct  4 06:02:19 cdrom-detect: LOOPDEV='KINGSTON' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso'
Oct  4 06:02:19 cdrom-detect: trying loopmount on (/dev/disk/by-label/KINGSTON)...
Oct  4 06:02:19 kernel: [  226.383202] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  4 06:02:19 kernel: [  226.395607] loop: module loaded
Oct  4 06:02:19 cdrom-detect: CD-ROM mount succeeded: device=/loop/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660
Oct  4 06:02:19 kernel: [  226.474076] ISO 9660 Extensions: Microsoft Joliet Level 3
Oct  4 06:02:19 kernel: [  226.474149] ISO 9660 Extensions: RRIP_1991A
Oct  4 06:02:19 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44'
Oct  4 06:02:19 cdrom-detect: Detected CD with 'stable' (wheezy) distribution


This patch will increase the size of the (uncompressed) initrd by about
37KB: 36KB for the required "loop.ko" module, and 1KB of extra boot
script. In a 200MB or 300MB ISO image, this is surely immaterial.

Since the patch adds a previously unavailable feature, which will not
operate unless activated by an explicit boot parameter, there seems to
be no reason why it should not be applied. Compare it with the current
procedure for installing Debian from a USB flashdrive:

http://www.debian.org/releases/stable/amd64/ch04s03.html

As an administrative note, I recommend that this bug report be retitled;
the current title no longer reflects what we are discussing, and the
availability of my patch is surely now the most important issue relating
to it.


-- Ian Bruce

[debian-iso-loopmount.2.diff (text/x-patch, attachment)]

Message #45 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: 724931@bugs.debian.org, debian-boot@lists.debian.org, Frédéric Boiteux <fboiteux@free.fr>, Frans Pop <elendil@planet.nl>, Krzysztof Mazurczyk <k.mazurczyk@update.poznan.pl>

Subject: Re: PATCH: improved ISO loopmount option

Date: Fri, 04 Oct 2013 12:45:24 +0200

Hi,

@Ian: Thanks for working on this feature.

On 04.10.2013 11:27, ian_bruce@fastmail.net wrote:
> Testing seems to show that the patch works well; the relevant portion of
> the installation log is reproduced below.

Sadly it only "seems" to work well, but in fact, your previous (and I 
assume also this) patch break apt-setup trying to add the CD to 
/etc/sources.list. I am currently working on this problem and hope to 
finish this soon.

Furthermore, for consistency I suggest to rename the boot option to 
'findiso=', since the live image already uses this option. They use a 
implementation fairly similar to your first patch and I am sure, that 
they would gladly add any additional features implemented in the 
debian-installer.
The iso-scan package on the other hand seems not be in use currently.
(I really don't care for the name, but it should be the same for all 
Debian ISOs. It's bad enough, that this is inconsistent between 
distributions!)

Best regards,
Andreas

Message #50 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org

Subject: Re: PATCH: improved ISO loopmount option

Date: Fri, 4 Oct 2013 06:02:46 -0700

On Fri, 04 Oct 2013 12:45:24 +0200
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:

> Sadly it only "seems" to work well, but in fact, your previous (and I
> assume also this) patch break apt-setup trying to add the CD to
> /etc/sources.list. I am currently working on this problem and hope to
> finish this soon.

I'm sorry to hear that. I'll look into it as well; the error log you
included in your previous message seemed to indicate that assumptions
were being made about what block device represented the ISO, or where it
was mounted. Since the "cdrom-detect" script already figures this out,
this information just needs to be exported (via a file?) to other
scripts.

> Furthermore, for consistency I suggest to rename the boot option to
> 'findiso=', since the live image already uses this option. They use a
> implementation fairly similar to your first patch

The effect of my second patch is that the ISO no longer has to be
"found"; we can specify EXACTLY where (what filesystem, what pathname)
it is located, and so that parameter name is now even more
inappropriate, and that implementation quite incompatible with mine.

> and I am sure, that they would gladly add any additional features
> implemented in the debian-installer.

I'm not at all sure about that. I've previously found them to be quite
unwilling to make obviously beneficial changes, without being able to
provide any rational reason for that refusal.

http://lists.debian.org/debian-live/2013/08/threads.html#00054
http://lists.debian.org/debian-live/2013/09/threads.html#00000

And notice that so far, loopmount ISN'T implemented in the Debian
installer; this discussion doesn't yet involve anybody except you and
me.

> The iso-scan package on the other hand seems not be in use currently.

It turns out that it is, but only in the "hd-media" images, not the
standard ISOs.

http://lists.debian.org/debian-boot/2013/09/msg00097.html

Which as you and I have both pointed out, is actually kind of dumb.

> (I really don't care for the name, but it should be the same for all
> Debian ISOs. It's bad enough, that this is inconsistent between
> distributions!)

I agree on both counts, but as I said above, there's not much point in
making the names consistent, if the behavior isn't.

For example, the iso-scan package in Ubuntu allows you to specify, via a
boot parameter, the pathname for the image file, whereas the Debian
version apparently doesn't, even if it is included in the initrd. So it
would actually be better if the names WERE different, so as not to
delude people into thinking that the features are the same.

On another issue, I have some advice about your ISO-patching script. (I
don't patch ISOs, but only initrds, because I prefer that the official
MD5 and SHA256 hashes can still be verified.) You said it had to be
executed with root privileges, presumably so that file ownership and
device files come out right. You should investigate the "fakeroot"
utility, which obviates the need for this.

And is it possible to rename the bug report to something more
descriptive?


-- Ian Bruce

Message #55 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: Christian Perrier <bubulle@debian.org>, 724931@bugs.debian.org, 724933@bugs.debian.org

Subject: Patch for bugs #724931 and #724933

Date: Sun, 06 Oct 2013 04:06:17 +0200

[Message part 1 (text/plain, inline)]

tags 724931 patch
tags 724933 patch
thanks

Hi,

I made some changes/improvements to the patch provided by Ian:
 * Support a 'firmware' folder on the same device as the ISO: Due to 
being mounted, it was not automatically checked with the old patch.
 * Removed "DISTRIB_LABEL" variable: Such things should not be 
hard-coded! Furthermore, this is unrelated to the 'loopmount=' option 
(and I don't know what the benefit of this should be). Ian, if you want 
something like this you probably should open another bug and explain 
there the need for this further.
 * Added support for kfreebsd/hurd and any filesystem by using blkid to 
detect the filesystem of the device to be mounted.
 * Added iteration loop over $dev_given (i.e. is it used as 
'loopmount=DEVICE:ISO'), "usb-partition", "cd" and "partition". This is 
to ensure, that the most likely sources are looked at first.
 * Changed/added log messages
 * Added comments
 * Corrected formatting and removed trailing whitespaces

The new apt-cdrom-setup patch enables the ISOs to be added to 
/etc/apt/sources.list:
 * Don't (u)mount for ISOs
 * Added a if-clause to the chroot_cleanup_localmounts hack, to prevent 
it from moving a file, that is possibly not even there (especially since 
it get's removed in 40cdrom and 41cdset).
 * Added cd-set support: ISOs from the set (i.e. starting with the same 
string up to 'CD-' (or 'DVD-' for DVDs resp. 'BD-' for bluerays)) in the 
same directory can be loaded. (The entries for the additional files are 
removed, when the installation is finished.)

I would also suggest to add a loopback.cfg (similar to the one I 
attached) to /boot/grub/. (By the way, I noticed that the /boot folder 
is missing from the debian-7.1.0-i386-netinst.iso. It should be added 
there.) If it is possible, this should also be done for /isolinux. Then 
it can be loaded by grub2 with e.g.:
menuentry "Debian Testing Netinstall (64 bit)" {
	set gfxpayload=auto
	set iso_path="/ISO/debian-testing-amd64-netinst.iso"
	export iso_path
	boot_options=locale=de_DE.UTF-8
	loopback loop $iso_path
	set root=(loop)
	configfile /boot/grub/loopback.cfg
	loopback -d loop
}

I also updated the script to apply the patches for testing purposes:
 This script has to be executed with root privileges!
 It's first argument is the path/filename of the Debian install ISO 
file to be patched.
 The filename should contain amd64 or i386, indicating which 
architecture is used.
 The ISO will be mounted to /mnt, which will be umounted before.
 If a loopback.cfg is in the directory where the script is executed, it 
will be included in /boot/grub/.
 If the architecture is i386, every '.amd' in the loopback.cfg will be 
changed to '.386'.
 The script's second argument is the patch file to be applied to initrd 
(e.g. initrd_loopmount.patch).
 It's third argument is the patch file to be applied to the 
apt-cdrom-setup udeb (e.g. apt-cdrom-setup_loopmount.patch).
 It will create a patched ISO named ${ISO}-loopmount.iso in the 
directory where it is executed.

I tested with the following ISOs:
 * debian-testing-amd64-netinst-loopmount.iso
 * debian-7.1.0-i386-netinst-loopmount.iso
 * debian-7.1.0-amd64-netinst-loopmount.iso
 * debian-7.1.0-amd64-CD-1.iso
 * debian-7.1.0-amd64-CD-2.iso (+)
 * debian-7.1.0-amd64-CD-3.iso (+)
 * debian-7.1.0-amd64-DVD-1-loopmount.iso
 * debian-7.1.0-amd64-DVD-2-loopmount.iso (+)
(+) These ISOs were not modified, but just copied to the same directory 
as the first ISO of the set on the USB stick.

Problems that I noticed with this patch:
 * The CD-set installation (CD 1, 2 and 3) works well, until it 
actually tries to install the packages. Then tasksel fails, complaining 
about unauthenticated packages:
   For the installation to continue, one has to change the following 
line in /target/usr/bin/tasksel
	push @cmd, qw{apt-get -q -y -o APT::Install-Recommends=true -o 
APT::Get::AutomaticRemove=true install};
   to:
	push @cmd, qw{apt-get -q -y --force-yes -o APT::Install-Recommends=true 
-o APT::Get::AutomaticRemove=true install};
   This seems OK, since I changed a package in the original CD and thus 
the Release.gpg is not a good signature. But I did not change anything 
on CD-2 or CD-3 that I loaded. Therefore this error should also occur if 
only installing from the first CD, but it does not: the installation 
works fine (though without print-server and laptop task)!
   I'm not sure what the correct behaviour should be, but this is 
clearly inconsistent: Either it should reject installation from the 
first CD, since that is the 'bad' one, or it should also allow 
installation with multiple CDs. Perhaps the later is the case, but I 
misconfigured the additional CDs? I had to use 'deb file:' for this, 
since when using 'deb cdrom:', the installer stops, when the next CD 
needs to be mounted, and I found no way to automate this. Apparently apt 
does not support the concept of multiple CD drives.

 * When the installation (including gnome desktop) from the CD set (CDs 
1, 2 and 3) or the DVD-1 is nearly finished, the error
   "Could not build the hash file for en-common"
   occurs and it is suggested, that one sends a complaint to the 
maintainer of the package that provides 'en'.
   The error message in /var/log/syslog are the following two lines:
   aspell-autobuildhash: processing: en [en-common]
   Error: /dev/null:1: The key "/usr/bin/aspell" is unknown.
   I think the error is unrelated to this patch, but I am not sure, 
since I cannot test it with the original ISO, because that doesn't 
support multiple CDs when booting from USB (and my spare USB for 
isohybrid testing is too small for the DVD).

On 04.10.2013 15:02, ian_bruce@fastmail.net wrote:
> The effect of my second patch is that the ISO no longer has to be
> "found"; we can specify EXACTLY where (what filesystem, what pathname)
> it is located, and so that parameter name is now even more
> inappropriate, and that implementation quite incompatible with mine.
I agree that now the functionality is (from the user's point of view) 
different to that in the live image, because of the possibility to 
specify the device by "loopmount=DEVICE:ISO". Using such a string with 
the findiso from the live image surely doesn't work.
So I think the best is to use the name loopmount and when (if?) the new 
functionality is added to the live image, they should change their name 
accordingly.

>> and I am sure, that they would gladly add any additional features
>> implemented in the debian-installer.
>
> I'm not at all sure about that. I've previously found them to be quite
> unwilling to make obviously beneficial changes, without being able to
> provide any rational reason for that refusal.
>
> http://lists.debian.org/debian-live/2013/08/threads.html#00054
> http://lists.debian.org/debian-live/2013/09/threads.html#00000
I read the discussion and let me add my opinion: I think the Debian live 
team wants to avoid any additional work, which is quite understandable, 
since they probably have enough to do already (and do quite a good job). 
But in this special case I wonder, if it wouldn't have been less work 
just to include the two packages than to write all those mails...

> And notice that so far, loopmount ISN'T implemented in the Debian
> installer; this discussion doesn't yet involve anybody except you and
> me.
I think (and very much hope) that a number of people on the debian-boot 
mailing list also read this discussion and are therefore 'involved'.
Before, the patch had problems, which are solved now. So I think this 
patch is ready for being included in all the Debian installers, since 
from the remaining two problems, that I could detect with this patch, 
one is probably solved by an official build (Release.gpg issue), and the 
other is apparently a bug in a different package (aspell).

When the patch is included, the documentation for installation from USB 
needs to be updated.

>> The iso-scan package on the other hand seems not be in use currently.
>
>It turns out that it is, but only in the "hd-media" images, not the
>standard ISOs.
Oh, I forgot the hd-media images, because ... ugh ... well ... they are 
not included on any Debian installation ISO! ;)

> (I don't patch ISOs, but only initrds, because I prefer that the
> official MD5 and SHA256 hashes can still be verified.)
I also prefer the official signed hashes, but this script is meant for 
testing purposes only, and will be obsolete, as soon as this patch is 
included in the Debian installer.

> You said it had to be
> executed with root privileges, presumably so that file ownership and
> device files come out right. You should investigate the "fakeroot"
> utility, which obviates the need for this.
The script attached to this mail needs root privileges to (u)mount the 
ISO, so it wouldn't make much difference, if I used fakeroot for the rest.

> And is it possible to rename the bug report to something more
> descriptive?
Feel free to rename it to whatever you think is best by sending a mail 
with the following content ('NEW_TITLE' appropriately replaced) to 
control@bugs.debian.org:
retitle 724931 NEW_TITLE
thanks

I hope the loopmount option will soon be in the official Debian 
installation ISOs.

Best regards,
Andreas

[Apply-Patches.sh (application/x-shellscript, attachment)]

[initrd_loopmount.patch (text/x-patch, attachment)]

[apt-cdrom-setup_loopmount.patch (text/x-patch, attachment)]

[loopback.cfg (text/plain, attachment)]

Message #62 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: 724931@bugs.debian.org

Cc: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Subject: PATCH: improved(2) ISO loopmount option

Date: Sun, 6 Oct 2013 06:04:40 -0700

[Message part 1 (text/plain, inline)]

(I see that Andreas has recently posted a set of patches. The patches I
have attached below are not based on that work, although they address
some of the same problems. At the end of this message, are some comments
about how our alternative solutions might be combined.)


On Fri, 4 Oct 2013 06:02:46 -0700
<ian_bruce@fastmail.net> wrote:

> Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:
> 
>> Sadly it only "seems" to work well, but in fact, your previous (and I
>> assume also this) patch break apt-setup trying to add the CD to
>> /etc/sources.list. I am currently working on this problem and hope to
>> finish this soon.
> 
> I'm sorry to hear that. I'll look into it as well; the error log you
> included in your previous message seemed to indicate that assumptions
> were being made about what block device represented the ISO, or where
> it was mounted. Since the "cdrom-detect" script already figures this
> out, this information just needs to be exported (via a file?) to other
> scripts.

I think the problem is related to this bug:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608201

I have therefore adopted the (partial?) solution which was applied in
that case, which is that ISO images which are found somewhere other than
an actual CD, do not get included in "sources.list". Which seems kind of
unfortunate; the same USB flashdrive could be used again, or an actual
CD might show up later, but that's how they resolved it, and it's not
what we're mainly concerned with right now. If a better solution is
developed sometime later for those situations, it should apply to the
loopmount case as well.

So for now, the solution is to export the fact of the loopmount to the
configuration database, where it can later be checked by the "apt-setup"
package, and handled the same way as the other non-CD cases.

Two patchfiles are attached: a slightly altered one for "cdrom-detect",
and a new one for "apt-setup". I haven't actually tested the latter,
because I'm working with the "netinst" ISO, which doesn't seem to
include that package.

Here are the relevant sections of the installer logs:


    "loopmount=KINGSTON:/linux/debian-7.1.0-amd64-i386-netinst.iso"

Oct  6 07:35:06 cdrom-detect: Searching for Debian installation media...
Oct  6 07:35:06 cdrom-detect: removable devices: (/dev/sr0)
Oct  6 07:35:06 cdrom-detect: LOOPDEV='KINGSTON' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso'
Oct  6 07:35:06 cdrom-detect: trying loop-mount on (/dev/disk/by-label/KINGSTON)...
Oct  6 07:35:06 kernel: [  322.483186] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  6 07:35:06 kernel: [  322.495521] loop: module loaded
Oct  6 07:35:06 kernel: [  322.571304] ISO 9660 Extensions: Microsoft Joliet Level 3
Oct  6 07:35:06 cdrom-detect: CD-ROM loop-mount succeeded: device=/dev/disk/by-label/KINGSTON/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660
Oct  6 07:35:06 kernel: [  322.578804] ISO 9660 Extensions: RRIP_1991A
Oct  6 07:35:06 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44'
Oct  6 07:35:06 cdrom-detect: Detected CD with 'stable' (wheezy) distribution


    "loopmount=/linux/debian-7.1.0-amd64-i386-netinst.iso"

Oct  6 08:01:19 cdrom-detect: Searching for Debian installation media...
Oct  6 08:01:19 cdrom-detect: removable devices: (/dev/sr0)
Oct  6 08:01:19 cdrom-detect: LOOPDEV='' LOOPFILE='/linux/debian-7.1.0-amd64-i386-netinst.iso'
Oct  6 08:01:19 cdrom-detect: trying loop-mount on (/dev/sda1 /dev/sda10 /dev/sda2 /dev/sda3 /dev/sda4 /dev/sda5 /dev/sda6 /dev/sda7 /dev/sda8 /dev/sda9 /dev/sdb1 /dev/sdb10 /dev/sdb2 /dev/sdb3 /dev/sdb4 /dev/sdb5 /dev/sdb6 /dev/sdb7 /dev/sdb8 /dev/sd
Oct  6 08:01:19 kernel: [  211.616937] FAT-fs (sda1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  6 08:01:19 kernel: [  211.664572] FAT-fs (sda10): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  6 08:01:19 kernel: [  211.740574] FAT-fs (sda2): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  6 08:01:19 kernel: [  211.804567] FAT-fs (sda3): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!

...

Oct  6 08:01:22 kernel: [  214.316771] FAT-fs (sdf1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
Oct  6 08:01:22 kernel: [  214.329130] loop: module loaded
Oct  6 08:01:22 kernel: [  214.399019] ISO 9660 Extensions: Microsoft Joliet Level 3
Oct  6 08:01:22 cdrom-detect: CD-ROM loop-mount succeeded: device=/dev/sdf1/linux/debian-7.1.0-amd64-i386-netinst.iso fstype=iso9660
Oct  6 08:01:22 kernel: [  214.406518] ISO 9660 Extensions: RRIP_1991A
Oct  6 08:01:22 cdrom-detect: Detected CD 'Debian GNU/Linux 7.1.0 "Wheezy" - Official Multi-architecture amd64/i386 NETINST #1 20130615-23:44'
Oct  6 08:01:22 cdrom-detect: Detected CD with 'stable' (wheezy) distribution


(Can these "filesystem will be case sensitive" log messages be shut off,
somehow? Isn't that kind of the point of VFAT?)

-------

Here are my comments on Andreas' latest patches, based on a quick
reading:

I don't know if you have looked at bug #608201; it seems like our
current problem is just another specific case of that general situation,
which awaits a general solution. You can see how it was dealt with
previously by searching for the strings "hybrid" and "usb-hdd" in the
file "apt-setup/generators/40cdrom". It appears that your changes to
that file are similar to mine; however, I think mine are preferable, in
this regard:

I see that you have copied (in this and several other places) this piece
of code, for determining if a loop-mount is being used:

    for arg in $(cat /proc/cmdline); do
        case $arg in
        loopmount=*)
            LOOPMOUNT=${arg#loopmount=}
            LOOPFILE=${LOOPMOUNT#*:}
            [ "$LOOPFILE" != "$LOOPMOUNT" ] && LOOPDEV=${LOOPMOUNT%:*}
            ;;
        esac
    done

Based on my current patches, this is now unnecessary. Do this instead:

    loopmount=$(db_getval cdrom-detect/cdrom_loopdev)

    if [ "$loopmount" ] ; then ... ; else ... ; fi

"$loopmount", if non-null, will contain the actual device (not pathname,
see "$(db_getval cdrom-detect/cdrom_device)" for that) on which the ISO
image file resides, if that is ever useful.

I'm not sure if I understand this:

    if [ "$ROOT" ] && [ "$LOOPMOUNT" ]; then
        save_label
    fi

Does that mean that the ISO will be specified in "sources.list"? If so,
that's valuable. Can that be made to work for the "hybrid" and "usb-hdd"
cases as well? (Search for those strings in "cdrom-detect.postinst" to
see what they mean.)

Your fix for locating firmware images is welcome, with the same comment
as above, about detecting loop-mounts. However, I did not see any
benefit in your changes to "cdrom-detect.postinst". It's easy enough for
somebody to add the proper filesystem types for Hurd and FreeBSD. My
current patch produces slightly more informative log entries, if you
care about that. (see above for examples)

Your work on CD-sets is, of course, important; my patches do not address
that issue.


-- Ian Bruce

[debian-iso-loopmount.3.diff (text/x-patch, attachment)]

[apt-setup.diff (text/x-patch, attachment)]

Message #67 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: 724931@bugs.debian.org, Ian Bruce <ian_bruce@fastmail.net>

Subject: Final? patch for bug #724931

Date: Tue, 08 Oct 2013 22:37:11 +0200

[Message part 1 (text/plain, inline)]

Control: retitle -1 ISO loopback support for Debian installer
Control: affects -1 + cdrom-detect + apt-setup

Hi,

I further improved the patch by solving three problems and merging with 
the second patch from Ian.

First the solved problems:
 * Filesystem support:
On 06.10.2013 04:06, Andreas Cadhalpun wrote:
> * Added support for kfreebsd/hurd and any filesystem by using blkid
> to detect the filesystem of the device to be mounted.
 While this is theoretically true, practically it is not, because not 
all filesystem drivers are included in the initrd. For example neither 
the ntfs nor the ext4 driver are included. For this patch to be useful, 
it has to be possible to loopmount from USB or hard disk. Therefore I 
suggest to include (in addition to the loop-module) the following 
modules (I updated the Apply_Patches.sh script accordingly):
  Highly recommended filesystem drivers:
     * ext2/ext3/ext4 (currently default Debian file system)
     * ntfs (currently default Windows file system)
     * udf (probably the future of USB file systems)
    These together are about 450 kB.
  Optional filesystem drivers:
     * btrfs
     * jfs
     * squashfs
     * xfs
    These together are about 650 kB.
 Adding all of them to both initrd's makes the ISO approximately 2 MB 
larger, which should be no problem.

 * aspell fails to install when using CD-set: This was due to the 
following code in 41cdset:
$logoutput $chroot $ROOT apt-cdrom -d $mount_point  add \
			   -o Dir::Etc::SourceList=/dev/null \
			   </dev/null;
I used /dev/null as argument for Dir::Etc::SourceList since the returned 
list is not used and I thought it would simply do nothing, but 
apparently it breaks the system somehow. Changing /dev/null to $tmp 
there and afterwards removing the $tmp file resolved the problem.

 * CD-set authentication issue: I checked using the ISO as 'deb file:' 
repositories in an installed system. There the CD-2 works without 
problems, but CD-1 complains, as it should, since it is modified. But 
this is no explanation of the inconsistent behaviour of apt during 
installation. Perhaps 'deb cdrom:' repositories do generally not get 
checked for a correctly signed Release.gpg, but an additional 'deb 
file:' triggers a check?
As a workaround for this problem, I changed the 'deb file:' entries to 
'deb [ trusted=yes ] file:'. This should not be a security problem, 
since, when the checksum of the ISO is identical to the downloaded 
signed one, the user can be sure of the integrity of all software on the 
ISO. If it is not (or the user does not check) anything could be on the 
ISO anyway. Since the [ trusted=yes ] flag does no harm, I included it 
in this patch (if it works with an officially signed CD-1 without this 
flag, then it can be removed).

Furthermore I noticed, that the loopback.cfg I uploaded utilized 
'loopback=' instead of 'loopmount=', which I corrected now. I testet, 
that grub2 can load it with e.g.:
menuentry "Debian Testing netinstall loopback (amd64)" {
	set gfxpayload=auto
	set iso_path="/ISO/debian-testing-amd64-netinst-loopmount.iso"
	export iso_path
	boot_options=locale=de_DE.UTF-8
	export boot_options
	loopback loop $iso_path
	set root=(loop)
	configfile /boot/grub/loopback.cfg
	loopback -d loop
}

The Apply_Patches.sh script now uses xorriso and produces isohybrid ISOs 
that can be directly dd'ed to an USB stick. I checked, and this still 
works, at least for the debian-7.1.0-amd64-CD-1.iso (though the 
installer does not use the native resolution of the display).

I tested this patch with all the following ISOs loopmounted using 
loopback.cfg from an USB with UDF file system and it seems to work 
perfectly:
 * debian-7.1.0-amd64-netinst.iso
 * debian-7.1.0-i386-netinst.iso
 * debian-7.1.0-amd64-CD-1.iso
 * debian-7.1.0-amd64-CD-2.iso (+)
 * debian-7.1.0-amd64-CD-3.iso (+)
 * debian-7.1.0-amd64-DVD-1.iso
 * debian-7.1.0-amd64-DVD-2.iso (+)
(+): Not modified, just copied to the folder of the first ISO in the set.
I did not inclued the testing ISOs in that list, although the patch 
seems to work there as well, because the testing ISOs are currently in 
no good shape (e.g. bug #725714).


On 06.10.2013 15:04, ian_bruce@fastmail.net wrote:
> I think the problem is related to this bug:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608201
I haven't noticed bug #608201, but indeed this is related.

> I have therefore adopted the (partial?) solution which was applied in
> that case, which is that ISO images which are found somewhere other
> than an actual CD, do not get included in "sources.list".
I understand the solution differently: The ISOs get included in the 
sources.list, but for some reason I don't know, the default behaviour of 
'40cdrom' is to unmount all CDs/ISOs and let 'apt-cdrom add' mount them 
again. (This is strange, since CD sets are handled in 41cdset and not in 
40cdrom.) Because 'apt-cdrom add' does detect only CDs, not USB-HDD nor 
isohybird USBs nor loopmounted ISOs, this default behaviour has to be 
changed in those cases.
The choosen method in bug #608201 was to export from 'cdrom-detect', 
whether it is an isohybrid or USB/USB-HDD and then from this determine 
in '40cdrom', whether or not the device is detectable by 'apt-cdrom 
add', i.e. is an actual disk in a drive. This is rather complicated, so 
I moved the determination of mountability to 'cdrom-detect' and exported 
this information for '40cdrom'. (Since I don't know, if the 
isohybrid/USB-HDD information is needed anywhere else, I still export that.)

> If a better solution is developed sometime later for those
> situations, it should apply to the loopmount case as well.
From my point of view, it should work if '40cdrom' just never unmounts 
anything and 'apt-cdrom add' is never allowed to mount anything, which 
would eliminate the need for exporting this information from 
'cdrom-detect' to '40cdrom'.


> Two patchfiles are attached: a slightly altered one for
> "cdrom-detect", and a new one for "apt-setup". I haven't actually
> tested the latter, because I'm working with the "netinst" ISO, which 
> doesn't seem to include that package.
The debian-7.1.0-amd64-netinst.iso contains the relevant files in:
/pool/main/a/apt-setup/apt-cdrom-setup_0.79_all.udeb


> It appears that your changes to that file are similar to mine;
> however, I think mine are preferable, in this regard:
>
> I see that you have copied (in this and several other places) this
> piece of code, for determining if a loop-mount is being used:
>
>      for arg in $(cat /proc/cmdline); do
>          case $arg in
>          loopmount=*)
>              LOOPMOUNT=${arg#loopmount=}
>              LOOPFILE=${LOOPMOUNT#*:}
>              [ "$LOOPFILE" != "$LOOPMOUNT" ] && LOOPDEV=${LOOPMOUNT%:*}
>              ;;
>          esac
>      done
>
> Based on my current patches, this is now unnecessary. Do this instead:
>
>      loopmount=$(db_getval cdrom-detect/cdrom_loopdev)
I actually thought about using debconf to export the 'loopmount' 
variable, but since I didn't know how it works, I decided to use the 
'quick and dirty' approach, that I knew to work. Using db_set is surely 
better, but your implementation does not work. I checked the manual:
 http://manpages.ubuntu.com/manpages/lucid/en/man7/debconf-devel.7.html
One needs to create the appropriate templates in the 
cdrom-detect.templates file, which I added now (and also for 
cdrom_options and cdrom_mountable). Using loopdev instead of loopmount 
to determine, whether an ISO is mounted, is preferable, because even if 
the loopmount boot parameter is set, but the ISO could not be loaded and 
instead e.g. a CD is loaded, none of the loopmount specific changes in 
the code after cdrom-detect will be executed.

> I'm not sure if I understand this:
>
>      if [ "$ROOT" ] && [ "$LOOPMOUNT" ]; then
>          save_label
>      fi
>
> Does that mean that the ISO will be specified in "sources.list"?
No it does not. It just saves the label of the current CD for '41cdset', 
which needs to know this. (It probably would be better to use debconf 
here as well.)
The entry in sources.list is created with the following command:
$logoutput $chroot $ROOT apt-cdrom add \
   -o Dir::Etc::SourceList=$tmp \
   </dev/null; then
cat $ROOT$tmp >> $file
Note that these entries only get to the actual sources.list, if the 
script exits successfully and otherwise they are ignored.

> However, I did not see any benefit in your changes
> to "cdrom-detect.postinst".
Then let me explain them to you. The remaining differences between my 
latest patch and your second patch in cdrom-detect are:
 * I did not include the $DISTRIB_LABEL, because I don't know how to do 
it best, and it works fine without (although it perhaps has to check 
more devices). If it helps you, the actually used $DISTRIB_LABEL is 
included on the ISO: In the file .disk/mkisofs is the command, with 
which the ISO was created and the option -V 'Debian 7.1.0 amd64 1' 
produces the label.
  You commented it out with the note:
# this should wait for a proper solution for bug #608201
  I see no connection to that bug, since that has nothing to do with 
the installer device not being found, but rather it not being included 
in the apt database.

In cdrom-detect.postinst:
 * In my patch the filesystem of the device to be mounted is determined 
automatically. Ian wrote:
> It's easy enough for somebody to add the proper file system types for
> Hurd and FreeBSD.
  But it is even easier not having to do so! (And if it is so easy, why 
didn't you do it?) Furthermore your implementation suggested, that ext4 
was supported, which was not, because it was not in the initrd. With my 
patch any filesystem driver in the initrd can be used.

 * I export cdrom_options from cdrom-detect, so that the 'loop' option 
is used later on, when necessary (in 40cdrom and 41cdset).

 * Changed the log message, if the cd mount fails, to give 
$loopdev${device#/loop} as device.

 * Changed the log message from 'removable devices' to 
'CD/maybe-usb-floppy devices', because USB sticks are commonly referred 
to as removable devices as well, but not included in the devices here.

 * Changed an if statement, so that the CD/USB-HDD/isohybrid detection 
is executed, when loop-mounting the ISO fails. A 'break' prevents this, 
when the loop-mounting was successful.

 * Unmount /loop before trying to mount an ISO to it, because it is 
possible that the user selects cdrom-detect a second time, which would 
lead to an error, if loop was already mounted in the first time.

 * Changed the if-clause to determine which devices to look for to a 
for loop over $dev_given "usb-partition" "cd" "partition", which is 
better for two reasons:
   - The most likely devices (given device, usb devices) are looked at 
first.
   - If a $LOOPDEV is given, but no such device can be mounted, then 
the other devices are tried as well.

 * Added log messages with $device and $fstype, before a device is 
mounted, since this information is most useful, if the mount fails, so a 
message after successful mounting is not enough.

 * Do the cd_mountable determination in cdrom-detect and export the 
value via debconf.

 * Added source code comments, harmonised formatting and removed 
trailing whitespaces.

 * Added cdrom-detect/cdrom_options (for load-install-cd, 40cdrom, 
41cdsest), cdrom-detect/cdrom_loopdev (for 40cdrom, 41cdset) and 
cdrom-detect/cdrom_mountable (for 40cdrom) to the cdrom-detect.templates 
file.


That being said, I think you agree, that this patch is ready to be 
included in the Debian installer, because no known problems exist, and 
it is tested quite thoroughly.

Best regards,
Andreas

[Apply-Patches.sh (application/x-shellscript, attachment)]

[apt-cdrom-setup_loopmount_2.patch (text/x-patch, attachment)]

[initrd_loopmount_2.patch (text/x-patch, attachment)]

[loopback.cfg (text/plain, attachment)]

Message #78 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian Perrier <bubulle@debian.org>

Cc: 724931@bugs.debian.org, Ian Bruce <ian_bruce@fastmail.net>

Subject: Bug#724931: Please include the patch in git

Date: Sat, 12 Oct 2013 20:03:53 +0200

[Message part 1 (text/plain, inline)]

Control: affects -1 + hw-detect mountmedia

Hi,

the patch for this bug affects the following packages:
 * apt-setup
 * cdrom-detect
 * hw-detect (check-missing-firmware)
 * mountmedia
Since among the maintainers of all these packages is Christian Perrier, 
I'm sending this to you.

A short summary:
The patch created by Ian Bruce and myself adds ISO loopback support for 
the Debian installer using a new boot parameter, to be used as follows:

loopmount=[DEVICE:]ISO

Here ISO specifies the path to the ISO, i.e. 
/ISO/debian-7.1.0-amd64-CD-1.iso.
DEVICE is for the name or UUID of the device, on which the ISO is 
located. Giving this is optional and if it is not given, all devices are 
searched for the ISO.
If the boot parameter is not given (or no ISO could be found), 
everything works essentially as before.

For the patch to work, the loop-module is needed in the initrd, so I 
suggest to make it a dependency of cdrom-detect.
I furthermore highly recommend to make the ext2/ext3/ext4, ntfs and udf 
modules dependencies of cdrom-detect, since these are the most common 
filesystems and thus being able to loopmount from them would be good. 
(Fat is somewhat outdated.)
The patch makes it possible to boot using USB-HDD from any filesystems 
with drivers in the initrd.

The patch also cleans up the somewhat messy workaround for bug #608201.

For ease of use, I propose to add a loopback.cfg similar to the the 
attached one to the ISO in the folder /boot/grub/. This would allow to 
easily mount the ISO using grub2.


I tested loopmounting with the following ISOs. (They were modified with 
the attached Apply_Patches.sh.)
 * debian-7.1.0-amd64-netinst.iso
 * debian-7.1.0-i386-netinst.iso
 * debian-7.1.0-amd64-CD-1.iso
 * debian-7.1.0-amd64-CD-2.iso (+)
 * debian-7.1.0-amd64-CD-3.iso (+)
 * debian-7.1.0-amd64-DVD-1.iso
 * debian-7.1.0-amd64-DVD-2.iso (+)
(+): Not modified, just copied to the folder of the first ISO in the set.
This worked without problems. To make sure all other boot mechanisms 
still work, I tested them for the patched debian-7.1.0-amd64-CD-1.iso:
 * Isohybrid: OK
 * USB-HDD: OK
 * CD: I can't open the CD drive with the button the on the drive. I 
have to change to another TTY and use 'eject'. (This problem exists also 
with the original ISO image, see #726137.)

Since the patch works well and does no harm, I ask you to include it in git.

Changes since the last patch:
 * finish.d/10apt-cdrom-setup: Remove the whole 'deb file:' line. With 
the last patch, an empty line remained.
 * The script mountmedia uses 'mount -tauto', but this only tries to 
mount as fat and nothing else, so I changed this to detect the 
filesystem with blkid. Now firmware can be loaded from a harddrive/USB 
with any of the filesystems, for which drivers are in the initrd.
 * Removed $FATFS from cdrom-detect and instead let the filesystem be 
automatically detected using blkid. With this it is possible to use 
USB-HDD for all filesystems, for which drivers are in the initrd.
 * Removed iso-hybrid and usb-hdd templates, since it works well without.

Best regards,
Andreas

[Apply-Patches.sh (application/x-shellscript, attachment)]

[apt-cdrom-setup_loopmount_3.patch (text/x-patch, attachment)]

[initrd_loopmount_3.patch (text/x-patch, attachment)]

[loopback.cfg (text/plain, attachment)]

Message #85 received at 724931@bugs.debian.org (full text, mbox, reply):

From: <ian_bruce@fastmail.net>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: Christian Perrier <bubulle@debian.org>, 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 13 Oct 2013 01:21:16 -0700

I'm not done with this yet. I'm working on a more general patch with new
features, which will be forthcoming shortly. I would ask that nothing
major be done until that is ready.

The current version is certainly ready for testing, although Andreas
already seems to have done so extensively.


On Sat, 12 Oct 2013 20:03:53 +0200
Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:

> The patch created by Ian Bruce and myself adds ISO loopback support
> for the Debian installer using a new boot parameter, to be used as
> follows:
> 
> loopmount=[DEVICE:]ISO

"loopback" is the name of a virtual network device; the proper term in
this context is in fact "loopmount", hence the name of the boot
parameter.

> Here ISO specifies the path to the ISO, i.e.
> /ISO/debian-7.1.0-amd64-CD-1.iso.

Relative to the root directory of some block device, of course.

> DEVICE is for the name or UUID of the device, on which the ISO is
> located. Giving this is optional and if it is not given, all devices
> are searched for the ISO.

It specifies the filesystem label or UUID, as found in
/dev/disk/by-label/ or /dev/disk/by-uuid/, or returned by /sbin/blkid.

At least in my version of the patch, if it is not specified, then
partitions and CD/DVD drives are searched, but not other devices.

> If the boot parameter is not given (or no ISO could be found),
> everything works essentially as before.

As far as I know, if the "loopmount" parameter is not specified, then
everything works EXACTLY as before (by design).

> For the patch to work, the loop-module is needed in the initrd, so I
> suggest to make it a dependency of cdrom-detect. I furthermore highly
> recommend to make the ext2/ext3/ext4, ntfs and udf modules
> dependencies of cdrom-detect, since these are the most common
> filesystems and thus being able to loopmount from them would be good.

It appears that the ext4 module would be sufficient to also mount
ext2/ext3, whereas the reverse would not be true.

https://ext4.wiki.kernel.org/index.php/Frequently_Asked_Questions#What_is_the_difference_between_ext2.2C_ext3.2C_and_ext4.3F

https://ext4.wiki.kernel.org/index.php/Frequently_Asked_Questions#Can_I_mount_existing_Ext3_as_Ext4.3F_And_vice_versa.3F_Similarly_from_Ext2_to_Ext4_and_its_reverse.3F

https://ext4.wiki.kernel.org/index.php/Ext4_Howto#Compatibility

There might be some value in including NTFS, so that you could
loop-mount from Windoze partitions.

I don't know what usage UDF gets (besides DVDs) that would justify
including it in the initrd.

> (Fat is somewhat outdated.)

VFAT is by no means outdated, since it is used on almost every USB
flashdrive in existence. You might expect that it would eventually be
replaced for this purpose by F2FS, but that certainly hasn't happened
yet.

Anyway, it's already in the initrd.

> The patch makes it possible to boot using USB-HDD from any filesystems
> with drivers in the initrd.

Actually, from any supported block device with a supported filesystem.
It appears that most standard PATA/SATA/SCSI/CDROM/USB drivers are
already included in the initrd, so the only thing that would need to be
added is "loop.ko" and a few filesystem drivers.

> The patch also cleans up the somewhat messy workaround for bug
> #608201.

A proper fix would be for all ISO images to be treated the same,
regardless of whether they were contained in a CD, a disk partition, or
a loop-mounted file. I'm not sure why this shouldn't be possible, but
it's not the issue we're mainly concerned with at the moment.

> For ease of use, I propose to add a loopback.cfg similar to the the
> attached one to the ISO in the folder /boot/grub/. This would allow to
> easily mount the ISO using grub2.

I can supply similar config files for Syslinux, allowing the use of the
original boot menus with a loop-mounted ISO.

> I tested loopmounting with the following ISOs. (They were modified
> with the attached Apply_Patches.sh.)
>
>   * debian-7.1.0-amd64-netinst.iso
>   * debian-7.1.0-i386-netinst.iso
>   * debian-7.1.0-amd64-CD-1.iso
>   * debian-7.1.0-amd64-CD-2.iso (+)
>   * debian-7.1.0-amd64-CD-3.iso (+)
>   * debian-7.1.0-amd64-DVD-1.iso
>   * debian-7.1.0-amd64-DVD-2.iso (+)
>
> (+): Not modified, just copied to the folder of the first ISO in the
> set.

As I have suggested previously, you don't actually have to modify the
ISOs for testing; you can just patch an external copy of the initrd and
boot with that. That way, the official MD5 and SHA256 hashes still
verify.

> This worked without problems. To make sure all other boot mechanisms
> still work, I tested them for the patched debian-7.1.0-amd64-CD-1.iso:
>
>   * Isohybrid: OK
>   * USB-HDD: OK

Thanks for testing all that.

> * CD: I can't open the CD drive with the button the on the drive. I
> have to change to another TTY and use 'eject'. (This problem exists
> also with the original ISO image, see #726137.)

I think I also ran into this at some point.

> Since the patch works well and does no harm, I ask you to include it
> in git.

I think this will be a highly useful feature, and may become the primary
method of booting Debian. (Does anybody really bother burning install
CDs anymore?)

However, as I said above, I'm still working on some improvements, so
it's not quite done yet.

> Changes since the last patch:

> * The script mountmedia uses 'mount -t auto', but this only tries to
> mount as fat and nothing else, so I changed this to detect the
> filesystem with blkid. Now firmware can be loaded from a harddrive/USB
> with any of the filesystems, for which drivers are in the initrd.

That's what "mount -t auto" is supposed to do. From the manual page:

    If no -t option is given, or if the auto type is specified, mount
    will try to guess the desired type. Mount uses the blkid library for
    guessing the filesystem type; if that does not turn up anything that
    looks familiar, mount will try to read the file /etc/filesystems,
    or, if that does not exist, /proc/filesystems.

So I don't know why using /sbin/blkid directly would make any
difference. Maybe there's some peculiarity about the Busybox version of
"mount".

> * Removed $FATFS from cdrom-detect and instead let the filesystem be
> automatically detected using blkid. With this it is possible to use
> USB-HDD for all filesystems, for which drivers are in the initrd.

For the reasons given above, I'm not very confident about this. I think
it's better to specify the relevant filesystem types directly.

"mount -t vfat,ext4,iso9660" actually works; I suggest we just add ntfs
or whatever seems desirable to that list.

> * Removed iso-hybrid and usb-hdd templates, since it works well
> without.

I concur. That logic is better handled in another way.


-- Ian Bruce

Message #90 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Ben Hutchings <ben@decadent.org.uk>

To: ian_bruce@fastmail.net, 724931@bugs.debian.org

Cc: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, Christian Perrier <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 13 Oct 2013 14:57:48 +0100

[Message part 1 (text/plain, inline)]

On Sun, 2013-10-13 at 01:21 -0700, ian_bruce@fastmail.net wrote:
> On Sat, 12 Oct 2013 20:03:53 +0200
> Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> wrote:
[...]
> > For the patch to work, the loop-module is needed in the initrd, so I
> > suggest to make it a dependency of cdrom-detect. I furthermore highly
> > recommend to make the ext2/ext3/ext4, ntfs and udf modules
> > dependencies of cdrom-detect, since these are the most common
> > filesystems and thus being able to loopmount from them would be good.
> 
> It appears that the ext4 module would be sufficient to also mount
> ext2/ext3, whereas the reverse would not be true.
[...]

The ext2 and ext3 modules are also being removed from Debian kernel
packages, starting with Linux 3.11.

Ben.

-- 
Ben Hutchings
Beware of bugs in the above code;
I have only proved it correct, not tried it. - Donald Knuth

[signature.asc (application/pgp-signature, inline)]

Message #95 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: ian_bruce@fastmail.net

Cc: 724931@bugs.debian.org, Christian Perrier <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 13 Oct 2013 17:14:39 +0200

[Message part 1 (text/plain, inline)]

Hi,

On 13.10.2013 10:21, ian_bruce@fastmail.net wrote:
> I'm not done with this yet. I'm working on a more general patch with new
> features, which will be forthcoming shortly. I would ask that nothing
> major be done until that is ready.
I'm curious, what features do you want to add?

> "loopback" is the name of a virtual network device; the proper term in
> this context is in fact "loopmount", hence the name of the boot
> parameter.
Yes, loopback is the lo network interface, but for some reason I don't 
understand, GRUB uses the term loopback for booting from an ISO:
https://www.gnu.org/software/grub/manual/grub.html#Loopback-booting
This is why I constantly get confused between loopback and (the more 
descriptive) loopmount.

>> If the boot parameter is not given (or no ISO could be found),
>> everything works essentially as before.
>
> As far as I know, if the "loopmount" parameter is not specified, then
> everything works EXACTLY as before (by design).
In my latest patch, some changes are effective, even if loopmount is not 
used, e.g.:
 * I cleared up the workaround for bug #608201, so that in the future 
it should not be necessary to change apt-cdrom-setup, if one adds a new 
booting method.
 * Since I included more filesystem drivers in the initrd, I changed 
the code so that USB-HDD also works from filesystems other than FAT.
 * I also exported the boot options from cdrom-detect and imported them 
in several places, instead of determining the again. This should not 
have any effect, if loopmount is not used, but if it is, the 'loop' 
option is used. If in the future some changes are made with the boot 
options in cdrom-detect, they will be respected by apt-cdrom-setup.

> It appears that the ext4 module would be sufficient to also mount
> ext2/ext3, whereas the reverse would not be true.
I just tested loopmounting from an ext2 filesystem, while only the ext4 
filesystem driver was in the initrd: It worked, blkid identified it as 
ext4 and mounting as such only gave an info message:
kernel: [   13.170123] EXT4-fs (sdb1): mounted filesystem without 
journal. Opts: (null)
And according to Ben there will be no ext2/ext3 modules in the kernel 
starting with 3.11. So just add the ext4 module.

> I don't know what usage UDF gets (besides DVDs) that would justify
> including it in the initrd.
I have to admit, that I didn't know more than that a month ago.

>> (Fat is somewhat outdated.)
>
> VFAT is by no means outdated, since it is used on almost every USB
> flashdrive in existence. You might expect that it would eventually be
> replaced for this purpose by F2FS, but that certainly hasn't happened
> yet.
FAT32 is no modern filesystem, since it doesn't support files larger 
than 4 GB, e.g. the debian-7.1.0-amd64-DVD-2.iso with 4.7 GB. The reason 
for it to be still widespread is, that it is supported by (nearly?) all 
operating systems.
While F2FS might be modern and optimized for flash drives, it is only 
supported in Linux and thus cannot replace FAT32.
UDF on the other hand is used for DVDs and most operating systems can 
read (and even write to) it. Furthermore it supports large files and 
thus is the most probable replacement for FAT32. For further explanation 
see:
http://duncanlock.net/blog/2013/05/13/using-udf-as-an-improved-filesystem-for-usb-flash-drives/
Already, some USB sticks sold are formatted with UDF.
So I recommend to add UDF support now, although it will probably not be 
relevant for the broad audience until several years have passed by.

>> The patch also cleans up the somewhat messy workaround for bug
>> #608201.
>
> A proper fix would be for all ISO images to be treated the same,
> regardless of whether they were contained in a CD, a disk partition, or
> a loop-mounted file. I'm not sure why this shouldn't be possible, but
> it's not the issue we're mainly concerned with at the moment.
In fact, with my patch, they are treated practically the same, with the 
one exception of CD set support, which is only available for actual 
disks in a drive and loopmounted ISOs.
It is probably possible to extend this support to isohybrid and USB-HDD, 
but it is assumed, that one does not want to use multiple USB sticks to 
install Debian.

>> For ease of use, I propose to add a loopback.cfg similar to the the
>> attached one to the ISO in the folder /boot/grub/. This would allow to
>> easily mount the ISO using grub2.
>
> I can supply similar config files for Syslinux, allowing the use of the
> original boot menus with a loop-mounted ISO.
That would be great.

> As I have suggested previously, you don't actually have to modify the
> ISOs for testing; you can just patch an external copy of the initrd and
> boot with that. That way, the official MD5 and SHA256 hashes still
> verify.
The problem is, that I also have to modify the apt-cdrom-setup.udeb, 
that is not in the initrd, but gets loaded afterwards from 
/pool/main/a/apt-setup.

>> * CD: I can't open the CD drive with the button the on the drive. I
>> have to change to another TTY and use 'eject'. (This problem exists
>> also with the original ISO image, see #726137.)
>
> I think I also ran into this at some point.
Perhaps you could drop this info at bug #726137 for confirmation of the 
problem.

>> Since the patch works well and does no harm, I ask you to include it
>> in git.
>
> I think this will be a highly useful feature, and may become the primary
> method of booting Debian. (Does anybody really bother burning install
> CDs anymore?)
I agree fully.

> However, as I said above, I'm still working on some improvements, so
> it's not quite done yet.
I would be glad for any improvements, though it already works quite well.

>> * The script mountmedia uses 'mount -t auto', but this only tries to
>> mount as fat and nothing else, so I changed this to detect the
>> filesystem with blkid. Now firmware can be loaded from a harddrive/USB
>> with any of the filesystems, for which drivers are in the initrd.
>
> That's what "mount -t auto" is supposed to do. From the manual page:
>
>      If no -t option is given, or if the auto type is specified, mount
>      will try to guess the desired type. Mount uses the blkid library for
>      guessing the filesystem type; if that does not turn up anything that
>      looks familiar, mount will try to read the file /etc/filesystems,
>      or, if that does not exist, /proc/filesystems.
>
> So I don't know why using /sbin/blkid directly would make any
> difference. Maybe there's some peculiarity about the Busybox version of
> "mount".
I also don't know why '-tauto' does not work.
With 'mount $1 -tauto $MNT' it gives a lot of the following warnings:
UDF-fs: warning (device sdc1): udf_fill_super: No partition found (1)
FAT-fs (sdc1): utf8 is not a recommended IO charset for FAT filesystems, 
filesystem will be case sensitive!
And it does not actually load the firmware.
With 'mount -t $fs_type $1 $MNT' and $fs_type determined with blkid it 
just works (see attached syslogs).

>> * Removed $FATFS from cdrom-detect and instead let the filesystem be
>> automatically detected using blkid. With this it is possible to use
>> USB-HDD for all filesystems, for which drivers are in the initrd.
>
> For the reasons given above, I'm not very confident about this. I think
> it's better to specify the relevant filesystem types directly.
>
> "mount -t vfat,ext4,iso9660" actually works; I suggest we just add ntfs
> or whatever seems desirable to that list.
Sorry, I must have missed your reasoning for why specifying the 
supported filesystems directly is better. I think you only wrote it 
would be 'easy enough' to give the filesystems by hand.
But, as I said, it is easier not having to change the source code for 
every driver added to or removed from the initrd. And I can see no 
drawbacks in that.
Furthermore, in my opinion, there should be as little as possible 
differences between the used kernel (Linux/kFreeBSD/Hurd), but these 
have different names for the filesystems. So I think it is best to let 
blkid determine the right filesystem and leave the rest independent of 
the used kernel.

Best regards,
Andreas

[syslog-blkid.short (text/plain, attachment)]

[syslog-tauto.short (text/plain, attachment)]

Message #100 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, Ian Bruce <ian_bruce@fastmail.net>

Subject: Re: Bug#724931: Please include the patch in git

Date: Wed, 30 Oct 2013 20:38:55 +0100

[Message part 1 (text/plain, inline)]

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):
> Control: affects -1 + hw-detect mountmedia
> 
> Hi,
> 
> the patch for this bug affects the following packages:
>  * apt-setup
>  * cdrom-detect
>  * hw-detect (check-missing-firmware)
>  * mountmedia
> Since among the maintainers of all these packages is Christian
> Perrier, I'm sending this to you.
> 
> A short summary:
> The patch created by Ian Bruce and myself adds ISO loopback support
> for the Debian installer using a new boot parameter, to be used as
> follows:

../...

Ian later asked for more time as he's working on more general fixes.

So, should I wait or shouldn't I?

[signature.asc (application/pgp-signature, inline)]

Message #105 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>, 724931@bugs.debian.org, Ian Bruce <ian_bruce@fastmail.net>

Subject: Re: Bug#724931: Please include the patch in git

Date: Fri, 01 Nov 2013 17:06:11 +0100

[Message part 1 (text/plain, inline)]

Hi,

I don't know why, but I didn't receive your email. I just read it on 
bugs.debian.org.

Using the patched ISOs I noticed an error in the patch: In 41cdset 
$filename was incorrectly used instead of $ISOname. Therefore too many 
ISOs were considered as part of the set. The corrected patch is 
attached. I also removed the unused $ISOend variable.

Furthermore I noticed that there  are more official ISOs than I had in 
mind, when I wrote the cdset-patch:
If the first ISO is called debian-7.2.0-amd64-CD-1.iso, 41cdset looks 
for other ISOs with names debian-7.2.0-amd64-CD-*[^1].*[.]iso and thus 
finds e.g. debian-7.2.0-amd64-CD-2.iso. But there exists also the 
debian-update-7.2.0-amd64-CD-1.iso, which would not be found. And if you 
started with a debian-7.2.0-kfreebsd-amd64-CD-1.iso, 41cdset would not 
find debian-7.2.0-amd64-CD-2.iso.
Possible ways to treat this problem:
 * One could just adapt the script for these cases, but I think that 
the detailed structure of the ISO names should not be hardcoded.
 * Alternatively one could just check every *.iso, but this has the 
disadvantage, that if you have many other ISOs in the same folder, you 
would have to click through many messages in the installer, stating that 
this or that ISO is no Debian ISO.
 * I think the best way would be to mention in the documentation what 
ISOs 41cdset looks for. Any ISO could easily be renamed to match the 
assumptions.

I have also tried to apply the patch to the kfreebsd ISO to check, 
whether it works there as well, but I have been unable to extract the 
initrd. It seems not to be packed as gzipped cpio archive.

I haven't heard from Ian since his last mail on 13th October and I have 
no clue what he has in mind as a more general solution. Perhaps he has 
been busy otherwise.

In general I think, that it would be good to include the patch now and 
add any additional improvements later.

Best regards,
Andreas

[apt-cdrom-setup_loopmount_4.patch (text/x-patch, attachment)]

Message #110 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, Ian Bruce <ian_bruce@fastmail.net>

Subject: Re: Bug#724931: Please include the patch in git

Date: Fri, 1 Nov 2013 19:08:13 +0100

[Message part 1 (text/plain, inline)]

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):

> I haven't heard from Ian since his last mail on 13th October and I
> have no clue what he has in mind as a more general solution. Perhaps
> he has been busy otherwise.
> 
> In general I think, that it would be good to include the patch now
> and add any additional improvements later.

I re-added this to my TODO list. I'm just leaving a few more days as an
opportunity for Ian to react...

[signature.asc (application/pgp-signature, inline)]

Message #115 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, 724931@bugs.debian.org

Cc: Ian Bruce <ian_bruce@fastmail.net>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 23 Nov 2013 22:31:11 +0100

[Message part 1 (text/plain, inline)]

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):

> In general I think, that it would be good to include the patch now
> and add any additional improvements later.

I just committed the changes to apt-setup. I will probably soon upload
the package with them . PLEASE TEST AS SOON AS POSSIBLE.

[signature.asc (application/pgp-signature, inline)]

Message #122 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>

Cc: 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 23 Nov 2013 23:52:28 +0100

Hi,

thanks for committing the patch to apt-setup [1].
These changes have to be accompanied by the appropriate changes in 
cdrom-detect (and vice versa), as otherwise the installer will not work 
anymore.
Probably this should be ensured with a versioned dependency.
The changes in hw-detect and mountmedia are necessary for loading 
non-free firmware from filesystems different from FAT.

So please commit the changes to the other packages as well.

Best regards,
Andreas


1: 
http://anonscm.debian.org/gitweb/?p=d-i/apt-setup.git;a=commit;h=baf5e4aea1d15b0c8a04d9f8c2a186e76c26bb18


On 23.11.2013 22:31, Christian PERRIER wrote:
> Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):
>
>> In general I think, that it would be good to include the patch now
>> and add any additional improvements later.
>
> I just committed the changes to apt-setup. I will probably soon upload
> the package with them . PLEASE TEST AS SOON AS POSSIBLE.
>
>

Message #127 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 24 Nov 2013 08:28:09 +0100

[Message part 1 (text/plain, inline)]

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):
> Hi,
> 
> thanks for committing the patch to apt-setup [1].
> These changes have to be accompanied by the appropriate changes in
> cdrom-detect (and vice versa), as otherwise the installer will not
> work anymore.
> Probably this should be ensured with a versioned dependency.
> The changes in hw-detect and mountmedia are necessary for loading
> non-free firmware from filesystems different from FAT.

The problem is that, unless I'm mistaken, patches were not sent as
patches against the relevant packages, but as patches against the ISO
images.

So, it makes it fairly hard, when a bit far from the context, to
isolate what belongs to which package. The fact that many versions of
patches were sent to bug reports doesn't make it easier, too..:-)

So, could you isolate a patch that could apply cleanly to cdrom-detect
package tree....or point me to it?

I think I managed to apply the right changes to apt-setup, but that
probably need to be checked, too.


TIA,

[signature.asc (application/pgp-signature, inline)]

Message #132 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>

Cc: 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 24 Nov 2013 10:04:17 +0100

[Message part 1 (text/plain, inline)]

Hi,

I just created patches against cdrom-detect-1.46, hw-detect-1.98 and 
mountmedia-0.23. I hope that helps.

For the patch to work, the loop-module is needed in the initrd, so I
suggest to make it a dependency of cdrom-detect.
I furthermore highly recommend to make the ext4, ntfs and udf
modules dependencies of cdrom-detect, since these are the most common
filesystems and thus being able to loopmount from them would be good.

Best regards,
Andreas

[cdrom-detect.patch (text/x-patch, attachment)]

[hw-detect.patch (text/x-patch, attachment)]

[mountmedia.patch (text/x-patch, attachment)]

Message #137 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 24 Nov 2013 12:50:47 +0100

[Message part 1 (text/plain, inline)]

clone 724931 -1 -2 -3
reassign -1 cdrom-detect
reassign -2 hw-detect
reassign -3 mountmedia
thanks

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):
> Hi,
> 
> I just created patches against cdrom-detect-1.46, hw-detect-1.98 and
> mountmedia-0.23. I hope that helps.
> 
> For the patch to work, the loop-module is needed in the initrd, so I
> suggest to make it a dependency of cdrom-detect.

That (imho) won't work : if that module is not in D-I kernel then it
should be added there.


> I furthermore highly recommend to make the ext4, ntfs and udf
> modules dependencies of cdrom-detect, since these are the most common
> filesystems and thus being able to loopmount from them would be good.

The same stands here, but I suspect all these modules are already in
D-I kernel image packages. Needs to be carefully checked anyway.

[signature.asc (application/pgp-signature, inline)]

Message #144 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>

Cc: 724931@bugs.debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sun, 24 Nov 2013 19:14:39 +0100

Hi,

On 24.11.2013 12:50, Christian PERRIER wrote:
> That (imho) won't work : if that module is not in D-I kernel then it
> should be added there.
Why do you think this wouldn't work?

> The same stands here, but I suspect all these modules are already in
> D-I kernel image packages. Needs to be carefully checked anyway.
The udebs for these modules are on the D-I ISO (pool/main/l/linux), but 
not installed in the initrd (in 
<initrd>/lib/modules/3.*-amd64/kernel/fs/), where they are needed.

What do you think is the best way to get these modules into the initrd?

Best regards,
Andreas

Message #149 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>

Cc: 724931@bugs.debian.org

Subject: Bug#724931: Please include the patch in git

Date: Sat, 30 Nov 2013 13:48:14 +0100

[Message part 1 (text/plain, inline)]

Hi,

I have researched how to include the needed modules in the initrd.

If I understand it correctly, it would be possible to add the modules as 
dependencies of cdrom-detect.
But that would not be feasible, because the kernel modules have the 
version of the kernel in their name and therefore the dependencies would 
have to be updated every time a new kernel is released.

Instead I think the best way is to change the config files for the 
debian-installer [1] that determine what udebs are included in which 
initrd. They reside in installer/build/pkg-lists.

For the modules needed for loopmount I attached a patch against the 
debian-installer. In this patch I just added the loop-, ext4-, ntfs- and 
udf-modules to all config files that include cdrom-detect. This comes 
close to making them dependencies of cdrom-detect, but it is much 
easier, since in these config files one can use the variable 
${kernel:Version} and doesn't have to update that manually.

Please include that patch in the debian-installer and then release the 
new versions of cdrom-detect, apt-setup, hw-detect and mountmedia.

Best regards,
Andreas


1: http://anonscm.debian.org/gitweb/?p=d-i/debian-installer.git;a=tree

[debian-installer.patch (text/x-patch, attachment)]

Message #154 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Christian PERRIER <bubulle@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, kibi@debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 30 Nov 2013 18:16:13 +0100

[Message part 1 (text/plain, inline)]

clone 724931 -1
reassign -1 debian-installer
retitle -1 Please add needed modules for ISO loopmount to work in D-I
thanks

Quoting Andreas Cadhalpun (andreas.cadhalpun@googlemail.com):
> Hi,
> 
> I have researched how to include the needed modules in the initrd.
> 
> If I understand it correctly, it would be possible to add the
> modules as dependencies of cdrom-detect.
> But that would not be feasible, because the kernel modules have the
> version of the kernel in their name and therefore the dependencies
> would have to be updated every time a new kernel is released.
> 
> Instead I think the best way is to change the config files for the
> debian-installer [1] that determine what udebs are included in which
> initrd. They reside in installer/build/pkg-lists.
> 
> For the modules needed for loopmount I attached a patch against the
> debian-installer. In this patch I just added the loop-, ext4-, ntfs-
> and udf-modules to all config files that include cdrom-detect. This
> comes close to making them dependencies of cdrom-detect, but it is
> much easier, since in these config files one can use the variable
> ${kernel:Version} and doesn't have to update that manually.
> 
> Please include that patch in the debian-installer and then release
> the new versions of cdrom-detect, apt-setup, hw-detect and
> mountmedia.

Thanks, Andreas.

I therefore clone this bug report and assign it to debian-installer
itself, so that the needed actions are ataken on time there, too, if
we want this feature.

I was already suspecting that having Cyril Brulebois (CC'ed) look
specifically at all this is really needed as this feature is indeed
quite invasive. So, well, let's get his input.

I have the feeling that, given the time you invested in this, we
should have some cinfidence that your proposal  both makes
sense....and will not break D-I.....but someone less optimistic and
confident than me might want to have a closer look...:-)

[signature.asc (application/pgp-signature, inline)]

Message #161 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Christian PERRIER <bubulle@debian.org>

Cc: 724931@bugs.debian.org, kibi@debian.org

Subject: Re: Bug#724931: Please include the patch in git

Date: Mon, 02 Dec 2013 16:53:41 +0100

Hi,

this is now the second time that I did not get your mail. If you have 
any idea what could cause that, please let me know, because it is not 
really practical to always check bugs.debian.org.

On Topic: It's always better, if more people look at a patch, so I'd be 
grateful for KiBi's thoughts on it, especially concerning the effects 
for kFreeBSD, since I could not test these, because I did not manage to 
unpack the initrd from the kFreeBSD ISO.

I don't know really what you mean by invasive, but I assume you mean, 
that the installer could break, if apt-setup and cdrom-detect are not 
both updated at the same time. So let me clarify this a bit:
The functionality to loopmount is not invasive at all, since the 
necessary code for this is only executed, if the boot parameter 
'loopmount=' is given.

What makes this patch more invasive is the fact that it cleans up the 
somewhat messy workaround for bug #608201.
Up to now the situation was, that for every non-CD boot method 
(usb-hdd/isohybrid) a template was exported from cdrom-detect and 
imported in apt-setup to check there, whether cdset-detection should 
automatically (u)mount \cdrom.
I moved the code to determine this to cdrom-detect and only exported the 
result (cdrom_mountable) to apt-setup, so that in the future one does 
not have to change apt-setup (changes are needed only for cdset 
support), if one adds a new boot method.
Thus if not both patches (cdrom-detect and apt-setup) are applied at the 
same time, apt-setup will fail, because it looks for a template that 
does not exist.

Best regards,
Andreas

Message #166 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Tue, 3 Dec 2013 00:46:02 +0100

[Message part 1 (text/plain, inline)]

Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2013-12-02):
> this is now the second time that I did not get your mail. If you
> have any idea what could cause that, please let me know, because it
> is not really practical to always check bugs.debian.org.

Gmail, spam folder? It's a royal pain anyway since it believes it knows
better than users how many mails one should get…

> On Topic: It's always better, if more people look at a patch, so I'd
> be grateful for KiBi's thoughts on it, especially concerning the
> effects for kFreeBSD, since I could not test these, because I did
> not manage to unpack the initrd from the kFreeBSD ISO.
> 
> I don't know really what you mean by invasive, but I assume you
> mean, that the installer could break, if apt-setup and cdrom-detect
> are not both updated at the same time. So let me clarify this a bit:
> The functionality to loopmount is not invasive at all, since the
> necessary code for this is only executed, if the boot parameter
> 'loopmount=' is given.
> 
> What makes this patch more invasive is the fact that it cleans up
> the somewhat messy workaround for bug #608201.
> Up to now the situation was, that for every non-CD boot method
> (usb-hdd/isohybrid) a template was exported from cdrom-detect and
> imported in apt-setup to check there, whether cdset-detection should
> automatically (u)mount \cdrom.
> I moved the code to determine this to cdrom-detect and only exported
> the result (cdrom_mountable) to apt-setup, so that in the future one
> does not have to change apt-setup (changes are needed only for cdset
> support), if one adds a new boot method.
> Thus if not both patches (cdrom-detect and apt-setup) are applied at
> the same time, apt-setup will fail, because it looks for a template
> that does not exist.

Sorry, will take a while for me to get to it, finalizing the move from
ravel to dillon, and setting up a few things on my side to help me work
on d-i. Thanks for bearing with me…

Mraw,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #171 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Cyril Brulebois <kibi@debian.org>

Cc: 724931@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Tue, 03 Dec 2013 16:48:33 +0100

On 03.12.2013 00:46, Cyril Brulebois wrote:
> Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2013-12-02):
>> this is now the second time that I did not get your mail. If you
>> have any idea what could cause that, please let me know, because it
>> is not really practical to always check bugs.debian.org.
>
> Gmail, spam folder? It's a royal pain anyway since it believes it knows
> better than users how many mails one should get…
Strangely, these two mails did not end up in the spam folder (nor any 
other folder), they just seem to have vanishes somewhere on the way. But 
I get all(?) other mails from Christian (and everybody else).

> Sorry, will take a while for me to get to it, finalizing the move from
> ravel to dillon, and setting up a few things on my side to help me work
> on d-i. Thanks for bearing with me…
That's no problem for me, since I'm personally using the patch already.

Best regards,
Andreas

Message #176 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 8 Mar 2014 02:13:55 +0100

[Message part 1 (text/plain, inline)]

Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2013-12-02):
> On Topic: It's always better, if more people look at a patch, so I'd
> be grateful for KiBi's thoughts on it, especially concerning the
> effects for kFreeBSD, since I could not test these, because I did
> not manage to unpack the initrd from the kFreeBSD ISO.

The easiest way to get feedback concerning kFreeBSD is contacting
debian-bsd@.

> I don't know really what you mean by invasive, but I assume you
> mean, that the installer could break, if apt-setup and cdrom-detect
> are not both updated at the same time. So let me clarify this a bit:
> The functionality to loopmount is not invasive at all, since the
> necessary code for this is only executed, if the boot parameter
> 'loopmount=' is given.
> 
> What makes this patch more invasive is the fact that it cleans up
> the somewhat messy workaround for bug #608201.
> Up to now the situation was, that for every non-CD boot method
> (usb-hdd/isohybrid) a template was exported from cdrom-detect and
> imported in apt-setup to check there, whether cdset-detection should
> automatically (u)mount \cdrom.
> I moved the code to determine this to cdrom-detect and only exported
> the result (cdrom_mountable) to apt-setup, so that in the future one
> does not have to change apt-setup (changes are needed only for cdset
> support), if one adds a new boot method.
> Thus if not both patches (cdrom-detect and apt-setup) are applied at
> the same time, apt-setup will fail, because it looks for a template
> that does not exist.

Given the apt-cdrom regression we're hitting (#740673), I don't feel
like shipping this amount of additional modifications in jessie alpha 1
images; on the other hand, not uploading what's in apt-setup's master
currently would mean not using updated l10n material, which isn't too
nice. I'm tempted to branch "iso-loopback" from current master, so that
it's easily found afterwards, and to revert the ISO loopback bits for
now.

Christian, can you please confirm that it makes sense on an l10n point
of view?

Mraw,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #181 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Cyril Brulebois <kibi@debian.org>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 08 Mar 2014 18:11:19 +0100

[Message part 1 (text/plain, inline)]

Hi,

On 08.03.2014 02:13, Cyril Brulebois wrote:
> Given the apt-cdrom regression we're hitting (#740673), I don't feel
> like shipping this amount of additional modifications in jessie alpha 1
> images; on the other hand, not uploading what's in apt-setup's master
> currently would mean not using updated l10n material, which isn't too
> nice. I'm tempted to branch "iso-loopback" from current master, so that
> it's easily found afterwards, and to revert the ISO loopback bits for
> now.

I generally agree that it would be good to have an alpha 1 installer to 
use for testing regressions of this patch, but on the other hand using 
loopmount to install Debian *works*.
So you might want to reconsider applying this patch now, as it seems to 
be currently the only way to install Debian jessie directly. ;)
Yesterday I made a new installation using the patched:
Debian GNU/Linux testing "Jessie" - Official Snapshot amd64 CD Binary-1 
20140203-06:21

I'm only half-joking, it really works, otherwise I would have reported a 
bug.

I fixed a few things in my previous patch (see attachments):
 * Fix typo in finish-install.d/10apt-cdrom-setup: cdrom/$j -> cdrom$j
 * Always test for other CDs, except if it is a netinst CD.
   Otherwise one can't use CD-2 together with 
debian-testing-amd64-kde-CD-1.iso of type full_cd/single. (Maybe I 
didn't get how this is supposed to work?)
 * Mount further ISOs if at least four parts (seperated by -) of the 
name are the same. Example:
    a) debian-testing-amd64-CD-1.iso
    b) debian-testing-amd64-kde-CD-1.iso
    c) debian-testing-amd64-CD-2.iso
    d) debian-testing-amd64-CD-2-local-changes.iso
    e) debian-testing-i386-CD-2.iso
  a) and b) would load c) or d), but not e).
 * Fix crash, if the filesystem is not know (e.g. unpartitioned).

Pease add the attached patches on top of current git, even if you don't 
want to apply the patch now and instead move it to another branch.

Best regards,
Andreas

[apt-setup-loopmount.patch (text/x-diff, attachment)]

[cdrom-detect-loopmount.patch (text/x-diff, attachment)]

[mountmedia-loopmount.patch (text/x-diff, attachment)]

Message #186 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 8 Mar 2014 19:04:03 +0100

[Message part 1 (text/plain, inline)]

Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-08):
> I generally agree that it would be good to have an alpha 1 installer
> to use for testing regressions of this patch, but on the other hand
> using loopmount to install Debian *works*.

Sorry, but until it's been pushed to the masses, I can't blindly trust
that. Many moving parts already, not going to add a critical one when
we already have a (known) regression in apt-setup/apt-cdrom. Especially
not a few days before a release.

Mraw,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #191 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Cyril Brulebois <kibi@debian.org>

Cc: 724931@bugs.debian.org, 740673@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 08 Mar 2014 20:40:14 +0100

Hi,

On 08.03.2014 19:04, Cyril Brulebois wrote:
> Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-08):
>> I generally agree that it would be good to have an alpha 1 installer
>> to use for testing regressions of this patch, but on the other hand
>> using loopmount to install Debian *works*.
>
> Sorry, but until it's been pushed to the masses, I can't blindly trust
> that. Many moving parts already, not going to add a critical one when
> we already have a (known) regression in apt-setup/apt-cdrom. Especially
> not a few days before a release.

I didn't really suggest applying the patch without further testing from 
someone other than me. I just wanted to express my amazement that 
loopmount works, while the usual way doesn't.

I looked more closely and found that commenting out the following in 
40cdrom fixes the problem:
	# Allow apt-cdrom to manage mounting/unmounting CDs in /target
	if [ "$cd_mountable" ]; then
		rm -f $ROOT/etc/apt/apt.conf.d/00NoMountCDROM

		$logoutput umount /target/media/cdrom* || true
		$logoutput umount /cdrom || true
	fi

At least I managed to install Debian jessie in a Virtual Machine.
So if there are no compelling reasons to have this (Note: I don't know, 
what this is supposed to do.), I would suggest removing this as a fix 
for Bug #740673.

Best regards,
Andreas

Message #196 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, 740673@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Sat, 8 Mar 2014 20:49:35 +0100

[Message part 1 (text/plain, inline)]

Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-08):
> I looked more closely and found that commenting out the following in
> 40cdrom fixes the problem:
> 	# Allow apt-cdrom to manage mounting/unmounting CDs in /target
> 	if [ "$cd_mountable" ]; then
> 		rm -f $ROOT/etc/apt/apt.conf.d/00NoMountCDROM
> 
> 		$logoutput umount /target/media/cdrom* || true
> 		$logoutput umount /cdrom || true
> 	fi
> 
> At least I managed to install Debian jessie in a Virtual Machine.
> So if there are no compelling reasons to have this (Note: I don't
> know, what this is supposed to do.), I would suggest removing this
> as a fix for Bug #740673.

Err, no.

We had something that worked well enough for a while, it broke due to a
change in apt-cdrom, which was confirmed to be unintentional. So let's
fix that for now, and *maybe* rethink stuff after that.

I'm not goint to drop random lines, especially not based on “I don't
know what this is supposed to do”…

Mraw,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #201 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Fri, 14 Mar 2014 19:21:29 +0100

[Message part 1 (text/plain, inline)]

Cyril Brulebois <kibi@debian.org> (2014-03-08):
> Given the apt-cdrom regression we're hitting (#740673), I don't feel
> like shipping this amount of additional modifications in jessie alpha 1
> images; on the other hand, not uploading what's in apt-setup's master
> currently would mean not using updated l10n material, which isn't too
> nice. I'm tempted to branch "iso-loopback" from current master, so that
> it's easily found afterwards, and to revert the ISO loopback bits for
> now.

FWIW I've done that in apt-setup, which is awaiting another patch before
an upload (~ in an hour).

Mraw,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #206 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: Cyril Brulebois <kibi@debian.org>

Cc: 724931@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Mon, 24 Mar 2014 17:06:27 +0100

Hi,

On 08.03.2014 02:13, Cyril Brulebois wrote:
> Given the apt-cdrom regression we're hitting (#740673), I don't feel
> like shipping this amount of additional modifications in jessie alpha 1
> images;

Given that Bug #740673 is fixed and jessie alpha 1 is released, can you 
review and upload these patches now?

Best regards,
Andreas

Message #211 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Cyril Brulebois <kibi@debian.org>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

Cc: 724931@bugs.debian.org, Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Mon, 24 Mar 2014 17:11:43 +0100

[Message part 1 (text/plain, inline)]

Andreas Cadhalpun <andreas.cadhalpun@googlemail.com> (2014-03-24):
> Given that Bug #740673 is fixed and jessie alpha 1 is released, can
> you review and upload these patches now?

Let me be very clear: being pushy isn't going to get your stuff reviewed
sooner (probably the contrary, in fact).

Annoyed,
KiBi.

[signature.asc (application/pgp-signature, inline)]

Message #216 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Michael Schierl <schierlm@gmx.de>

To: 724931@bugs.debian.org

Subject: Status of this bug for jessie?

Date: Sat, 29 Mar 2014 14:04:04 +0100

Hi all,


I really don't want to be pushy (I appreciate that you all spend your
free time making Debian a better distribution) but am just curious about
the chances to get all the bits and pieces mentioned in this bug report
merged before Jessie gets frozen (so that one can take an official
Debian Jessie netinst ISO and drop it onto a loopback-boot enabled USB
drive next to installers for Ubuntu and other Linux distros that already
support it, and it will "just work").

I guess I am not the only person who likes to have one USB drive with
all their Linux install and live media present all the time, and
currently this means for me that whenever a new d-i comes out (that I
want to use), I have to collect patches from bug reports and build my
own d-i (and test it on a VM to avoid regressions) so that I can
loopback boot it from my USB drive.

As this means a considerable amount of effort (multiplied by the number
of people who are doing this all for themselves), if the odds are bad to
get it into official Jessie installers, perhaps we (who are regularly or
occasionally monitoring this bug) can coordinate our efforts and provide
an unofficial Debian Jessie netinstaller image somewhere (no idea about
how hosting stuff works within the Debian project, but for me even some
prebuilt and tested .iso on a Dropbox or similar file hoster would be
good enough - YMMV).


Just my 2 cents,


Michael

Message #221 received at 724931@bugs.debian.org (full text, mbox, reply):

From: adrian15 <adrian15sgd@gmail.com>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, Cyril Brulebois <kibi@debian.org>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Re: Bug#724931: Please include the patch in git

Date: Tue, 18 Aug 2015 10:47:40 +0200

[Message part 1 (text/plain, inline)]

El 08/03/14 a las 18:11, Andreas Cadhalpun escribió:
> I fixed a few things in my previous patch (see attachments):
>  * Fix typo in finish-install.d/10apt-cdrom-setup: cdrom/$j -> cdrom$j
>  * Always test for other CDs, except if it is a netinst CD.
>    Otherwise one can't use CD-2 together with 
> debian-testing-amd64-kde-CD-1.iso of type full_cd/single. (Maybe I 
> didn't get how this is supposed to work?)
>  * Mount further ISOs if at least four parts (seperated by -) of the 
> name are the same. Example:
>     a) debian-testing-amd64-CD-1.iso
>     b) debian-testing-amd64-kde-CD-1.iso
>     c) debian-testing-amd64-CD-2.iso
>     d) debian-testing-amd64-CD-2-local-changes.iso
>     e) debian-testing-i386-CD-2.iso
>   a) and b) would load c) or d), but not e).
>  * Fix crash, if the filesystem is not know (e.g. unpartitioned).
>
> Pease add the attached patches on top of current git, even if you 
> don't want to apply the patch now and instead move it to another branch.
>
> Best regards,
> Andreas

Can you please explain why you are using: get_fstype () function which 
it's based on blkid instead of just using the old method of relying in 
auto function from the kernel itself?

This is used in both cdrom-detect.patch and mountmedia.patch.

Please, be aware, that I'm not telling you your approach is incorrect. 
It seems we are lacking the explanation or rationale on why you made 
that decision in order to evaluate that change in a fair manner.

Thank you very much!

adrian15
-- 
Support free software. Donate to Super Grub Disk. Apoya el software libre. Dona a Super Grub Disk. http://www.supergrubdisk.org/donate/

[Message part 2 (text/html, inline)]

Message #226 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>

To: adrian15 <adrian15sgd@gmail.com>, Cyril Brulebois <kibi@debian.org>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Tue, 18 Aug 2015 19:28:28 +0200

Hi adrian15,

On 18.08.2015 10:47, adrian15 wrote:
> Can you please explain why you are using: get_fstype () function which it's based
> on blkid instead of just using the old method of relying in auto function from the kernel itself?

The reason is simply that 'mount -tauto' didn't work, while explicitly specifying the
type found with blkid works fine.

Doing some archeology reveals the relevant error messages from syslog:
With 'mount -tauto':
kernel: [  109.257009] UDF-fs: warning (device sda9): udf_fill_super: No partition found (1)
kernel: [  109.378443] FAT-fs (sda9): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
main-menu[550]: (process:4539): mount: mounting /dev/sda9 on /media failed: Invalid argument

With blkid:
[   80.943104] EXT4-fs (sda9): mounted filesystem with ordered data mode. Opts: (null)

These happened during check_missing_firmware, i.e. this comes from mountmedia.
I think that convinced me not to use 'mount -t auto' in cdrom-detect.

However, that was two years ago. Much could have changed in the meantime.

> This is used in both cdrom-detect.patch and mountmedia.patch.
> 
> Please, be aware, that I'm not telling you your approach is incorrect. It seems we are lacking
> the explanation or rationale on why you made that decision in order to evaluate that change in a fair manner.

I'm curious: Why are you asking that now?

Best regards,
Andreas

Message #231 received at 724931@bugs.debian.org (full text, mbox, reply):

From: adrian15 <adrian15sgd@gmail.com>

To: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>, Cyril Brulebois <kibi@debian.org>, 724931@bugs.debian.org

Cc: Christian PERRIER <bubulle@debian.org>

Subject: Re: Bug#724931: Please include the patch in git

Date: Thu, 20 Aug 2015 00:06:58 +0200

El 18/08/15 a las 19:28, Andreas Cadhalpun escribió:
> Hi adrian15,
>
> On 18.08.2015 10:47, adrian15 wrote:
>> Can you please explain why you are using: get_fstype () function which it's based
>> on blkid instead of just using the old method of relying in auto function from the kernel itself?
> The reason is simply that 'mount -tauto' didn't work, while explicitly specifying the
> type found with blkid works fine.
>
> Doing some archeology reveals the relevant error messages from syslog:
> With 'mount -tauto':
> kernel: [  109.257009] UDF-fs: warning (device sda9): udf_fill_super: No partition found (1)
> kernel: [  109.378443] FAT-fs (sda9): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
> main-menu[550]: (process:4539): mount: mounting /dev/sda9 on /media failed: Invalid argument
>
> With blkid:
> [   80.943104] EXT4-fs (sda9): mounted filesystem with ordered data mode. Opts: (null)
>
> These happened during check_missing_firmware, i.e. this comes from mountmedia.
> I think that convinced me not to use 'mount -t auto' in cdrom-detect.
>
> However, that was two years ago. Much could have changed in the meantime.
Ok, I will try to reproduce it and see if it still happens.
>> This is used in both cdrom-detect.patch and mountmedia.patch.
>>
>> Please, be aware, that I'm not telling you your approach is incorrect. It seems we are lacking
>> the explanation or rationale on why you made that decision in order to evaluate that change in a fair manner.
> I'm curious: Why are you asking that now?
I'm in debconf15 trying to push forward some improvements that I'm 
interested of as this one. Having around real people to speed things 
helps but don't raise your expectations too early.
> Best regards,
> Andreas

adrian15
-- 
Support free software. Donate to Super Grub Disk. Apoya el software libre. Dona a Super Grub Disk. http://www.supergrubdisk.org/donate/

Message #236 received at 724931@bugs.debian.org (full text, mbox, reply):

From: connedc8@box565.bluehost.com

To: 724931@bugs.debian.org

Subject: Our UPS courier can not contact you (parcel #1057833)

Date: Sun, 12 Feb 2017 18:09:54 -0700

[Message part 1 (text/plain, inline)]

Dear Customer,

We can not deliver your parcel arrived at February 11.

Please check the attachment for details!

Thanks,
Michael Beasley,
UPS Senior Support Manager.

[UPS-Receipt-1057833.zip (application/zip, attachment)]

Message #241 received at 724931@bugs.debian.org (full text, mbox, reply):

From: sp99@sp99.nazwa.pl

To: 724931@bugs.debian.org

Subject: Please recheck your delivery address (UPS parcel 05726654)

Date: Mon, 13 Mar 2017 00:27:46 +0100

[Message part 1 (text/plain, inline)]

Dear Customer,

We can not deliver your parcel arrived at March 11.

You can find more details in this e-mail attachment!

With sincere appreciation,
Walter Whitehead,
UPS Mail Delivery Clerk.

[UPS-Delivery-Details-05726654.zip (application/zip, attachment)]

Message #246 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Jonathan Michalon <johndescs@gmail.com>

To: 724931@bugs.debian.org

Subject: ISO loopback support for Debian installer

Date: Mon, 26 Jun 2017 23:10:27 +0200

Hi,

Wanted to use a multiboot USB key with Debian 9 and found this wish. I
second this. Maybe for Debian 10? :)
A workaround is to use the hd-media initrd, possibly with d-i "shared"
options, which is kind of neat for a multiboot (but beware that modules
you load from ISOs have to be compatible with you running kernel).

Best,

-- 
Jonathan

Message #251 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Julian Rüger <jr98@gmx.net>

To: 724931@bugs.debian.org

Subject: ISO loopback support for Debian installer

Date: Wed, 30 May 2018 17:15:25 +0200

For what it's worth, I have also been bitten by the missing support for
loop.ko in the netinst iso many times (since wheezy). Ultimately,
support for grub2 iso-boot in some form is exactly what I hope for.

Thanks, guys!

Best,
Julian

Message #256 received at 724931@bugs.debian.org (full text, mbox, reply):

From: Aleksei Shargalin <shargalin@infotec.ru>

To: 724931@bugs.debian.org

Subject: ISO images need loop.ko and iso-scan

Date: Wed, 16 Dec 2020 20:56:52 +0300

Hi!


It's the end of 2020 and we still need initrd from hd-media if we wont 
to loopback-mount ISO file during install.

Why not to include iso-scan into netinst and other cd images?


Best regards, Aleksei Shargalin

Send a report that this bug log contains spam.