Debian Bug report logs - #696123
partman-base: need progress wrapper for non-libparted-based filesystem operations

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: submit@bugs.debian.org

Subject: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 17 Dec 2012 00:11:13 +0000

Package: partman-base
Version: 162
Severity: wishlist

parted 3.x removes most of the ped_file_system_* entry points that
required extensive copies of file system code in libparted, with the
exception of some limited support for HFS+ and FAT resizing (see its
NEWS file for more details; facilities such as file system probing are
still available).  d-i still uses these for important functions.  In
most cases we fall back to external utilities, but if there are any
remaining places where we don't then we'll need to fix those.

However, simply falling back to external utilities is not a
functionally-equivalent option right now, because in general we have no
support for handling progress information from those utilities.  Most of
them have some kind of progress interface, usually requiring some
special option and designed for human consumption rather than machine
parsing.

Rather than the Sisyphean task of giving all the appropriate utilities
machine-parseable progress interface, I instead suggest that we study
the current progress interfaces and try to identify some common factors
that can be parsed heuristically (e.g. solid bar of characters;
"\-/|"-style spinner; percentage output), and write a wrapper utility
that can translate some of the common ones into debconf progress output.
I think this ought to be a single program in partman-base, since there
are only so many ways people have invented for console progress
interfaces and we don't want to keep reinventing the wheel.  We can then
use that to wrap various mkfs, resize, etc. utilities, and in the
process fix some ancient bugs about lack of progress feedback in various
operations since some cases haven't actually used libparted's file
system support for a while.

The relevant parted_server commands to grep for in various partman-*
modules for ideas on the kinds of utilities that need to be wrapped are:

  CREATE_FILE_SYSTEM
  CHECK_FILE_SYSTEM
  RESIZE_PARTITION

(Theoretically also COPY_PARTITION but we have no fallback for that; we
may have to either desupport this or reimplement it by hand somehow.)

I looked at doing this once but unfortunately I seem to have lost the
source code for my attempt; I'd only really made a start at it anyway.
I'm filing this bug as a memory aid, and because people keep asking me
what's happening with packaging of parted 3.x and I want to have
something reasonably coherent to point them to.

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #12 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Phillip Susi <psusi@ubuntu.com>

To: 696123@bugs.debian.org

Date: Sun, 16 Dec 2012 21:33:03 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm confused.  My understanding is that partman was already using
external utilities for everything besides fat, hfs, and ext2 resize,
and that there were already bug reports covering the lack of progress
indication with those.  Parted3 appears to have moved the resize
functions for fat and hfs to a separate library, so partman should be
able to just link to the new library and be no worse off than before,
with the exception of ext2, and well, who really cares about that?

What you describe here seems to be targeted at fixing the lack of
progress indication already covered by other bugs.  While providing
progress indication is nice, I'm not sure why it is still blocking the
migration to parted3.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJQzoRfAAoJEJrBOlT6nu75mVgH/A6LQ34N3rAl2nCy1TH9dvB4
FWZiIhNcDriBkN2AFjfbTMs37P59N5Pf4xm+r3O9X4uwV7dr5nF7oyDx1DYIgOWB
i08jyfm6HIsBnoOxCvDr/zVumFN3AEfJ3Od0AyDzufZJ+WIEsrFUOKHv1iUHgXem
co1cGKfGgJiRd0ObbgSsm91VzXK3cimR7yL4AeXzqls7TwLJEx0YVpUw1bGv8Aaz
iA0Az5xce+v0IjWBP60WNUSkL2F4GQXJCZvHeCVI7T/G+gtUk9OlXxyzFtQAVl8Q
EpM2/JTXJZ/SzF7eMWVe0khVJURgDtn9p2sICzwSIXFMXgu45k/DUiudpkv5CHg=
=yS46
-----END PGP SIGNATURE-----

Message #17 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Ben Hutchings <ben@decadent.org.uk>

To: Colin Watson <cjwatson@debian.org>, 696123@bugs.debian.org

Subject: Re: Bug#696123: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 17 Dec 2012 02:45:41 +0000

[Message part 1 (text/plain, inline)]

On Mon, 2012-12-17 at 00:11 +0000, Colin Watson wrote:
[...]
> Rather than the Sisyphean task of giving all the appropriate utilities
> machine-parseable progress interface, I instead suggest that we study
> the current progress interfaces and try to identify some common factors
> that can be parsed heuristically (e.g. solid bar of characters;
> "\-/|"-style spinner; percentage output), and write a wrapper utility
> that can translate some of the common ones into debconf progress output.
[...]

I have trouble believing that this is going to be easier than patching
the most common utilities to add an option for a machine-readable
progress indicator.

Ben.

-- 
Ben Hutchings
Life is like a sewer:
what you get out of it depends on what you put into it.

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Phillip Susi <psusi@ubuntu.com>

To: Ben Hutchings <ben@decadent.org.uk>, 696123@bugs.debian.org

Cc: Colin Watson <cjwatson@debian.org>

Subject: Re: Bug#696123: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Sun, 16 Dec 2012 22:22:49 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I have trouble believing that this is going to be easier than
> patching the most common utilities to add an option for a
> machine-readable progress indicator.

Not too long ago I was looking at patching gparted to support this and
while it did not seem difficult to process the output of the e2fsprogs
utilities for the progress information, I was wondering why the
various fs utilities don't already have a flag to generate machine
readable output... and I think some of the utilities supported this,
and some did not, even within e2fsprogs.

IIRC, I found that at least some of them already have a flag to write
output to a specified auxiliary fd to indicate progress, it just
seemed that a standard needed decided on and implemented in all of them.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJQzpAJAAoJEJrBOlT6nu75XrkH/RdWFitPR0tsOXWDv5rFtYuu
mWSLXzcVT/meq8o/pfr0QthOjggeFNG5Bl1cKSQYEVs6kwLd37HP1CW93jSLqjHp
9t99OgIaE1xH320yp2fpX4/ScMl8eEcV8tOUzPlyth+RDMU0tI11lMOIKZb/eeha
I74zQ3CkBY6tT026ZSodsaq1bua6dmY6lzgYvxIImXTYDY58OvjL5LbmM/l4urJS
ZlVMTIxfE7FYX+YPKwYcqn1j5MirQlbniaFgY1Pi6eVv26KqghKuuZ5RH16Fuxdi
qcMA52Xq5J3fZl5O0kMsZLBBvM3B3g1ORDypHc+gEwrrGPVPWjfOHj7BCSsnx/E=
=VINZ
-----END PGP SIGNATURE-----

Message #27 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Ben Hutchings <ben@decadent.org.uk>

Cc: 696123@bugs.debian.org

Subject: Re: Bug#696123: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 17 Dec 2012 09:47:51 +0000

On Mon, Dec 17, 2012 at 02:45:41AM +0000, Ben Hutchings wrote:
> On Mon, 2012-12-17 at 00:11 +0000, Colin Watson wrote:
> [...]
> > Rather than the Sisyphean task of giving all the appropriate utilities
> > machine-parseable progress interface, I instead suggest that we study
> > the current progress interfaces and try to identify some common factors
> > that can be parsed heuristically (e.g. solid bar of characters;
> > "\-/|"-style spinner; percentage output), and write a wrapper utility
> > that can translate some of the common ones into debconf progress output.
> [...]
> 
> I have trouble believing that this is going to be easier than patching
> the most common utilities to add an option for a machine-readable
> progress indicator.

I have no such trouble; we're going to need a wrapper to translate into
the debconf protocol *anyway*, and having it understand a few different
syntaxes is not that troublesome.  IME getting the file descriptor
plumbing right on these things is usually the most fiddly part. :-)

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #32 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Phillip Susi <psusi@ubuntu.com>

Cc: Ben Hutchings <ben@decadent.org.uk>, 696123@bugs.debian.org

Subject: Re: Bug#696123: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 17 Dec 2012 09:50:39 +0000

On Sun, Dec 16, 2012 at 10:22:49PM -0500, Phillip Susi wrote:
> Not too long ago I was looking at patching gparted to support this and
> while it did not seem difficult to process the output of the e2fsprogs
> utilities for the progress information, I was wondering why the
> various fs utilities don't already have a flag to generate machine
> readable output... and I think some of the utilities supported this,
> and some did not, even within e2fsprogs.

Quite.  Of course we should use it where it exists.

> IIRC, I found that at least some of them already have a flag to write
> output to a specified auxiliary fd to indicate progress, it just
> seemed that a standard needed decided on and implemented in all of them.

This is a complete red herring: we don't need a standard.  We invoke
them all independently anyway, so invoking them in different ways and
even handling different styles of output is not a problem.  Trying to
invent and enforce a standard just takes you down an unnecessary rathole
of months spent trying to convince maintainers to change their
machine-parseable output, any of whom might point out with good reason
that it's machine-parseable and therefore is not supposed to change
frivolously.

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #41 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Phillip Susi <psusi@ubuntu.com>

To: 696123@bugs.debian.org

Cc: Colin Watson <cjwatson@debian.org>

Subject: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 10 Feb 2014 15:06:43 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Say Colin, it has been over a year so I thought I would ping on this.
 Is this really the only issue that is still preventing the parted3
transition?  If so I think it may be time go go ahead and make the
move rather than continue to block on this, which really is a very
minor issue given that ext4 only takes seconds to format these days
thanks to lazy_itable_init.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS+TFTAAoJEI5FoCIzSKrwc8IIAIjkcTN5Bc42LyL3JdLqihNL
7k6BZL5XPbSHaf+qiBKSs1T5fDa16ZEqYZUzP7nfDHDCcnXAHt1+R2bZ1sAIfyRK
NAzLxD+DhD+oPsyT61XHMfMh/HiNLLwZk1sieNQhq/tQ4hzJVZ45YQJ/JE+z2ht1
v21Nr+fIA+p1PEipQvjQSMECGBXLlE5+opxJI7vMvSQ1brmOUvJxi9qSlIQQNov5
eBBTaPSBEa25cBB13LtNWNwEsK2PIN7k3fBJa/IfF3lMe172WCVZEb2fvRKoKSz+
iWP0j+6DX5Wlaxj1IAQhTOc9UvdHRvNMuQN3lLQSH8C7G+KKx4Yw0bSCgP2FWAU=
=9Yk2
-----END PGP SIGNATURE-----

Message #46 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Phillip Susi <psusi@ubuntu.com>

Cc: 696123@bugs.debian.org

Subject: Re: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Mon, 10 Feb 2014 21:36:18 +0000

On Mon, Feb 10, 2014 at 03:06:43PM -0500, Phillip Susi wrote:
> Say Colin, it has been over a year so I thought I would ping on this.
>  Is this really the only issue that is still preventing the parted3
> transition?  If so I think it may be time go go ahead and make the
> move rather than continue to block on this, which really is a very
> minor issue given that ext4 only takes seconds to format these days
> thanks to lazy_itable_init.

Phillip, you've asked me about this a number of times.  I think the work
involved is fairly clear, although I don't seem to have managed to make
time for it personally.  Perhaps, rather than continuing to ask me about
this, you could have a go at the necessary d-i integration?  I don't
think it should be that hard really.

-- 
Colin Watson                                       [cjwatson@debian.org]

Message #51 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Phillip Susi <psusi@ubuntu.com>

To: Colin Watson <cjwatson@debian.org>

Cc: 696123@bugs.debian.org

Subject: Re: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Tue, 11 Feb 2014 15:43:43 -0500

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/10/2014 4:36 PM, Colin Watson wrote:
> Phillip, you've asked me about this a number of times.  I think the
> work involved is fairly clear, although I don't seem to have
> managed to make time for it personally.  Perhaps, rather than
> continuing to ask me about this, you could have a go at the
> necessary d-i integration?  I don't think it should be that hard
> really.

Could you point me in the direction of how to test with my locally
built parted3 udebs?  I tried using debian-installer and putting it in
the local directory but it is ignored I guess because it doesn't go in
the initrd, but is loaded later from the pool.  In the case of the
netinst image, it just downloads the official 2.3 udeb from the mirror
since there is no pool, and when I tried make build_cdrom_isolinux, it
just prepared the kernel and initrd in a directory rather than making
an iso.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJS+ot+AAoJEI5FoCIzSKrwXb0H/16Zfn8tA4aZW59RphthGzsM
7r+WRJ2hW8fsdU7Xq8xzxy1W2caECYnqTm/+mKHRdLOcsEEdx+WwsRquGYSLEhyP
5eNQiBAHnjGajaPyEXioG/BUX9mP6x7DrbgtsswtQg5TD3DQCuOMWpx4LwRPLlvk
ZuQPwib3Qb+5l6c8ZVKPV4wFxV5N9xxCuXqt5uA6AUCLTYDhK22S8bustnRxiTt5
XK2TM0flD9kcc3w1pjAW1rhQVbRwgEP+GwOq77GvJhMJUUV7qLwQMclOefKIk0kQ
YYQnjnq0nqoOQ3P3AbLA2XwsresutRay/AEpAEL32fFcRKy2BX6gcKjO97LXpE0=
=yPlS
-----END PGP SIGNATURE-----

Message #56 received at 696123@bugs.debian.org (full text, mbox, reply):

From: Colin Watson <cjwatson@debian.org>

To: Phillip Susi <psusi@ubuntu.com>

Cc: 696123@bugs.debian.org

Subject: Re: partman-base: need progress wrapper for non-libparted-based filesystem operations

Date: Tue, 11 Feb 2014 21:55:34 +0000

On Tue, Feb 11, 2014 at 03:43:43PM -0500, Phillip Susi wrote:
> On 2/10/2014 4:36 PM, Colin Watson wrote:
> > Phillip, you've asked me about this a number of times.  I think the
> > work involved is fairly clear, although I don't seem to have
> > managed to make time for it personally.  Perhaps, rather than
> > continuing to ask me about this, you could have a go at the
> > necessary d-i integration?  I don't think it should be that hard
> > really.
> 
> Could you point me in the direction of how to test with my locally
> built parted3 udebs?  I tried using debian-installer and putting it in
> the local directory but it is ignored I guess because it doesn't go in
> the initrd, but is loaded later from the pool.

You're nearly there; in addition to having it in build/localudebs/, you
also need to add it to some appropriate file in build/pkg-lists/.  For
example build/pkg-lists/local would do.

Or you could just build a monolithic image, which is an image type
intended for this kind of testing.

> In the case of the netinst image, it just downloads the official 2.3
> udeb from the mirror since there is no pool, and when I tried make
> build_cdrom_isolinux, it just prepared the kernel and initrd in a
> directory rather than making an iso.

cdrom is mostly not a very convenient image type to use, since it needs
to be paired with debian-cd or similar to construct the actual CD.  Try
"make rebuild_netboot", or as I say "make rebuild_monolithic" to build
all plausible udebs into the image.

-- 
Colin Watson                                       [cjwatson@debian.org]

Send a report that this bug log contains spam.