Debian Bug report logs - #808265
e2fsprogs: support btrfs compression in filefrag

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: e2fsprogs: support btrfs compression in filefrag

Date: Fri, 18 Dec 2015 01:16:14 +0100

Package: e2fsprogs
Version: 1.42.13-1
Severity: wishlist
Tags: upstream


Hey Ted.

It would be nice if filefrag supports btrfs compression.
Right now, it seems to assume that each 128KiB compression "block"
is one extent, though, AFAIU discussion on linux-btrfs,
it's in realliy one extent.

Example:
/var/log/daemon.1.log is on a btrfs with *no* compression:
# filefrag -v /var/log/daemon.log.1 
Filesystem type is: 9123683e
File size of /var/log/daemon.log.1 is 1436385 (351 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..     220:   62079406..  62079626:    221:            
   1:      221..     255:   13664057..  13664091:     35:   62079627:
   2:      256..     319:   15844162..  15844225:     64:   13664092:
   3:      320..     350:   13515969..  13515999:     31:   15844226: last,eof
/var/log/daemon.log.1: 4 extents found

=> it actually *is* fragmented.

Copying it somewhere else, with no --reflink set:
# cp /var/log/daemon.log.1 ~/
# filefrag -v ~/daemon.log.1
Filesystem type is: 9123683e
File size of /root/daemon.log.1 is 1436385 (351 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..     350:   42929147..  42929497:    351:             last,eof
/root/daemon.log.1: 1 extent found


Now copying it to another btrfs, mounted with compress-force=zlib:
# cp /var/log/daemon.log.1 /mnt/
# filefrag -v /mnt/daemon.log.1 
Filesystem type is: 9123683e
File size of /mnt/daemon.log.1 is 1436385 (351 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..      31:       3072..      3103:     32:             encoded
   1:       32..      63:       3078..      3109:     32:       3104: encoded
   2:       64..      95:       3084..      3115:     32:       3110: encoded
   3:       96..     127:       3090..      3121:     32:       3116: encoded
   4:      128..     159:       3096..      3127:     32:       3122: encoded
   5:      160..     191:       3102..      3133:     32:       3128: encoded
   6:      192..     223:       3108..      3139:     32:       3134: encoded
   7:      224..     255:       3114..      3145:     32:       3140: encoded
   8:      256..     287:       3120..      3151:     32:       3146: encoded
   9:      288..     319:       3125..      3156:     32:       3152: encoded
  10:      320..     350:       3132..      3162:     31:       3157: last,encoded,eof
/mnt/daemon.log.1: 11 extents found

Shows exactly those 128KiB blocks distribution.


Cheers,
Chris.

Message #10 received at submit@bugs.debian.org (full text, mbox, reply):

From: Theodore Ts'o <tytso@mit.edu>

To: Christoph Anton Mitterer <calestyo@scientia.net>, 808265@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: Bug#808265: e2fsprogs: support btrfs compression in filefrag

Date: Fri, 18 Dec 2015 18:25:21 -0500

On Fri, Dec 18, 2015 at 01:16:14AM +0100, Christoph Anton Mitterer wrote:
> 
> It would be nice if filefrag supports btrfs compression.
> Right now, it seems to assume that each 128KiB compression "block"
> is one extent, though, AFAIU discussion on linux-btrfs,
> it's in realliy one extent.

How, exactly, do you propose that filefrag understand this?  It is
getting the information from the fiemap ioctl:

https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt

and the problem is that fiemap returns the logical length of the
extent, but not the *physical* length of the extent.  So this is why
the filefrag is show what appears to be overlapping extents:

>  ext:     logical_offset:        physical_offset: length:   expected: flags:
>    0:        0..      31:       3072..      3103:     32:             encoded
>    1:       32..      63:       3078..      3109:     32:       3104: encoded

Presumably, the first extent for logical blocks 0 -- 31 is taking only
6 blocks --- but there's no way filefrag can know that, not having any
magical or psychic abilities.  It could be that that the blocks only
take 5 blocks, and there is a one block "hole" meaning there was true
fragmentation --- but there's no way to tell that, and having filefrag
send e-mail to start a discussion on linux-btrfs is obviously not
something that is going to work given the current state of Artificial
Intelligence.  :-)

We could display:

   ext:     logical_offset:        physical_offset: length:   expected: flags:
    0:        0..      31:       3072..      ????:     32:             encoded
    1:       32..      63:       3078..      ????:     32:       3104: encoded
    ...

... since the encoded flag means that it is possible that physical
length is^H^H^H might be different from the logical length.  But even
that is not guaranteed; it could be that the blocks are encrypted, and
the physical length and logical lengths of the extent are the same ---
all FM_ENCODED means is that it is not safe to try to read the data
blocks directly if you expect to get the actual data.  (One of the
uses of fiemap is for bootloaders who want to store the block list so
they can boot the system without understanding the low-level file
system.)

So I'm afraid there's not much I can do here.  My suggestion is that
you kick off a discussion on linux-btrfs about adding a new, expanded
ioctl to the kernel that provides a superset of the FIEMAP ioctl,
which also returns the physical length.

Best regards,

					- Ted

Message #20 received at 808265@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.net>

To: Theodore Ts'o <tytso@mit.edu>, 808265@bugs.debian.org

Subject: Re: Bug#808265: e2fsprogs: support btrfs compression in filefrag

Date: Sat, 19 Dec 2015 01:02:38 +0100

[Message part 1 (text/plain, inline)]

Hey Ted.

Thanks for your detailed reply.
I took the liberty to forward that to linux-btrfs just in case they
wouldn't know already.

As for this bug... as you prefer, we could either let it open until
this should ever get resolved... or we could mark it wontfix till then.

Cheers,
Chris

On Fri, 2015-12-18 at 18:25 -0500, Theodore Ts'o wrote:
> On Fri, Dec 18, 2015 at 01:16:14AM +0100, Christoph Anton Mitterer
> wrote:
> > 
> > It would be nice if filefrag supports btrfs compression.
> > Right now, it seems to assume that each 128KiB compression "block"
> > is one extent, though, AFAIU discussion on linux-btrfs,
> > it's in realliy one extent.
> 
> How, exactly, do you propose that filefrag understand this?  It is
> getting the information from the fiemap ioctl:
> 
> https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt
> 
> and the problem is that fiemap returns the logical length of the
> extent, but not the *physical* length of the extent.  So this is why
> the filefrag is show what appears to be overlapping extents:
> 
> >  ext:     logical_offset:        physical_offset:
> > length:   expected: flags:
> >    0:        0..      31:       3072..      3103:     32:          
> >    encoded
> >    1:       32..      63:       3078..      3109:     32:       310
> > 4: encoded
> 
> Presumably, the first extent for logical blocks 0 -- 31 is taking
> only
> 6 blocks --- but there's no way filefrag can know that, not having
> any
> magical or psychic abilities.  It could be that that the blocks only
> take 5 blocks, and there is a one block "hole" meaning there was true
> fragmentation --- but there's no way to tell that, and having
> filefrag
> send e-mail to start a discussion on linux-btrfs is obviously not
> something that is going to work given the current state of Artificial
> Intelligence.  :-)
> 
> We could display:
> 
>    ext:     logical_offset:        physical_offset:
> length:   expected: flags:
>     0:        0..      31:       3072..      ????:     32:           
>   encoded
>     1:       32..      63:       3078..      ????:     32:       3104
> : encoded
>     ...
> 
> ... since the encoded flag means that it is possible that physical
> length is^H^H^H might be different from the logical length.  But even
> that is not guaranteed; it could be that the blocks are encrypted,
> and
> the physical length and logical lengths of the extent are the same --
> -
> all FM_ENCODED means is that it is not safe to try to read the data
> blocks directly if you expect to get the actual data.  (One of the
> uses of fiemap is for bootloaders who want to store the block list so
> they can boot the system without understanding the low-level file
> system.)
> 
> So I'm afraid there's not much I can do here.  My suggestion is that
> you kick off a discussion on linux-btrfs about adding a new, expanded
> ioctl to the kernel that provides a superset of the FIEMAP ioctl,
> which also returns the physical length.
> 
> Best regards,
> 
> 					- Ted

[smime.p7s (application/x-pkcs7-signature, attachment)]

Send a report that this bug log contains spam.