Debian Bug report logs - #869557
apt: please make the output of apt-ftparchive reproducible

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: submit@bugs.debian.org

Subject: apt: please make the output of apt-ftparchive reproducible

Date: Mon, 24 Jul 2017 10:34:15 +0100

[Message part 1 (text/plain, inline)]

Source: apt
Version: 1.5~beta1
Severity: wishlist
User: reproducible-builds@lists.alioth.debian.org

Hi,

According to Colin Percival, the output of apt-ftparchive is not
reproducible:

  https://news.ycombinator.com/item?id=14834520

At least for the "packages" subcommand, it appears we do non-deterministic
directory traversal using ftw(3) and output the results as we go along.


(Sorry for the usual lack of patch (!) but I was wondering if you had any
implementation ideas/pointers before I went down the wrong rabbithole. AIUI
we either need to make the traversal determinstic — is this even possible
with ftw(3)? — or collect all the results, sort, and then output which seems
a little bit more invasive. Thoughts?)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb, Debian Project Leader
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[apt.diff.txt (text/plain, attachment)]

Message #10 received at 869557@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 869557@bugs.debian.org

Subject: Re: apt: please make the output of apt-ftparchive reproducible

Date: Mon, 24 Jul 2017 15:06:36 +0100

[Message part 1 (text/plain, inline)]

tags 869557 + patch
thanks

Hi,

> apt: please make the output of apt-ftparchive reproducible

*Extremely* WIP patch attached:

  commit 489251d0bcf32883e9028aa0278168c167f2f6e9
  Author: Chris Lamb <lamby@debian.org>
  Date:   Mon Jul 24 15:04:03 2017 +0100
  
      Make the output of ftp-archive reproducible by sorting non-deterministic file orders prior to output. (Closes: #869557)
  
   ftparchive/writer.cc | 19 ++++++++++++++++++-
   ftparchive/writer.h  |  3 +++
   2 files changed, 21 insertions(+), 1 deletion(-)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb, Debian Project Leader
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[0001-Make-the-output-of-ftp-archive-reproducible-by-sorti.patch (text/x-diff, attachment)]

Message #17 received at 869557@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: Chris Lamb <lamby@debian.org>, 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Wed, 26 Jul 2017 12:12:15 +0200

[Message part 1 (text/plain, inline)]

Hi,

On Mon, Jul 24, 2017 at 03:06:36PM +0100, Chris Lamb wrote:
> > apt: please make the output of apt-ftparchive reproducible

I have to say, the moo change was entertaining, but while I see why
someone might want that [0], I fail to see how that effects
reproducibility of anything. Is there some sort of usecase or …
how did someone even find it out? apt-ftparchive isn't exactly the
most used/friendly/maintained tool… so is this in the end found by
codesearch.d.n or what?

[0] It helps if you produce patches/diffs for indexes – e.g. pdiffs, but
I have yet to meet someone who does it for apt-ftparchive (excluding
myself for testing… sometimes I think the only reason it still even runs
is that we use it for testing apt itself).


The patch is simple enough even after some style changes (we are c++11,
so we could use emplace_back e.g.), but it seems to be "incompatible"
with the other way of feeding files into the machine: A file listing as
the order wouldn't be the given one anymore but whatever apt sorts
– which might or might not be what you/the user wants.

I guess memory/performance footprint would be negligible as repositories
big enough to be visible have machines big enough to hide it, so for me
it is really "just" a matter of "what is the point?".  After all,
sometime down the line someone will come and "ask" why we did that
change breaking their carefully engineered setup and it would be handy
to have more of an argument than "but … the DPL said so!"… ;)


Best regards

David Kalnischkies

P.S.: apt-sortpkgs can help all those poor folks with unreproducible
index files now – it will also make indexes reproducible across tools
(assuming they all generate the same fields).

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 869557@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: David Kalnischkies <david@kalnischkies.de>, 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Wed, 26 Jul 2017 12:31:39 +0100

Dear David,

> I have to say, the moo change was entertaining, but while I see why
> someone might want that [0], I fail to see how that effects
> reproducibility of anything.

I worry that you have misunderstood my bug report and patch. As I
mention in the initial report, it was actually raised by a user who
explicitly expressed a need and/or desire for it and they have
subsequently thanked me for taking the effort to work on fixing their
issue.

There is no relation whatsoever to the "moo" change you reference and
am confused why you thought it might be so.

I was also disappointed to read that you — or anyone — might think that
my position as the current DPL would have any standing whatsoever on
the applicability of bug reports or technical issues.


Best wishes,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #27 received at 869557@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: Chris Lamb <lamby@debian.org>

Cc: 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Wed, 26 Jul 2017 19:36:00 +0200

[Message part 1 (text/plain, inline)]

Hi,

Seems like we perfectly misunderstood each other. :)

On Wed, Jul 26, 2017 at 12:31:39PM +0100, Chris Lamb wrote:
> > I have to say, the moo change was entertaining, but while I see why
> > someone might want that [0], I fail to see how that effects
> > reproducibility of anything.
> 
> I worry that you have misunderstood my bug report and patch. As I
> mention in the initial report, it was actually raised by a user who
> explicitly expressed a need and/or desire for it and they have
> subsequently thanked me for taking the effort to work on fixing their
> issue.

Do'h. hackernews references fail for me as they (as so many sites) are
annoying via tor (and was offline at time of reading), so my feeble
overprotective brain skipped over it… add that it is usertagged as
reproducible-builds and you get someone really confused in return.

Still, I don't see a big issue – or perhaps I see it as the biggest
issue that apt-ftparchive is used as you have yourself noted in the
thread "So many tools to do the same thing" and apt-ftparchive is pretty
old and low-level in this sea of options.

As noted in the P.S. the indexes can always be post-processed with
apt-sortpkgs which gets you reproducible indexes far better than sorting
by filepaths will, as the later can be effected by locale and is
a (probably very small) price maybe not everyone wants to pay.
Especially if an explicit order was already defined by a file list.


> I was also disappointed to read that you — or anyone — might think that
> my position as the current DPL would have any standing whatsoever on
> the applicability of bug reports or technical issues.

That makes two then, as that would indeed be a lousy argument. Hence me
asking why, ignoring that it was already provided, but skipped over.
Still, for most bugs I would prefer if the actual user is reporting them
rather than a proxy (DPL or not) and without external references simple
because it is easier to justify working for a user who went through the
trouble to report a bug.


Perhaps it helps a bit if I explain a bit where I am coming from:
apt-ftparchive is in pretty low-maintenance mode from our side,
basically just ensuring it isn't breaking too hard for the few existing
users. And with users I mainly mean launchpad which seems to use the
libdb part nobody else does, our testcases which use 'generate' and many
homegrown scripts. The later group would usually be better of using
a different tool, but doesn't for various good or less good reasons
(none mentioned in the thread).

At least the first two aren't effected as they use filelists. A good
part of the last group isn't effected either – so I wondered if someone
is actually really benefiting from this or if its just asked for
"because" like https. Henry Ford maybe said "If I had asked what they
wanted, they would have said 'faster horses'". We could be sorting the
output, but a user wanting that could have that already now with
apt-sortpkgs instead of waiting 2+ years for it (= optimism, archive
builders tend to be very slowly updated, which makes the feedback loop
if we break something so tiring). And if a more integrated feeling is
wanted, perhaps apt-ftparchive isn't the tool the user is looking for in
the first place (compare 'faster horses').


So yes, I still wonder why and if its a worthwhile time investment for
us as well as for the user to work on/use apt-ftparchive – and as said
style issues with the patch and the problem that it effects filelist
which it shouldn't.  Lastly, we have basically no test covering this
which conflicts with the no-new-untested code rule we try to enforce
meaning yet more work.

(Then again, in the time I wrote the mails, I could have probably just
written a few alibi tests and fix the patch, … oh well.)


Best regards

David Kalnischkies

[signature.asc (application/pgp-signature, inline)]

Message #32 received at 869557@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: Chris Lamb <lamby@debian.org>, 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Fri, 28 Jul 2017 19:24:54 +0200

[Message part 1 (text/plain, inline)]

So, after looking a bit more closely:

You weren't actually sorting (the predicate for std::sort expects
a boolean return, but you were using string.compare() which returns
an int like strcmp does) and calls with filelist hadn't changed there
order as predicted by me but were "just" entirely broken as they weren't
generating content anymore, so, in summary: no biggy. ;)

Attached revised patch has a testcase for this as well, but before
committing this to master I would prefer someone to run this against an
actual repository first just in case as as said apt-ftparchive doesn't
get a whole lot of attention and testing and I would like to avoid
fixing regressions on emergency in stable buster.


Best regards

David Kalnischkies

[0001-ftparchive-sort-discovered-filenames-before-writing-.patch (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #37 received at 869557@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: David Kalnischkies <david@kalnischkies.de>, 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Fri, 28 Jul 2017 18:49:36 +0100

Hi David,

> entirely broken as they weren't generating content anymore, so,
> in summary: no biggy. ;)

Odd, I wasn't getting that. Just to point out that my diff was
_explicitly_ labelled "*Extremely* WIP"! Curious I wasn't getting a
warning too? C++ is not my mother tongue, alas.

Anyway, Thanks for fixing everything up!


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb, Debian Project Leader
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #42 received at 869557@bugs.debian.org (full text, mbox, reply):

From: Colin Percival <cperciva@tarsnap.com>

To: 869557@bugs.debian.org

Subject: Re: Bug#869557: apt: please make the output of apt-ftparchive reproducible

Date: Sun, 30 Jul 2017 06:34:18 +0000

[Original complainer chiming in here]

Thanks Chris & David for working on this.  As it turns out, the problem was
in a sense a case of PEBKAC -- we didn't realize that apt-sortpkgs existed!
Now that we know that, the random ordering coming out of apt-ftparchive is
no longer an issue for us... although one could argue that there is instead
a documentation bug, in that apt-ftparchive(1) should say something along
the lines of "packages will be listed in an unpredictable order; you may
wish to run apt-sortpkgs to fix this".

Since the question of why we're using apt-ftparchive: For reasons of paranoia
(we run an encrypted online backup service, so it's important to ensure that
the binaries people are running are the right ones) we build packages and sign
everything in a very locked-down environment.  Using low level tools makes it
possible to do exactly what we want under exactly the right conditions, in a
way which isn't possible with a high level tool which does everything.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

Message #45 received at 869557-submitter@bugs.debian.org (full text, mbox, reply):

From: David Kalnischkies <david@kalnischkies.de>

To: 869557-submitter@bugs.debian.org

Subject: Bug#869557 in apt marked as pending

Date: Fri, 04 Aug 2017 22:24:51 +0000

Control: tag 869557 pending

Hello,

Bug #869557 in apt reported by you has been fixed in the Git repository. You can
see the commit message below, and you can check the diff of the fix at:

    https://anonscm.debian.org/cgit/apt/apt.git/diff/?id=d108e01

(this message was generated automatically based on the git commit message)
---
commit d108e019d3ca74c31a1ab646ddef5c60744a5e7d
Author: David Kalnischkies <david@kalnischkies.de>
Date:   Fri Jul 28 18:20:14 2017 +0200

    ftparchive: sort discovered filenames before writing indexes
    
    If 'apt-ftparchive packages /path/to/files/' (or sources) is used the
    files to include in the generated index (on stdout) were included in the
    order in which they were discovered, which isn't a very stable order
    which could lead to indexes changing without actually changing content
    causing needless changes in the repository changing hashsums, pdiffs,
    rsyncs, downloads, ….
    
    This does not effect apt-ftparchive calls which already have an order
    defined via a filelist (like generate) which will still print in the
    order given by the filelist.
    
    Note that a similar effect can be achieved by post-processing index
    files with apt-sortpkgs.
    
    Closes: 869557
    Thanks: Chris Lamb for initial patch & Stefan Lippers-Hollmann for testing

Message #52 received at 869557-close@bugs.debian.org (full text, mbox, reply):

From: Julian Andres Klode <jak@debian.org>

To: 869557-close@bugs.debian.org

Subject: Bug#869557: fixed in apt 1.5~beta2

Date: Thu, 17 Aug 2017 17:49:06 +0000

Source: apt
Source-Version: 1.5~beta2

We believe that the bug you reported is fixed in the latest version of
apt, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 869557@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated apt package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 17 Aug 2017 19:28:00 +0200
Source: apt
Binary: apt libapt-pkg5.0 libapt-inst2.0 apt-doc libapt-pkg-dev libapt-pkg-doc apt-utils apt-transport-https
Architecture: source
Version: 1.5~beta2
Distribution: unstable
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Description:
 apt        - commandline package manager
 apt-doc    - documentation for APT
 apt-transport-https - Transitional package: curl-https download transport for APT
 apt-utils  - package management related utility programs
 libapt-inst2.0 - deb package format runtime library
 libapt-pkg-dev - development files for APT's libapt-pkg and libapt-inst
 libapt-pkg-doc - documentation for APT development
 libapt-pkg5.0 - package management runtime library
Closes: 811181 858877 868127 869425 869557 870675 871275
Changes:
 apt (1.5~beta2) unstable; urgency=medium
 .
   [ M. Willis Monroe ]
   * Minor grammar fix
 .
   [ Zhou Mo ]
   * zh_CN.po: update Simplified Chinese programs translation
 .
   [ David Kalnischkies ]
   * don't expect more downloads from failed transactions
   * remove reference to a-t-debtorrent in description
   * ignore SIGPIPE in dump solver if forwarding
   * support compressed extended_states file for bug triage
   * don't move failed pdiff indexes out of partial (Closes: 869425)
   * don't try to parse all fields starting with HTTP as status-line
   * send weak-only hashes to methods
   * fail earlier if server answers with too much data
   * fail early in http if server answer is too small as well
   * use FileFd to parse all apt configuration files
   * show warnings instead of errors if files are unreadable (LP: #1701852)
   * reimplement and document auth.conf (Closes: #811181)
   * lookup login info for proxies in auth.conf
   * allow the auth.conf to be root:root owned
   * update URI scheme descriptions in sources.list(5)
   * show a warning for Debian shutting down FTP services
   * suggest using auth.conf for sources with passwords
   * ftparchive: sort discovered filenames before writing indexes.
     Thanks to Chris Lamb for initial patch & Stefan Lippers-Hollmann for testing
     (Closes: 869557)
   * don't keep configuration files open needlessly
   * don't hang if multiple sources use unavailable method (Closes: 870675)
 .
   [ Beatrice Torracca ]
   * Italian manpage translation update (Closes: 858877)
 .
   [ Apollon Oikonomopoulos ]
   * Handle supported components with slashes in sources.list (Closes: #868127)
 .
   [ Julian Andres Klode ]
   * Drop cacheiterators.h include
   * methods/aptmethod.h: Add missing fileutl.h include
   * Reformat and sort all includes with clang-format
   * cacheiterators: Warn about direct include and don't include pkgcache.h
   * Update gitignore with new files
   * Use C++11 threading support instead of pthread
   * Always warn if --force-yes is validly specified, not just if used
   * Work around float rounding change in gcc 7 on i386
   * Handle GCC 7 std::string operator ABI break (Closes: #871275)
   * debian: Update symbols for libapt-pkg5.0
 .
   [ Paul Wise ]
   * Support zero delay for the various APT::Periodic activities
   * Support seconds, minutes, hours and days for APT::Periodic intervals
   * Switch from /org to /srv in example apt-ftparchive configuration
 .
   [ Balint Reczey ]
   * Gracefully terminate process when stopping apt-daily-upgrade (LP: #1690980)
 .
   [ Dominik ]
   * doc: Add '--allow-unauthenticated' to '--force-yes'
Checksums-Sha1:
 c2df9b299098ef0808f78c3ebbf21d1c873aa09a 2598 apt_1.5~beta2.dsc
 3407383a7fbf657b593ea9bc25d549b00c2c8640 2094844 apt_1.5~beta2.tar.xz
 7966ef2ea16c0e2d98d6b6f7dc17b4fd09b64c85 8066 apt_1.5~beta2_source.buildinfo
Checksums-Sha256:
 aaa01458752d8674808cd840aeeafacf4a531ace17259428d9b3d67886ad376a 2598 apt_1.5~beta2.dsc
 5a9866b613a65929cd8852cd8813a1065bfe700928b66a91c5181a03196db4f5 2094844 apt_1.5~beta2.tar.xz
 8192235ea359c3849cb40afcd0de627cd130dd0436fc9a28d47c1e22c5c98dae 8066 apt_1.5~beta2_source.buildinfo
Files:
 3e8f485696568afa308951087eefe601 2598 admin important apt_1.5~beta2.dsc
 0ba1e229813604beea627beee8941765 2094844 admin important apt_1.5~beta2.tar.xz
 c1757e0d03dcccbce5f863d2d7429c5d 8066 admin important apt_1.5~beta2_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQJDBAEBCgAtFiEEzeVhi4gF/W4gLOnC1zw55WWAs4YFAlmV0loPHGpha0BkZWJp
YW4ub3JnAAoJENc8OeVlgLOGr8UP/3ewme1YZHZ+0Yqdx/NY0qnyFrGNBxQV+hQX
xO7ncGalAqF0bJ3nw9FrDpC1d7DVAlNOU9N3SsbGNSz+ZnPkqCvkssG0H/0pxXGQ
WkdkTH3dSgRDoJd6CvSHapt69/GBPrAqukE3jBZnA9KdKRXvdRlVyrAZ7lj2cvVM
eeC+vpy2sRf2+LgEUa179tniVH4DQkdNMu17xmmvLLairlr5fhEkf0GUv1GoOXJ+
A6eOnOYyvfgZpSlrAPU5hkJAS56mhG7MeGOnzdhVqJ9sJrWzBLr0QAMHg8+Kks0h
SKLrUeFGC5E1xlm+dHORukmTLyMp6UaxQTsjx6lfwDYFfxbxx3UJTSkcgV2ANpnN
70ZiWRaEa6OEpEl495GOrRN8jQYOFqfkX5yJrJ+/OadYXQstsIKR6Y/u73PeszKJ
ON1rEqYDwr6cEML4IJ4L0HaWlcc2dMMV5HOujQTJMsOubCHpTmyTYN2IZhNu3HfI
JesYFMc/CfX8VzdVIA9beZTbELM5IchB7uKUi1KheO6j4UXSlZkEj6KwVPonlA7v
i7s7qV31t0WW9muodl7lUX8jTAfvVo/Fopx+gTym/wW9sFWq/ATNy3Bqhlr7wUpM
miAVOeG/E/FAj33hp4S1zcSbRD+infVQPgXGbQiv+QQIzMarujQISupO3Y4OWFnx
+V6aKjWE
=aaby
-----END PGP SIGNATURE-----

Send a report that this bug log contains spam.