Debian Bug report logs - #735377
dpkg-source: 3.0 (native) silently ignores many binary files by default

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: peter green <plugwash@p10link.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: 3.0 (native) silently ignores many binary files by default.

Date: Wed, 15 Jan 2014 02:45:02 +0000

Package: dpkg-dev

In an ideal world source packages would always contain source code but 
sometimes source packages (especially private ones) need to contain 
binaries.

When attempting to build such a package using the 3.0 native format the 
source package build and binary package builds will succeed, but the 
"source" package won't actually contain the binaries that were in the 
source tree and so when someone comes to unpack said source package 
again to make a change they will find it is incomple.

IMO silently ignoring such files is a crazy default, either the files 
should be included or source package build should scream and die (thus 
prompting the packager that they need to either fix thier clean target 
or override something).

I can see some merit in silently ignoring VCS files and the like (though 
i'm far from convinced it's a good idea) but silently ignoring 
binaries/libraries is just crazy.

Message #10 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: peter green <plugwash@p10link.net>, 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Thu, 16 Jan 2014 08:44:14 +0100

Control: retitle -1 document how to un-ignore files ignored by default in "3.0 (native)" and "3.0 (quilt)"

On Wed, 15 Jan 2014, peter green wrote:
> When attempting to build such a package using the 3.0 native format
> the source package build and binary package builds will succeed, but
> the "source" package won't actually contain the binaries that were
> in the source tree and so when someone comes to unpack said source
> package again to make a change they will find it is incomple.
> 
> IMO silently ignoring such files is a crazy default, either the
> files should be included or source package build should scream and
> die (thus prompting the packager that they need to either fix thier
> clean target or override something).

It doesn't silently ignore random binary files, it ignores the files
that are on its default ignore list and which are documented
in dpkg-source --help:

  -I[<pattern>]            filter out files when building tarballs
                             (defaults to: -I*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids -I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -I.bzrignore -I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -I.gitmodules -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}).

So yes, *.a, *.la, *.o and *.so are ignored by default in native source package
and this is is so for as long as I can remember.

If you are unhappy with the default, please change it for your package
with debian/source/options:

tar-ignore=.git
tar-ignore=.gitignore

As soon as you start being explicit about the ignore rules, the default
ignore rules are no longer used but you can put the default ignore rule
back with a single "tar-ignore" (i.e. without value).

This probably ought to be better documented. Retitling the bug accordingly.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

Message #17 received at 735377@bugs.debian.org (full text, mbox, reply):

From: peter green <plugwash@p10link.net>

To: Raphael Hertzog <hertzog@debian.org>

Cc: 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Sat, 18 Jan 2014 00:05:03 +0000

Raphael Hertzog wrote:
> So yes, *.a, *.la, *.o and *.so are ignored by default in native source package
> and this is is so for as long as I can remember.
>   
Afaict if I leave a .so or similar file in the source tree (and don't 
override anything) the different source formats currently behave as follows

1.0 native: include the file in the tarball
1.0 non-native: scream and die
3.0 (native): quietly ignore the file
3.0 (quilt): scream and die

I can see the justification for quietly ignoring VCS files, they are 
unlikely to be used by the build and screaming and dying on them could 
be a pain since people may want to build from VCS checkouts.

But for  *.a, *.la, *.o and *.so . If they are present in the source 
tree then most likely one of two things is true.

1: The files are built as part of the package build process. In this 
case they should be removed by the clean target to ensure that freshly 
built versions are used in the package build.
2: The files are not built as part of the package build process and 
therefore need to be included in the source package (which of course 
makes the package unsuitable for Debian main).

In both of these cases quietly ignoring the files is broken behaviour. 
So I don't see any justification for quietly ignoring them being the 
default. Screaming and dieing would be a more sane behaviour and would 
be more consistent with the behaviour of non-native package builds.
> If you are unhappy with the default, please change it for your package
> with debian/source/options:
The problem is noticing the breakage in the first place. The packager 
takes his source tree, runs dpkg-buildpackage and the build completes 
succesfully so he assumes everything is ok. It's only later when someone 
(either the original packager or someone else) goes to work on the 
package again and unpacks the "source" package that they realise it does 
not contain everything needed for a successful build.

Message #22 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: peter green <plugwash@p10link.net>, 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Sat, 18 Jan 2014 02:13:54 +0100

Control: retitle -1 dpkg-source: 3.0 (native) silently ignores many binary files by default

Hi!

On Sat, 2014-01-18 at 00:05:03 +0000, peter green wrote:
> Raphael Hertzog wrote:
> >So yes, *.a, *.la, *.o and *.so are ignored by default in native source package
> >and this is is so for as long as I can remember.

> Afaict if I leave a .so or similar file in the source tree (and
> don't override anything) the different source formats currently
> behave as follows
> 
> 1.0 native: include the file in the tarball
> 1.0 non-native: scream and die
> 3.0 (native): quietly ignore the file
> 3.0 (quilt): scream and die
> 
> I can see the justification for quietly ignoring VCS files, they are
> unlikely to be used by the build and screaming and dying on them
> could be a pain since people may want to build from VCS checkouts.
> 
> But for  *.a, *.la, *.o and *.so . If they are present in the source
> tree then most likely one of two things is true.
> 
> 1: The files are built as part of the package build process. In this
> case they should be removed by the clean target to ensure that
> freshly built versions are used in the package build.
> 2: The files are not built as part of the package build process and
> therefore need to be included in the source package (which of course
> makes the package unsuitable for Debian main).
>
> In both of these cases quietly ignoring the files is broken
> behaviour. So I don't see any justification for quietly ignoring
> them being the default. Screaming and dieing would be a more sane
> behaviour and would be more consistent with the behaviour of
> non-native package builds.

Yeah, I'd tend to agree. I was looking into this the day you filed the
bug, and it seems like initially -I (w/o any argument) got a default set
of ignore wildcards (commit f548f40ba483fe8c2b0eab32ccfd1f3bd3cf769e),
and some time after that, that same ignore list was used unmodified
as the default for formats >= 2.0 (commit
ddf788627634080a39931061af354d8d8e15c433). The problem is, that as a
default for a user specified -I, it's fine, but as a default for a
source format, I'd say not so much. It also does not cover all current
object files, so it's not good at that either.

> >If you are unhappy with the default, please change it for your package
> >with debian/source/options:

> The problem is noticing the breakage in the first place. The
> packager takes his source tree, runs dpkg-buildpackage and the build
> completes succesfully so he assumes everything is ok. It's only
> later when someone (either the original packager or someone else)
> goes to work on the package again and unpacks the "source" package
> that they realise it does not contain everything needed for a
> successful build.

Another problem is that to remove just some of the ignore wildcards
is pretty tricky, as one has then to reproduce and carry over the
rest of the ignore list (see #615828 or #718984).

I've to say I'm a bit unhappy about the current -I and -i option
behaviours in general. :/

I'll ponder about changing the default (the object files part), I might
want to run it through debian-devel first though. And in addition I'll
probably add a way to unignore specific wildcards or (un)ignore groups
of wildcards, so that you could say to ignore vcs and object files, but
not backup files, for example.

Thanks,
Guillem

Message #29 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: peter green <plugwash@p10link.net>, 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Sun, 19 Jan 2014 10:40:05 +0100

Hi,

On Sat, 18 Jan 2014, peter green wrote:
> Raphael Hertzog wrote:
> >So yes, *.a, *.la, *.o and *.so are ignored by default in native source package
> >and this is is so for as long as I can remember.
> Afaict if I leave a .so or similar file in the source tree (and
> don't override anything) the different source formats currently
> behave as follows
> 
> 1.0 native: include the file in the tarball

Because ignore rules are not used for 1.0.

> 1.0 non-native: scream and die

Because the binary files are not representable in diff format and the
ignore rules are not used (and besides the diff ignore rules do not cover
those files).

> 3.0 (native): quietly ignore the file

yes

> 3.0 (quilt): scream and die

It depends on where the file is. In debian/, it's ignored just like in 3.0
(native) because the tar_ignore rules apply, outside of debian however,
it's the usual diff_ignore rules so they are not ignored but we get a
failure due to the unexpected binary file (which also can't be represented
in a quilt patch).

> But for  *.a, *.la, *.o and *.so . If they are present in the source
> tree then most likely one of two things is true.
> 
> 1: The files are built as part of the package build process. In this
> case they should be removed by the clean target to ensure that
> freshly built versions are used in the package build.
> 2: The files are not built as part of the package build process and
> therefore need to be included in the source package (which of course
> makes the package unsuitable for Debian main).
>
> In both of these cases quietly ignoring the files is broken
> behaviour. So I don't see any justification for quietly ignoring
> them being the default. Screaming and dieing would be a more sane
> behaviour and would be more consistent with the behaviour of
> non-native package builds.

That rather makes sense and some people have been bitten by the exclusion
of .so files when they wanted to install some ld linker scripts.

I would be willing to drop those tar_ignore rules but this is a backwards
incompatible change and I'm wary of breaking stuff. We should probably
seek further input from debian-devel.

Guillem, what's your opinion ?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

Message #34 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Sjoerd Simons <sjoerd@luon.net>

To: Raphael Hertzog <hertzog@debian.org>, 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Tue, 26 Apr 2016 09:10:06 +0200

On Thu, Jan 16, 2014 at 08:44:14AM +0100, Raphael Hertzog wrote:
> Control: retitle -1 document how to un-ignore files ignored by default in "3.0 (native)" and "3.0 (quilt)"
> 
> On Wed, 15 Jan 2014, peter green wrote:
> > When attempting to build such a package using the 3.0 native format
> > the source package build and binary package builds will succeed, but
> > the "source" package won't actually contain the binaries that were
> > in the source tree and so when someone comes to unpack said source
> > package again to make a change they will find it is incomple.
> > 
> > IMO silently ignoring such files is a crazy default, either the
> > files should be included or source package build should scream and
> > die (thus prompting the packager that they need to either fix thier
> > clean target or override something).
> 
> It doesn't silently ignore random binary files, it ignores the files
> that are on its default ignore list and which are documented
> in dpkg-source --help:
> 
>   -I[<pattern>]            filter out files when building tarballs
>                              (defaults to: -I*.a -I*.la -I*.o -I*.so -I.*.sw? -I*/*~ -I,,* -I.[#~]* -I.arch-ids -I.arch-inventory -I.be -I.bzr -I.bzr.backup -I.bzr.tags -I.bzrignore -I.cvsignore -I.deps -I.git -I.gitattributes -I.gitignore -I.gitmodules -I.hg -I.hgignore -I.hgsigs -I.hgtags -I.shelf -I.svn -ICVS -IDEADJOE -IRCS -I_MTN -I_darcs -I{arch}).
> 
> So yes, *.a, *.la, *.o and *.so are ignored by default in native source package
> and this is is so for as long as I can remember.
> 
> If you are unhappy with the default, please change it for your package
> with debian/source/options:
> 
> tar-ignore=.git
> tar-ignore=.gitignore
> 
> As soon as you start being explicit about the ignore rules, the default
> ignore rules are no longer used but you can put the default ignore rule
> back with a single "tar-ignore" (i.e. without value).

I tried this, dpkg-buildpackage then starts complaining:
  dpkg-source: warning: --tar-ignore= is not a valid option for Dpkg::Source::Package::V3::Native

This solution does work when switching back to source format 1 though


-- 
Neutrinos have bad breadth.

Message #39 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Guillem Jover <guillem@debian.org>

To: Sjoerd Simons <sjoerd@luon.net>, 735377@bugs.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Tue, 26 Apr 2016 10:54:15 +0200

Hi!

On Tue, 2016-04-26 at 09:10:06 +0200, Sjoerd Simons wrote:
> On Thu, Jan 16, 2014 at 08:44:14AM +0100, Raphael Hertzog wrote:
> > If you are unhappy with the default, please change it for your package
> > with debian/source/options:
> > 
> > tar-ignore=.git
> > tar-ignore=.gitignore
> > 
> > As soon as you start being explicit about the ignore rules, the default
> > ignore rules are no longer used but you can put the default ignore rule
> > back with a single "tar-ignore" (i.e. without value).
> 
> I tried this, dpkg-buildpackage then starts complaining:
>   dpkg-source: warning: --tar-ignore= is not a valid option for Dpkg::Source::Package::V3::Native

That's weird, it does work fine here, just checked with git master to
make sure there's been no regressions:

,---
dpkg-buildpackage: info: source package pkg
dpkg-buildpackage: info: source version 1.0
dpkg-buildpackage: info: source distribution sid
dpkg-buildpackage: info: source changed by Test <test@example.org>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build pkg-1.0
dpkg-source: info: using options from pkg-1.0/debian/source/options: --tar-ignore=.gitignore
 fakeroot debian/rules clean
dh clean
   dh_testdir
   dh_auto_clean
   dh_clean
 dpkg-source -b pkg-1.0
dpkg-source: info: using options from pkg-1.0/debian/source/options: --tar-ignore=.gitignore
dpkg-source: info: using source format '3.0 (native)'
dpkg-source: info: building pkg in pkg_1.0.tar.xz
dpkg-source: info: building pkg in pkg_1.0.dsc
[…]
`---

> This solution does work when switching back to source format 1 though

It should always work, otherwise I'd appreciate a reproducer.

Thanks,
Guillem

Message #44 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: 735377@bugs.debian.org

Cc: Sjoerd Simons <sjoerd@luon.net>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Tue, 26 Apr 2016 09:21:46 +0000

[Message part 1 (text/plain, inline)]

Hi,

On Tue, Apr 26, 2016 at 10:54:15AM +0200, Guillem Jover wrote:
> That's weird, it does work fine here, just checked with git master to
> make sure there's been no regressions:

we've been seeing the same behavior when uploading diffoscope. When I
build it, tests/data/test(1|2).(o|a) are not part of the source package,
when Rainer or Mattia build it, they are there as they should be…

.../diffoscope$ cat debian/source/options 
# Default ignore patterns contains *.o and *.a. So we need to define our
# own
# patterns to get them included.
--tar-ignore=.*.sw?
--tar-ignore=*/*~
--tar-ignore=,,*
--tar-ignore=.[#~]*
--tar-ignore=.deps
--tar-ignore=.git
--tar-ignore=.gitattributes
--tar-ignore=.gitignore
--tar-ignore=.gitmodules
.../diffoscope$ ls tests/data/*.o
tests/data/test1.o  tests/data/test2.o
.../diffoscope$ ls tests/data/*.a
tests/data/test1.a  tests/data/test2.a

I'm building with plain "debuild -S" and just confirmed this in sid and
jessie. 

Having done this, I checked my ~/.devscripts and found this:

DEBUILD_DPKG_BUILDPACKAGE_OPTS="-uc -us -I -i"

Is this why?


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #49 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Raphael Hertzog <hertzog@debian.org>

To: Holger Levsen <holger@layer-acht.org>

Cc: 735377@bugs.debian.org, Sjoerd Simons <sjoerd@luon.net>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Wed, 27 Apr 2016 11:15:27 +0200

On Tue, 26 Apr 2016, Holger Levsen wrote:
> Having done this, I checked my ~/.devscripts and found this:
> 
> DEBUILD_DPKG_BUILDPACKAGE_OPTS="-uc -us -I -i"
> 
> Is this why?

Yes. -I and --tar-ignore in debian/source/options are cumulative.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/

Message #54 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Raphael Hertzog <hertzog@debian.org>

Cc: 735377@bugs.debian.org, Sjoerd Simons <sjoerd@luon.net>, reproducible-builds@lists.alioth.debian.org

Subject: Re: Bug#735377: 3.0 (native) silently ignores many binary files by default.

Date: Wed, 27 Apr 2016 09:20:34 +0000

[Message part 1 (text/plain, inline)]

On Wed, Apr 27, 2016 at 11:15:27AM +0200, Raphael Hertzog wrote:
> > DEBUILD_DPKG_BUILDPACKAGE_OPTS="-uc -us -I -i"
> > Is this why?
> Yes. -I and --tar-ignore in debian/source/options are cumulative.

ok, thanks! I've had "-I -i" there for years & have just removed it now,
curious what future uploads of mine this will break ;-)


-- 
cheers,
	Holger

[signature.asc (application/pgp-signature, inline)]

Message #59 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 735377@bugs.debian.org

Cc: Holger Levsen <holger@layer-acht.org>

Subject: Re: dpkg-source: 3.0 (native) silently ignores many binary files by default

Date: Sun, 14 Aug 2016 13:39:41 +0100

Holger wrote:

> > > DEBUILD_DPKG_BUILDPACKAGE_OPTS="-uc -us -I -i"
> > > Is this why?
> > Yes. -I and --tar-ignore in debian/source/options are cumulative.
>
> ok, thanks! I've had "-I -i" there for years & have just removed it now,
> curious what future uploads of mine this will break ;-)

If it helps others, my .devscripts.conf was fine. What I needed to do was:

 - builder = debuild -i -I --changes-option=-S -uc -us
 + builder = debuild --changes-option=-S -uc -us --lintian-opts -i -I

.. in ~/.gbp.conf.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Message #64 received at 735377@bugs.debian.org (full text, mbox, reply):

From: Chris Lamb <lamby@debian.org>

To: 735377@bugs.debian.org

Cc: Holger Levsen <holger@layer-acht.org>

Subject: Fwd: Re: dpkg-source: 3.0 (native) silently ignores many binary files by default

Date: Sun, 14 Aug 2016 13:47:28 +0100

Holger wrote:

> > > DEBUILD_DPKG_BUILDPACKAGE_OPTS="-uc -us -I -i"

(I've filed a wishlist bug against devscripts as #834317 requesting a
warning if DEBUILD_DPKG_BUILDPACKAGE_OPTS contains such things.)


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

Send a report that this bug log contains spam.