Debian Bug report logs - #387686
debmirror: patch to allow fine grained mirroring

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Stephen Gran <sgran@debian.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: debmirror: patch to allow fine grained mirroring

Date: Sat, 16 Sep 2006 00:40:39 +0100

[Message part 1 (text/plain, inline)]

Package: debmirror
Version: 20060907
Severity: normal
Tags: patch

Hello Goswin,

Attached is a rough patch to allow more fine grained mirroring than is
currently possible.  The gist of the patch logic is that as
arch/dist/section tuples are parsed, on duplicate option or end of line,
defaults are filled in.  the data structure is stored as nested hashes
instead of arrays, so that it becomes possible to request something like:

$VAR1 = {
          'sid' => {
                     'amd64' => {
                                  'non-free' => 1,
                                  'contrib' => 1,
                                  'main' => 1
                                },
                     'i386' => {
                                 'contrib' => 1,
                                 'main/debian-installer' => 1,
                                 'main' => 1
                               }
                   },
          'etch' => {
                      'mips' => {
                                  'non-free' => 1,
                                  'contrib' => 1,
                                  'main/debian-installer' => 1,
                                  'main' => 1
                                }
                    },
          'sarge' => {
                       'powerpc' => {
                                      'non-free' => 1,
                                      'contrib' => 1,
                                      'main' => 1
                                    },
                       'sparc' => {
                                   'non-free' => 1,
                                   'contrib' => 1,
                                   'main/debian-installer' => 1,
                                   'main' => 1
                                 }
                     }
        };

(Data::Dumper output from a debug run)

As you can see, it allows more fine grained mirroring than is currently 
possible.  I would appreciate if you could apply this patch or something
like it.  I have not chased all the corner cases yet, so I can't say
with absolute confidence it introduces no regressions, but it seems sane
from the tests I've run so far.

Thanks,

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.17-2-686
Locale: LANG=en_US.UTF8, LC_CTYPE=en_US.UTF8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF8)

Versions of packages debmirror depends on:
ii  bzip2                         1.0.3-6    high-quality block-sorting file co
ii  libcompress-zlib-perl         1.42-1     Perl module for creation and manip
ii  liblockfile-simple-perl       0.2.5-7    Simple advisory file locking
ii  libnet-perl                   1:1.19-3   Implementation of Internet protoco
ii  libwww-perl                   5.805-1    WWW client/server library for Perl
ii  perl [libdigest-md5-perl]     5.8.8-6.1  Larry Wall's Practical Extraction 
ii  perl-modules [libnet-perl]    5.8.8-6.1  Core Perl modules
ii  rsync                         2.6.8-2    fast remote file copy program (lik

Versions of packages debmirror recommends:
ii  gnupg                         1.4.5-1    GNU privacy guard - a free PGP rep
ii  patch                         2.5.9-4    Apply a diff file to an original

-- no debconf information

-- 
 -----------------------------------------------------------------
|   ,''`.                                            Stephen Gran |
|  : :' :                                        sgran@debian.org |
|  `. `'                        Debian user, admin, and developer |
|    `-                                     http://www.debian.org |
 -----------------------------------------------------------------

[debmirror.patch (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Stephen Gran <sgran@debian.org>

To: 387686@bugs.debian.org

Subject: Re: debmirror: patch to allow fine grained mirroring

Date: Thu, 19 Oct 2006 01:04:13 +0100

[Message part 1 (text/plain, inline)]

This one time, at band camp, Stephen Gran said:
> Hello Goswin,
> 
> Attached is a rough patch to allow more fine grained mirroring than is
> currently possible.  The gist of the patch logic is that as
> arch/dist/section tuples are parsed, on duplicate option or end of line,
> defaults are filled in.

Hey,

Have you had a chance to look at this?  Do you mind if I NMU it?  I
would really like to see this feature make it into etch.

Thanks,
-- 
 -----------------------------------------------------------------
|   ,''`.                                            Stephen Gran |
|  : :' :                                        sgran@debian.org |
|  `. `'                        Debian user, admin, and developer |
|    `-                                     http://www.debian.org |
 -----------------------------------------------------------------

[signature.asc (application/pgp-signature, inline)]

Message #15 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Baurzhan Ismagulov <ibr@radix50.net>

To: 387686@bugs.debian.org

Subject: Re: debmirror: patch to allow fine grained mirroring

Date: Thu, 7 Dec 2006 00:14:56 +0100

Hello Stephen and Goswin,

On Sat, Sep 16, 2006 at 12:40:39AM +0100, Stephen Gran wrote:
> Attached is a rough patch to allow more fine grained mirroring than is
> currently possible.  The gist of the patch logic is that as
> arch/dist/section tuples are parsed, on duplicate option or end of line,
> defaults are filled in.  the data structure is stored as nested hashes
> instead of arrays, so that it becomes possible to request something like:

Wow, great! This is exactly what I was going to request! --dry-run shows
that the expected Packages have been downloaded. I'll test this more.
This is definitely a useful feature!

What I'd like to see:

1. Include --nosources in the tuple.

2. Documentation.

With kind regards,
Baurzhan.

P.S. My first mail was rejected by spamhaus. Where should I direct
     requests about b.d.o mail infrastructure?

Message #20 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Sergio Gelato <Sergio.Gelato@astro.su.se>

To: 387686@bugs.debian.org

Subject: fine-grained mirroring in debconf

Date: Thu, 18 Jan 2007 17:50:36 +0100

While this patch tries to address a real issue (also covered by wishlist 
bug #282636 and worked around by #369061 for the special case of sarge
and amd64), I find it unsatisfactory for the following reasons:

1. It isn't backwards-compatible, in that it changes the semantics of
previously-valid invocations of debmirror.

2. If I read it correctly,
	--dist=etch --arch=i386 --arch=amd64
is not equivalent to
	--dist=etch --arch=i386,amd64
as I at least would expect (and, in this particular case, as backwards 
compatibility would require) but to
	(etch,,i386) (sid,,amd64)
where I'm using a (dist,section,arch) tuple notation to avoid
confusion with the current debmirror semantics.

Currently, debmirror takes three sets (@dists, @sections, @arches) and
tries to mirror all tuples in the cross-product (except for a few that
do not exist in the official archive). I don't see any way to do more
fine-grained mirroring without supporing more set arithmetic. At a
minimum one would like a union operator (say, an --or switch on the
command line that does more or less the same thing as &add_data in the
proposed patch). This could also be achieved by multiple invocations of
debmirror, if the cleanup step could be taught not to delete valid files
from other invocations. A set substraction operator (--except ?) may also
be useful, but then you'd start needing parentheses.

So maybe the simplest way to rescue this patch is to add a --or switch
that invokes &add_data, and to let --dist, --section and --arch simply
accumulate into @dists, @sections and @arches as before.

Message #25 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>

To: 387686@bugs.debian.org

Cc: 387686-submitter@bugs.debian.org

Subject: Maybe fix for #387686 debmirror: patch to allow fine grained mirroring

Date: Tue, 19 Jun 2007 13:30:56 +0200

Hi,

I added a new option to debmirror (--localclean) to allow multiple
debmirror runs with different arguments to be run. This allows fine
grained mirroring by splitting the mirroring into multiple runs. E.g:

 debmirror -a i386  -s contrib -h ftp.debian.org --localcleanup /mirror/debian
 debmirror -a amd64 -s main    -h ftp.debian.org --localcleanup /mirror/debian

Make a mirror of i386 contrib and amd64 main.

I believe that this is a far simpler fix for the problem and does not
destroy backward compatibility. It also allows mirroring
ftp.debian.org and security.debian.org into a single local mirror and
as such is superior to the proposed patch.

Is that sufficient for you?

MfG
        Goswin

Message #33 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Stephen Gran <sgran@debian.org>

To: Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>, 387686-quiet@bugs.debian.org

Cc: 387686@bugs.debian.org, 387686-submitter@bugs.debian.org

Subject: Re: Bug#387686: Maybe fix for #387686 debmirror: patch to allow fine grained mirroring

Date: Tue, 19 Jun 2007 16:08:18 +0100

[Message part 1 (text/plain, inline)]

This one time, at band camp, Goswin von Brederlow said:
> Is that sufficient for you?

That's great, thanks very much.
-- 
 -----------------------------------------------------------------
|   ,''`.                                            Stephen Gran |
|  : :' :                                        sgran@debian.org |
|  `. `'                        Debian user, admin, and developer |
|    `-                                     http://www.debian.org |
 -----------------------------------------------------------------

[signature.asc (application/pgp-signature, inline)]

Message #46 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Roman Mamedov <roman@rm.pp.ru>

To: 387686@bugs.debian.org, 387686-subscribe@bugs.debian.org

Subject: Still not fixed

Date: Mon, 11 May 2009 21:31:25 +0600

[Message part 1 (text/plain, inline)]

> I added a new option to debmirror (--localclean) 

Sadly, there is still no such option in the current version of debmirror (20070123), and the proposed patch didn't get
applied either. I would like to see more fine grained mirroring too, specifically I need a way to mirror
three combinations: lenny-i386, squeeze-i386 and squeeze-amd64 - in one directory tree.

-- 
With respect,
Roman

[signature.asc (application/pgp-signature, attachment)]

Message #51 received at 387686@bugs.debian.org (full text, mbox, reply):

From: Joey Hess <joey@kitenet.net>

To: Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>

Cc: 387686@bugs.debian.org

Subject: Re: debmirror --localclean

Date: Sun, 5 Sep 2010 18:33:51 -0400

[Message part 1 (text/plain, inline)]

In 2007 you wrote:
> I added a new option to debmirror (--localclean) to allow multiple
> debmirror runs with different arguments to be run. This allows fine
> grained mirroring by splitting the mirroring into multiple runs. E.g:
> 
>  debmirror -a i386  -s contrib -h ftp.debian.org --localcleanup /mirror/debian
>  debmirror -a amd64 -s main    -h ftp.debian.org --localcleanup /mirror/debian
> 
> Make a mirror of i386 contrib and amd64 main.
> 
> I believe that this is a far simpler fix for the problem and does not
> destroy backward compatibility. It also allows mirroring
> ftp.debian.org and security.debian.org into a single local mirror and
> as such is superior to the proposed patch.

I guess this --localclean was not in the released verison of debmirror when
Frans took it over, and so was lost.

I'm not clear on what it did. Do you have the patch or remember? 

It must be different than --nocleanup somehow, but I don't understand
how. Using --nocleanup in this scenario wouldn't quite work, because
while the multiple sources could be combined, old files would never be expired.

-- 
see shy jo

[signature.asc (application/pgp-signature, inline)]

Send a report that this bug log contains spam.