Debian Bug report logs - #355178
'internal error: job made no progress' on 25G file

version graph

Package: librsync1; Maintainer for librsync1 is Andrey Rahmatullin <wrar@wrar.name>; Source for librsync1 is src:librsync.

Reported by: dean gaudet <dean@arctic.org>

Date: Fri, 3 Mar 2006 21:18:02 UTC

Severity: important

Tags: patch

Found in version librsync1/0.9.7-1

Fixed in version librsync/0.9.7-4

Done: Daniel Baumann <daniel@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to dean gaudet <dean@arctic.org>:
New Bug report received and forwarded. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: dean gaudet <dean@arctic.org>
To: submit@bugs.debian.org
Subject: 'internal error: job made no progress' on 25G file
Date: Fri, 3 Mar 2006 12:54:14 -0800 (PST)
Package: librsync1
Version: 0.9.7-1

i wonder if you would consider applying this patch from upstream to fix 
problems which occur with files over 4GB.

see:

https://sourceforge.net/tracker/index.php?func=detail&aid=1439412&group_id=56125&atid=479441

and 

https://sourceforge.net/tracker/?func=detail&atid=479439&aid=1110812&group_id=56125.

thanks
-dean


---------- Forwarded message ----------
Date: Tue, 28 Feb 2006 14:10:05 +1100
From: David Kempe <dave@solutionsfirst.com.au>
To: rdiff-backup list <rdiff-backup-users@nongnu.org>
Subject: [rdiff-backup-users] [Fwd: [librsync-devel] 'internal error: job made
   no progress' on 25G file - Bug Request ID: 1110812]

Hi, here is some good news from a guy who has been working for this on me -
should put an end to the 4GB file limit.
Can we please have some extensive testing on this patch to see if it works?
you will need to patch your librsync and rebuild rdiff-backup against the new
librsync I think.
Here is the patch:


-----------------------------------

brsync-0.9.7/mdfour.h librsync-0.9.7dev/mdfour.h
--- librsync-0.9.7/mdfour.h     2004-02-08 10:17:57.000000000 +1100
+++ librsync-0.9.7dev/mdfour.h  2006-02-27 14:35:26.086597250 +1100
@@ -24,7 +24,7 @@
#include "types.h"

struct rs_mdfour {
-    int                 A, B, C, D;
+    unsigned int        A, B, C, D;
#if HAVE_UINT64
    uint64_t            totalN;
#else
diff -urN librsync-0.9.7/patch.c librsync-0.9.7dev/patch.c
--- librsync-0.9.7/patch.c      2004-09-18 07:35:50.000000000 +1000
+++ librsync-0.9.7dev/patch.c   2006-02-27 14:39:46.862894750 +1100
@@ -214,12 +214,12 @@
    void            *buf, *ptr;
    rs_buffers_t    *buffs = job->stream;

-    len = job->basis_len;
-
    /* copy only as much as will fit in the output buffer, so that we
     * don't have to block or store the input. */
-    if (len > buffs->avail_out)
+    if (job->basis_len > buffs->avail_out)
        len = buffs->avail_out;
+    else
+        len = job->basis_len;

    if (!len)
        return RS_BLOCKED;



Severity set to `important'. Request was from dean gaudet <dean@arctic.org> to control@bugs.debian.org. Full text and rfc822 format available.

Tags added: patch Request was from dean gaudet <dean@arctic.org> to control@bugs.debian.org. Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to dean gaudet <dean@arctic.org>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #14 received at 355178@bugs.debian.org (full text, mbox):

From: dean gaudet <dean@arctic.org>
To: 355178@bugs.debian.org
Subject: just hit this on a 4G file
Date: Tue, 25 Jul 2006 22:09:33 -0700 (PDT)
i'm hitting this bug on a file which is only 4315422720 bytes...

it'd be great if you could apply the patch before etch release... it fixes 
the problem.  thanks!

-dean



Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Faheem Mitha <faheem@email.unc.edu>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #19 received at 355178@bugs.debian.org (full text, mbox):

From: Faheem Mitha <faheem@email.unc.edu>
To: 355178@bugs.debian.org
Subject: Debian debs with patch fix for #355178
Date: Fri, 10 Nov 2006 17:41:02 -0500 (EST)
Hi,

I just ran into this bug while using rdiff-backup.

This was with a 4.2 Gig Zope Data.fs database file. I ignored it for a
bit hoping it would go away. It didn't, but I was relieved to learn
that this was a known bug with a patch.

I spent quite a lot of time yesterday, patching librsync and rebuilding 
rdiff-backup against it.

Adding the patch is trivial. Basically, trim it a bit and add it as 
debian/patches/4Gigbug.patch.

You can get librsync and rdiff-backup debs and deb sources (for sarge) 
from

deb http://dulci.org/~faheem/debian/ sarge main contrib non-free
deb-src http://dulci.org/~faheem/debian/ sarge main contrib non-free

As expected, it made the problem go away.

It would be a good thing if this fix could go into librsync for etch.

Thanks.                                                       Faheem.



Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Michael Prokop <mika@grml.org>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #24 received at 355178@bugs.debian.org (full text, mbox):

From: Michael Prokop <mika@grml.org>
To: 355178@bugs.debian.org
Cc: dean gaudet <dean@arctic.org>, Faheem Mitha <faheem@email.unc.edu>
Subject: [#355178] unable to reproduce the >4GB librsync1 problem
Date: Sat, 18 Nov 2006 11:57:20 +0100
[Message part 1 (text/plain, inline)]
Hello,

my attention to this problem has been brought to by a grml-user who
pointed me to #355178.

I tried to verify the problem, but failed with using librsync1
0.9.7-1 and rdiff-backup 1.1.5-3 (from Debian/unstable).

I don't have any problems with files >4GB, see:

% ls -lah **
backup:
total 5.2G
drwxr-xr-x 3 mika mika 4.0K 2006-11-18 11:15 ./
drwxr-xr-x 4 mika mika 4.0K 2006-11-18 11:15 ../
drwx------ 3 mika mika 4.0K 2006-11-18 11:31 rdiff-backup-data/
-rw-r--r-- 1 mika mika 4.6G 2006-11-18 11:06 testfile
-rw-r--r-- 1 mika mika 524M 2006-11-18 10:58 testfile.tmp

original:
total 5.2G
drwxr-xr-x 2 mika mika 4.0K 2006-11-18 11:15 ./
drwxr-xr-x 4 mika mika 4.0K 2006-11-18 11:15 ../
-rw-r--r-- 1 mika mika 4.6G 2006-11-18 11:06 testfile
-rw-r--r-- 1 mika mika 524M 2006-11-18 10:58 testfile.tmp

Running rdiff-backup works without any problems:

% rdiff-backup --print-statistics original backup
--------------[ Session statistics ]--------------
StartTime 1163846987.00 (Sat Nov 18 11:49:47 2006)
EndTime 1163846987.71 (Sat Nov 18 11:49:47 2006)
ElapsedTime 0.71 (0.71 seconds)
SourceFiles 3
SourceFileSize 5485148160 (5.11 GB)
MirrorFiles 3
MirrorFileSize 5485148160 (5.11 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 0
ChangedSourceSize 0 (0 bytes)
ChangedMirrorSize 0 (0 bytes)
IncrementFiles 0
IncrementFileSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
--------------------------------------------------

So can you please provide the necessary steps to reproduce the problem?

regards,
-mika-
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to dean gaudet <dean@arctic.org>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #29 received at 355178@bugs.debian.org (full text, mbox):

From: dean gaudet <dean@arctic.org>
To: Michael Prokop <mika@grml.org>
Cc: 355178@bugs.debian.org, Faheem Mitha <faheem@email.unc.edu>
Subject: Re: [#355178] unable to reproduce the >4GB librsync1 problem
Date: Sat, 18 Nov 2006 10:39:44 -0800 (PST)
On Sat, 18 Nov 2006, Michael Prokop wrote:

> So can you please provide the necessary steps to reproduce the problem?

iirc it doesn't happen on every file >4GB.

try between a 32-bit and a 64-bit host -- that's when it was hitting me 
the worst.

-dean



Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Faheem Mitha <faheem@email.unc.edu>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #34 received at 355178@bugs.debian.org (full text, mbox):

From: Faheem Mitha <faheem@email.unc.edu>
To: dean gaudet <dean@arctic.org>
Cc: Michael Prokop <mika@grml.org>, 355178@bugs.debian.org
Subject: Re: [#355178] unable to reproduce the >4GB librsync1 problem
Date: Sat, 18 Nov 2006 14:00:22 -0500 (EST)

On Sat, 18 Nov 2006, dean gaudet wrote:

> On Sat, 18 Nov 2006, Michael Prokop wrote:
>
>> So can you please provide the necessary steps to reproduce the problem?
>
> iirc it doesn't happen on every file >4GB.
>
> try between a 32-bit and a 64-bit host -- that's when it was hitting me
> the worst.

Yes. For what it is worth, between a 32 bit and 64 bit host is where
it showed up for me.

                                                                 Faheem.



Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Allard Hoeve <allard@byte.nl>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #39 received at 355178@bugs.debian.org (full text, mbox):

From: Allard Hoeve <allard@byte.nl>
To: 355178@bugs.debian.org
Subject: Happens on my 32 bits Etch box
Date: Wed, 16 May 2007 12:42:20 +0200 (CEST)
Please fix this bug. This is easily reproducable on any 32 bits box by 
running for example rdiff-backup on some > 4 GB file.

Regards,

Allard




Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to "Thue Janus Kristensen" <thuejk@gmail.com>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #44 received at 355178@bugs.debian.org (full text, mbox):

From: "Thue Janus Kristensen" <thuejk@gmail.com>
To: 355178@bugs.debian.org
Subject: I am hitting this too
Date: Wed, 20 Jun 2007 12:05:04 +0200
I am also hitting this, while rdiff-backupping between two x86-32 systems.

A fix in the official Debian packages would be very handy, compared to
compiling, installing and maintaining my own packages.

Regards, Thue



Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Faheem Mitha <faheem@email.unc.edu>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #49 received at 355178@bugs.debian.org (full text, mbox):

From: Faheem Mitha <faheem@email.unc.edu>
To: Thue Janus Kristensen <thuejk@gmail.com>, 355178@bugs.debian.org
Subject: Re: Bug#355178: I am hitting this too
Date: Wed, 20 Jun 2007 13:27:15 -0400 (EDT)

On Wed, 20 Jun 2007, Thue Janus Kristensen wrote:

> I am also hitting this, while rdiff-backupping between two x86-32 systems.
>
> A fix in the official Debian packages would be very handy, compared to
> compiling, installing and maintaining my own packages.

Somebody really needs to do an NMU for this bug.

                                         Faheem.




Blocking bugs of 429797 added: 355178 Request was from Daniel Baumann <daniel@debian.org> to control@bugs.debian.org. (Sun, 15 Jul 2007 19:15:03 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org, Simon Law <sfllaw@debian.org>:
Bug#355178; Package librsync1. Full text and rfc822 format available.

Acknowledgement sent to Barrie Millar <solar@frondeg.co.uk>:
Extra info received and forwarded to list. Copy sent to Simon Law <sfllaw@debian.org>. Full text and rfc822 format available.

Message #56 received at 355178@bugs.debian.org (full text, mbox):

From: Barrie Millar <solar@frondeg.co.uk>
To: 355178@bugs.debian.org
Subject: Bug still exists
Date: Mon, 12 May 2008 19:13:46 +0100
Hi 

I can confirm this bug still exists. I'm encountering it on some large
files when using rdiff-backup. It either results in a process loop
maximising CPU load, or provides a librsync 107 error as illustrated in
this thread.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=429797

The initial report of this bug was opened in March 2006. I think it
really needs to be fixed :)

Regards,

Barrie




Reply sent to Daniel Baumann <daniel@debian.org>:
You have taken responsibility. Full text and rfc822 format available.

Notification sent to dean gaudet <dean@arctic.org>:
Bug acknowledged by developer. Full text and rfc822 format available.

Message #61 received at 355178-close@bugs.debian.org (full text, mbox):

From: Daniel Baumann <daniel@debian.org>
To: 355178-close@bugs.debian.org
Subject: Bug#355178: fixed in librsync 0.9.7-4
Date: Tue, 24 Jun 2008 09:02:36 +0000
Source: librsync
Source-Version: 0.9.7-4

We believe that the bug you reported is fixed in the latest version of
librsync, which is due to be installed in the Debian FTP archive:

librsync-dbg_0.9.7-4_i386.deb
  to pool/main/libr/librsync/librsync-dbg_0.9.7-4_i386.deb
librsync-dev_0.9.7-4_i386.deb
  to pool/main/libr/librsync/librsync-dev_0.9.7-4_i386.deb
librsync1_0.9.7-4_i386.deb
  to pool/main/libr/librsync/librsync1_0.9.7-4_i386.deb
librsync_0.9.7-4.diff.gz
  to pool/main/libr/librsync/librsync_0.9.7-4.diff.gz
librsync_0.9.7-4.dsc
  to pool/main/libr/librsync/librsync_0.9.7-4.dsc
rdiff_0.9.7-4_i386.deb
  to pool/main/libr/librsync/rdiff_0.9.7-4_i386.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 355178@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Daniel Baumann <daniel@debian.org> (supplier of updated librsync package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Tue, 24 Jun 2008 10:29:00 +0200
Source: librsync
Binary: librsync1 librsync-dev librsync-dbg rdiff
Architecture: source i386
Version: 0.9.7-4
Distribution: unstable
Urgency: high
Maintainer: Daniel Baumann <daniel@debian.org>
Changed-By: Daniel Baumann <daniel@debian.org>
Description: 
 librsync-dbg - rsync remote-delta algorithm library (debug)
 librsync-dev - rsync remote-delta algorithm library (development)
 librsync1  - rsync remote-delta algorithm library
 rdiff      - Binary diff tool for signature-based differences
Closes: 288758 355178
Changes: 
 librsync (0.9.7-4) unstable; urgency=high
 .
   * Adding patch from upstream to fix problem when transfering >4GB from
     a 32bit to 64bit host (Closes: #355178).
   * Updating manpage patch to remove dead link to Linuxcare Australia
     (Closes: #288758).
Checksums-Sha1: 
 cd8421f5c3056d7b65d540c59a0caf9227de5c7a 1187 librsync_0.9.7-4.dsc
 4bce416cdefd46a77e7479a1ccc020fb5ac938d2 4940 librsync_0.9.7-4.diff.gz
 edc9f58b01cdc622fb4df58e33a6ddf1fbf69a6a 68612 librsync1_0.9.7-4_i386.deb
 aa291673c3780ab34038a454894af1fcb85afb78 63306 librsync-dev_0.9.7-4_i386.deb
 6d2d690cf770726e6c524bec0dd5f72eed1de9b0 79566 librsync-dbg_0.9.7-4_i386.deb
 089561d882f7b62d3189288b57d99561bf937b4f 41358 rdiff_0.9.7-4_i386.deb
Checksums-Sha256: 
 3c3d68d398cfcabdc47533b6ad780bdbfd5545b67603633b02a42292f08811fa 1187 librsync_0.9.7-4.dsc
 2cfff99188445797dc18439cdd69804f50de7390561bc15fce7e1ce2e6749523 4940 librsync_0.9.7-4.diff.gz
 eac57e31894c4cd1b67f1ea0390ad2f58dbf7c830c31f0f88a062c6093871dd8 68612 librsync1_0.9.7-4_i386.deb
 0d21ca177f0ba334042ee2e1fc9243d24a7301c2a2715287736a2b226a2f7e23 63306 librsync-dev_0.9.7-4_i386.deb
 ace5acfd64f7c375004dcd6a3e8722c88853762c10a63fbffcf88059f20a5933 79566 librsync-dbg_0.9.7-4_i386.deb
 0ff5dd2f6dc507a34cd009dd18560923e2c6c2319a190ed583d5f7ddb97280b8 41358 rdiff_0.9.7-4_i386.deb
Files: 
 fc3678ce20b1d6f777403d00f2010b74 1187 utils optional librsync_0.9.7-4.dsc
 e09ed401de3c314d0aedc581c193cda8 4940 utils optional librsync_0.9.7-4.diff.gz
 393c9d71bbf102219343c55cca756b02 68612 libs optional librsync1_0.9.7-4_i386.deb
 e10b67b1bb295870e3b3e47453d67f81 63306 libdevel optional librsync-dev_0.9.7-4_i386.deb
 ae5ed8e6c21dca7a251dd95452510513 79566 devel extra librsync-dbg_0.9.7-4_i386.deb
 02bf359c25809e595348b17f219ef90b 41358 utils optional rdiff_0.9.7-4_i386.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkhgsQQACgkQ+C5cwEsrK55/5wCg39fXe3FGLUQ3ZgJ25irDozp5
TrwAniC7sQTYMZORusD9410Y2GvegCp+
=kO2t
-----END PGP SIGNATURE-----





Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Wed, 23 Jul 2008 07:34:22 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sat Apr 19 00:25:52 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.