Debian Bug report logs - #284274
fdupes: option to replace duplicates with hard links

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Rupert Levene <rlevene@uwaterloo.ca>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: fdupes: option to replace duplicates with hard links

Date: Sat, 04 Dec 2004 23:58:46 -0500

Package: fdupes
Version: 1.40-4
Severity: wishlist


It would be nice to have the option of telling fdupes to replace
duplicate files with hard links. This would be a more symmetric
behaviour than using symlinks.

-- System Information:
Debian Release: 3.1
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.8-1-686
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)

Versions of packages fdupes depends on:
ii  libc6                       2.3.2.ds1-18 GNU C Library: Shared libraries an

-- no debconf information

Message #10 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Samuel Thibault <samuel.thibault@ens-lyon.org>

To: 284274@bugs.debian.org

Cc: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: Re: fdupes: option to replace duplicates with hard links

Date: Wed, 28 Jun 2006 14:30:07 +0200

Rupert Levene, le Sat 04 Dec 2004 23:58:46 -0500, a écrit :
> It would be nice to have the option of telling fdupes to replace
> duplicate files with hard links. This would be a more symmetric
> behaviour than using symlinks.

I'd like this option too.

Message #15 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Javier Fernández-Sanguino Peña <jfs@computer.org>

To: 284274@bugs.debian.org

Cc: control@bugs.debian.org

Subject: Patch for the hardlink replacement bug request

Date: Fri, 31 Jul 2009 00:56:17 +0200

[Message part 1 (text/plain, inline)]

tags 284274 patch
thanks

Attached is a patch to the program sources (through the use of a dpatch patch
in the Debian package) that adds a new -L / --linkhard option to fdupes. This
option will replace all duplicate files with hardlinks which is useful in
order to reduce space.

It has been tested only slightly, but the code looks (to me) about right.

Please consider this patch and include it in the Debian package.

Regards

Javier

[fdupes_hardlink_repace.diff (text/x-diff, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Sandro Tosi <morph@debian.org>

To: Javier Fernández-Sanguino Peña <jfs@computer.org>, 284274@bugs.debian.org, Adrian Lopez <adrian2@caribe.net>

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Fri, 31 Jul 2009 21:48:11 +0200

[Message part 1 (text/plain, inline)]

Hi Javier,

2009/7/31 Javier Fernández-Sanguino Peña <jfs@computer.org>:
>
> tags 284274 patch
> thanks
>
> Attached is a patch to the program sources (through the use of a dpatch patch
> in the Debian package) that adds a new -L / --linkhard option to fdupes. This
> option will replace all duplicate files with hardlinks which is useful in
> order to reduce space.

Thanks for the patch!

> It has been tested only slightly, but the code looks (to me) about right.
>
> Please consider this patch and include it in the Debian package.

I've added upstream in the loop, so he can comment.

Hi Adrian,
a fellow Debian Developer sent me a patch on fdupes to replace
duplicate files with hardlink. It would be nice if you can merge it
fdupe original source code.

Cheers,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

[fdupes_hardlink_repace.diff (text/x-diff, attachment)]

Message #29 received at 284274-close@bugs.debian.org (full text, mbox, reply):

From: Sandro Tosi <morph@debian.org>

To: 284274-close@bugs.debian.org

Subject: Bug#284274: fixed in fdupes 1.50-PR2-3

Date: Sat, 29 May 2010 09:48:35 +0000

Source: fdupes
Source-Version: 1.50-PR2-3

We believe that the bug you reported is fixed in the latest version of
fdupes, which is due to be installed in the Debian FTP archive:

fdupes_1.50-PR2-3.diff.gz
  to main/f/fdupes/fdupes_1.50-PR2-3.diff.gz
fdupes_1.50-PR2-3.dsc
  to main/f/fdupes/fdupes_1.50-PR2-3.dsc
fdupes_1.50-PR2-3_amd64.deb
  to main/f/fdupes/fdupes_1.50-PR2-3_amd64.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 284274@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Sandro Tosi <morph@debian.org> (supplier of updated fdupes package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sat, 29 May 2010 10:08:59 +0200
Source: fdupes
Binary: fdupes
Architecture: source amd64
Version: 1.50-PR2-3
Distribution: unstable
Urgency: low
Maintainer: Sandro Tosi <morph@debian.org>
Changed-By: Sandro Tosi <morph@debian.org>
Description: 
 fdupes     - identifies duplicate files within given directories
Closes: 284274 537138 544568
Changes: 
 fdupes (1.50-PR2-3) unstable; urgency=low
 .
   * debian/control
     - updated Homepage field
     - updated upstream location
     - added Vcs-* fields (now using Git on git.debian.org)
     - bump Standards-Version to 3.8.4 (no changes needed)
     - added ${misc:Depends} to Depends
   * debian/patches/50_bts284274_hardlinkreplace.dpatch
     - added -L / --linkhard to make fdupes replace files with hardlinks. Also
       update the manual page; thanks to Rupert Levene for the report and to
       Javier Fernández-Sanguino Peña for the patch; Closes: #284274
   * debian/patches/60_bts544568_fix_memleaks.dpatch
     - fix memory leaks in main(); thanks to jbdenis for the report and patch;
       Closes: #544568
   * debian/patches/70_bts537138_disambiguate_recurse.dpatch
     - disambiguate the options '--recurse' and '--recurse:' ; thanks to Pietro
       Battiston for the report; Closes: #537138
   * debian/source/format
     - set explicitly version 1.0 source format
Checksums-Sha1: 
 8158c2696dd04e51de39b2226f0b2e62a0ba14d8 1114 fdupes_1.50-PR2-3.dsc
 05061990a6944466a251800225a00a72707a64f3 9441 fdupes_1.50-PR2-3.diff.gz
 f8a8bd120f89a15b7a0df328ab0c3052c5cdfcc0 21058 fdupes_1.50-PR2-3_amd64.deb
Checksums-Sha256: 
 4ea15de81a4da57fbf0cff7ad22cf79ec551c9a02ad8cbb56f07a675179f52ee 1114 fdupes_1.50-PR2-3.dsc
 2c5e31866bdbc9b7206c2c10280f3aff79cb64a1c5c9ea558bee30744e71a405 9441 fdupes_1.50-PR2-3.diff.gz
 5afca833ed2ad2f1df7e57b00db5062f5ce3f42bcd1ad7926e6b95bc0a0e14cd 21058 fdupes_1.50-PR2-3_amd64.deb
Files: 
 cd8accd72d11c6b8075fcbf59cb13734 1114 utils optional fdupes_1.50-PR2-3.dsc
 cc6b6f7e77290a02289a7be5bc02b1ed 9441 utils optional fdupes_1.50-PR2-3.diff.gz
 45813dafc59527cc95ba9368965be69b 21058 utils optional fdupes_1.50-PR2-3_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkwAzMYACgkQAukwV0RN2VByFwCfTdNgD0Hu8aws78X09DYp4kwO
oDYAn2u/ZocTgbY+cOhBWhaIvo2OLFIj
=/jHh
-----END PGP SIGNATURE-----

Message #42 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Sandro Tosi <morph@debian.org>

To: 284274@bugs.debian.org

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Sun, 17 Jun 2012 21:54:59 +0200

Resending to the bug too, now that the bugs is reopened.

On Sun, Jun 17, 2012 at 9:40 PM, Sandro Tosi <morph@debian.org> wrote:
> unarchive 284274
> reopen 284274
> thanks
>
> On Fri, Jul 31, 2009 at 12:56 AM, Javier Fernández-Sanguino Peña
> <jfs@computer.org> wrote:
>>
>> tags 284274 patch
>> thanks
>>
>> Attached is a patch to the program sources (through the use of a dpatch patch
>> in the Debian package) that adds a new -L / --linkhard option to fdupes. This
>> option will replace all duplicate files with hardlinks which is useful in
>> order to reduce space.
>>
>> It has been tested only slightly, but the code looks (to me) about right.
>>
>> Please consider this patch and include it in the Debian package.
>
> As it turned out[1] this patch loses data if some of the file to
> replace are on different filesystems. I'm going to remove it from the
> package for now, but i'd be happy to evaluate a new patch.
>
> [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=677419
>
> Regards,
> --
> Sandro Tosi (aka morph, morpheus, matrixhasu)
> My website: http://matrixhasu.altervista.org/
> Me at Debian: http://wiki.debian.org/SandroTosi



-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Message #47 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Javier Fernández-Sanguino Peña <jfs@computer.org>

To: Sandro Tosi <morph@debian.org>

Cc: 284274@bugs.debian.org

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Mon, 18 Jun 2012 19:56:28 +0200

[Message part 1 (text/plain, inline)]

On Sun, Jun 17, 2012 at 09:40:49PM +0200, Sandro Tosi wrote:
> On Fri, Jul 31, 2009 at 12:56 AM, Javier Fernández-Sanguino Peña
> <jfs@computer.org> wrote:
> >
> > tags 284274 patch
> > thanks
> >
> > Attached is a patch to the program sources (through the use of a dpatch patch
> > in the Debian package) that adds a new -L / --linkhard option to fdupes. This
> > option will replace all duplicate files with hardlinks which is useful in
> > order to reduce space.
> >
> > It has been tested only slightly, but the code looks (to me) about right.
> >
> > Please consider this patch and include it in the Debian package.
> 
> As it turned out[1] this patch loses data if some of the file to
> replace are on different filesystems. I'm going to remove it from the
> package for now, but i'd be happy to evaluate a new patch.

It might not be too difficult to introduce a check in the patch that tries
the hard link and, if it fails, it restores the file and complains. I'll see
what I can do.

Regards

Javier

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Sandro Tosi <morph@debian.org>

To: 284274@bugs.debian.org, Javier Fernandez-Sanguino <jfs@computer.org>

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Wed, 20 Jun 2012 20:05:56 +0200

On Mon, Jun 18, 2012 at 7:56 PM, Javier Fernández-Sanguino Peña
<jfs@computer.org> wrote:
> It might not be too difficult to introduce a check in the patch that tries
> the hard link and, if it fails, it restores the file and complains. I'll see
> what I can do.

Having a working -L option would be awesome! I think that the above
algorithm would cause a lot of I/O if the files to link are big (order
of GBs). Maybe we can check if the files are on 2 different FSes and
not trying to hardlink them? I know it can cause a race condition, but
probably better than copying gigs around

Cheers,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Message #57 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Javier Fernandez-Sanguino <jfs@computer.org>

To: Sandro Tosi <morph@debian.org>

Cc: 284274@bugs.debian.org

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Thu, 21 Jun 2012 02:24:43 +0200

Hi there,

Maybe you did not understand the proposed algorithm, no copies are
involved, just file renaiming. I was thinking more in the lines of
doing this:

IF A and B are the same THEN
- move B to a temporary file (predefined name or random name, as long
as the file does not exist)
- create a hardlink B to A
    IF the hard link fails THEN
               * restore the temporary file to B (rename, not copy) and complain
    ELSE
              * emove the temporary file and declare success

I still have to investigate if detecting files across different
filesystems is something easy or not, but the approach above should
work (although there are some race conditions)


Regards

Javier

Message #62 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Sandro Tosi <morph@debian.org>

To: Javier Fernandez-Sanguino <jfs@computer.org>

Cc: 284274@bugs.debian.org

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Thu, 21 Jun 2012 08:46:35 +0200

Hi,

On Thu, Jun 21, 2012 at 2:24 AM, Javier Fernandez-Sanguino
<jfs@computer.org> wrote:
> Maybe you did not understand the proposed algorithm, no copies are
> involved, just file renaiming.

Ah Indeed, I didn't understand that - that's much better than what I
had understood on the first reply (delete + try to hardlink, if fails,
copy back the files from another location).

Thanks for clarifying & working on it.
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Message #67 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Albert Cahalan <acahalan@gmail.com>

To: 284274@bugs.debian.org

Cc: jfs@computer.org, morph@debian.org

Subject: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Sun, 21 Oct 2012 05:31:08 -0400

Getting rid of the only race condition that matters:

Create the link first, with an unused name. Instead of
relying on the return code, which may be wrong for NFS,
call stat to find out if you created the file. Rename the
link over top of the file it is intended to replace. In case
that fails, remove the temporary file.

I suggest temporary names that look like these:

.fdupes-vtYoH1PGPa4lj^LIOfL_i~
.fdupes-wz_7uNXC2R4-ftNq-gl,Z~
.fdupes-kf9_9EQmw-v0nv_-HcyKS~
.fdupes-BTR6AlGWjz@rVSC^+@j+-~
.fdupes--SaeXuxNfj1U0mltgmWNN~

(dotfile, string "fdupes", 128 random bits, tilde on end,
and nothing that would be likely to trip up a bash shell)

Note that you can't hope to support all the crazy things
that exist in current and **future** kernels. There are
numerous security modules, "mount --bind" tricks such
as file-on-file mounting, union filesystems, network fs
servers running on non-Linux systems, and so on. You
have to draw the line somewhere; just make a note in
the man page that the tool is intended for single-user
use in non-crazy situations.

Perfection is the enemy of good; we need this option
working again. Right now I'm desperately rewriting this
tool as a pile of nasty shell scripts, and I assure you
that I totally don't care about cross-filesystem issues.

Message #72 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Paul Seelig <pseelig@rumbero.org>

To: Javier Fernandez-Sanguino <jfs@computer.org>

Cc: Sandro Tosi <morph@debian.org>, 284274@bugs.debian.org

Subject: Re: Re: Bug#284274: Patch for the hardlink replacement bug request

Date: Sun, 10 Mar 2013 17:20:54 +0100

Hi,

for the time being, it would probably be much more reasonable to limit
that function to the current local filesystem only, instead of trying to
crack a nut with a sledgehammer.

Finding duplicates over filesystems should be considered being a special
use case which could/should be handled separately.

In the meantime, lots of users (e.g. not only yours truly) would be very
happy, if not even delighted, if they were able to deduplicate files via
hardlinking within the boundaries of a single file system.

Best regards,
Paul

On 06/21/2012 02:24 AM, Javier Fernandez-Sanguino wrote:
> I still have to investigate if detecting files across different
> filesystems is something easy or not, but the approach above should
> work (although there are some race conditions)
> 
> 
> Regards
> 
> Javier
> 

Message #77 received at 284274@bugs.debian.org (full text, mbox, reply):

From: Adam Porter <adam@alphapapa.net>

To: 284274@bugs.debian.org

Cc: 284274-subscribe@bugs.debian.org

Subject: Please re-add hardlink support

Date: Thu, 4 Jul 2013 10:44:57 -0500

There has been a newer hardlinks and hardlink-set-merging patch set
sitting on the Google Code bug report for almost two years:

http://code.google.com/p/fdupes/issues/detail?id=8#c5

This feature is sorely, sorely missed.  Running across chroots and
multiple symlinked filesystems is an extreme corner case which could
simply be left unsupported.  "We average users" would just like to
deduplicate photos and music, etc. in a single directory tree on a
single filesystem.

Message #89 received at 284274@bugs.debian.org (full text, mbox, reply):

From: mw2ph2rnr3@kalu.co.jp

To: 284274@bugs.debian.org

Subject: Notification status of your delivery (UPS 04680069)

Date: Wed, 22 Feb 2017 18:50:14 +0900

[Message part 1 (text/plain, inline)]

Dear Customer,



UPS courier was unable to contact you for your parcel delivery.



Please check the attachment for details!



Thanks and best regards,

Ronnie Andersen,

UPS Parcels Delivery Manager.

[UPS-Label-04680069.zip (application/zip, attachment)]

Message #94 received at 284274@bugs.debian.org (full text, mbox, reply):

From: goldenwatches@golden-watches.com

To: 284274@bugs.debian.org

Subject: Delivery Status Notification

Date: Fri, 28 Apr 2017 21:23:33 +0800

[Message part 1 (text/plain, inline)]

Dear Customer,

Please check the attachment for your item delivery details!

FedEx

-----BEGIN PGP PUBLIC KEY BLOCK-----

pt2kQh8jz2Y0RfCyormBBIymf8813rqNPXuKyqJ/8rIDoP/xugvatGX8qSXtnnSExMMA2/j7NinA
UvZSll4F/901Wk8vPe9Bdt4BVRd8Ln5Hm9RQzay+yb+HQyaMPr1ksU7aXJ9iSFVq9ajQdr6aV/K5
jj0prESOrsqY6qARna0meqVWaCc8JbgQxli+027wuRagmvYjd49Uu3EJmP8cRrvhhpzhZjY8vPAm
ceh3bU/OAv1TvSiTaKc3Q8uJm6r2Q1Hh2XrribGHLVqrZCTnqco/ASsQIvuidbta0axgXh2+V995
URLQ/5yL5Lcjla99UrwEqa6RG1rodEX+w41sxxNRncPzc6100LMEOoP5DCZfc6IxsdZVHRNbj/8e
6ZcSzQJ2gYkVP+ylnB9fmHeYfMCsox5du6KurOIAjVQ9wqtGXSWTXvFoIwJuqe2DU7mPqvgKa//r
Vow1y1QckoEWD9+UAg9PJ6k+b8etN/2iZKgQ4xNXSFO0vDIEgzt4THZqiTRpo12eXF7DRg493AS+
N+6DY8uMrPTSw0ntbMNNg1SaNlXw4Ui+ZyWBmZX6jeBQh6ky1tzuwRqfPPvPBHJ3bUiq037BRoeV
m7OJ9hM9xWGiJboxEaKhaOKarzTNTxQRU7e0hMTrzTOzk2znyb5Fi4S92DTmzAEjFT4cRtUhiELV
UWV3Mjva9msUxseb6DW2hWhxZ1alvDUdV1bQxC0VppkT6tw4crgsGwuCT7mTlJ9e18ZA25iIe3nA
cHGbRX6XdenQ4MANeefcKR6N83dAPywgnOKwTjsu1nM0tdFc8kKaaQJhouxV4xDGP/jSvkVx+rFw
4dslUtzGM6QpxE3G1qqcjc+rnxXTgU7Jjkagv/3N6fIEgKtyjOrw9oepBMBmMekzK9M5h7K+u17E
92nU0JlcyeTsRE0G6vCdNUYG8m80XyNgt3AWSUSvg5X34bdQrkEkr6nRwZMMBojGL9os53F47Wyi
uO6btvE9tz1ebms06vbzW5U7NzNloX4RZEApn6GLDi3g4yGmmZ87UV7EsLB1FOJYe7LXE5Ufe69i
cO2M6AcC0MD7Hjc2AazqhatJ2xK+1zQRsV60fT24mOPDtBsuXF5KqpaCydeAYV8EDbQ2GPu4JhQE
vP7MRUEjbSqzM5RFrA6Vu62zY0RxpKGFY33XXHSKhBSP2mLypBw8T1aXXaPI/pNZ+Y55d2TuXJ0S
aDydqYDJme1lj37x34QTvQ06eDroiibW88mDSabsaqGiKBfCJHjfvVXhNK5L67yVZIQxC7MbpCqm
vMkZK/HZvw5rfOWagT6CHxgFhI0ihtY7hDb2vXMQ3A==


-----END PGP PUBLIC KEY BLOCK-----

[FedEx-Parcel-ID-EANB27YN.zip (application/zip, attachment)]

Send a report that this bug log contains spam.