Debian Bug report logs - #818856
diffoscope: crashes on broken symlinks

[Message part 1 (text/plain, inline)]
Package: diffoscope
Version: 51
Severity: normal

I found a way to crash diffoscope with broken symlinks:

pabs@chianamo ~ $ mkdir foo bar 
pabs@chianamo ~ $ touch foo/baz
pabs@chianamo ~ $ ln -s asdasfahguwbagpiuasbdifgsabdf bar/baz
pabs@chianamo ~ $ diffoscope foo bar
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 177, in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 148, in run_diffoscope
    parsed_args.file1, parsed_args.file2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 92, in compare_root_paths
    return compare_directories(path1, path2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 104, in compare_directories
    return FilesystemDirectory(path1).compare(FilesystemDirectory(path2))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 157, in compare
    my_file, other_file, source=name)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 102, in compare_files
    if file1.has_same_content_as(file2):
  File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 143, in tool_check
    return original_function(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 186, in has_same_content_as
    other_size = os.path.getsize(other.path)
  File "/usr/lib/python3.5/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: 'bar/baz'

-- System Information:
Debian Release: stretch/sid
  APT prefers testing
  APT policy: (900, 'testing'), (860, 'testing-proposed-updates'), (850, 'buildd-testing-proposed-updates'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 'experimental'), (690, 'buildd-experimental'), (500, 'unstable-debug'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)

Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python3-libarchive-c   2.1-3
ii  python3-magic          1:5.25-2
ii  python3-pkg-resources  18.8-1
pn  python3:any            <none>

Versions of packages diffoscope recommends:
ii  acl                     2.2.52-3
ii  binutils-multiarch      2.26-5
ii  bzip2                   1.0.6-8
ii  caca-utils              0.99.beta19-2+b1
ii  colord                  1.2.12-1
ii  cpio                    2.11+dfsg-5
pn  default-jdk | java-sdk  <none>
ii  enjarify                20151118-1
ii  fontforge-extras        0.3-4
pn  fp-utils                <none>
ii  genisoimage             9:1.1.11-3
ii  gettext                 0.19.7-2
ii  ghc                     7.10.3-7
ii  ghostscript             9.18~dfsg-4
ii  gnupg                   1.4.20-4
pn  mono-utils              <none>
ii  pdftk                   2.02-3
ii  poppler-utils           0.38.0-2
ii  python3-debian          0.1.27
pn  python3-guestfs         <none>
ii  python3-rpm             4.12.0.1+dfsg1-3+b2
ii  python3-tlsh            3.4.4+20151206-1+b1
ii  rpm2cpio                4.12.0.1+dfsg1-3+b2
ii  sng                     1.1.0-1
ii  sqlite3                 3.11.1-1
ii  squashfs-tools          1:4.3-3
ii  unzip                   6.0-20
ii  vim-common              2:7.4.963-1+b2
ii  xz-utils                5.1.1alpha+20120614-2.1

Versions of packages diffoscope suggests:
ii  libjs-jquery  1.11.3+dfsg-4

-- no debconf information

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]
Hi,

> diffoscope: crashes on broken symlinks

(See also #810825 and #796262)

Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org / chris-lamb.co.uk
       `-

[Message part 1 (text/plain, inline)]
Hi, Pabs!

Please try an attached patch! Hopefully, it'll solve this problem. :)

Thank you!
Satyam
Satyam Zode
PICT, Pune

On Mon, Mar 21, 2016 at 8:02 AM, Paul Wise <pabs@debian.org> wrote:
> Package: diffoscope
> Version: 51
> Severity: normal
>
> I found a way to crash diffoscope with broken symlinks:
>
> pabs@chianamo ~ $ mkdir foo bar
> pabs@chianamo ~ $ touch foo/baz
> pabs@chianamo ~ $ ln -s asdasfahguwbagpiuasbdifgsabdf bar/baz
> pabs@chianamo ~ $ diffoscope foo bar
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 177, in main
>     sys.exit(run_diffoscope(parsed_args))
>   File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 148, in run_diffoscope
>     parsed_args.file1, parsed_args.file2)
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 92, in compare_root_paths
>     return compare_directories(path1, path2)
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 104, in compare_directories
>     return FilesystemDirectory(path1).compare(FilesystemDirectory(path2))
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 157, in compare
>     my_file, other_file, source=name)
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 102, in compare_files
>     if file1.has_same_content_as(file2):
>   File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 143, in tool_check
>     return original_function(*args, **kwargs)
>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 186, in has_same_content_as
>     other_size = os.path.getsize(other.path)
>   File "/usr/lib/python3.5/genericpath.py", line 50, in getsize
>     return os.stat(filename).st_size
> FileNotFoundError: [Errno 2] No such file or directory: 'bar/baz'
>
> -- System Information:
> Debian Release: stretch/sid
>   APT prefers testing
>   APT policy: (900, 'testing'), (860, 'testing-proposed-updates'), (850, 'buildd-testing-proposed-updates'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 'experimental'), (690, 'buildd-experimental'), (500, 'unstable-debug'), (1, 'experimental-debug')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores)
> Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
>
> Versions of packages diffoscope depends on:
> ii  python3-libarchive-c   2.1-3
> ii  python3-magic          1:5.25-2
> ii  python3-pkg-resources  18.8-1
> pn  python3:any            <none>
>
> Versions of packages diffoscope recommends:
> ii  acl                     2.2.52-3
> ii  binutils-multiarch      2.26-5
> ii  bzip2                   1.0.6-8
> ii  caca-utils              0.99.beta19-2+b1
> ii  colord                  1.2.12-1
> ii  cpio                    2.11+dfsg-5
> pn  default-jdk | java-sdk  <none>
> ii  enjarify                20151118-1
> ii  fontforge-extras        0.3-4
> pn  fp-utils                <none>
> ii  genisoimage             9:1.1.11-3
> ii  gettext                 0.19.7-2
> ii  ghc                     7.10.3-7
> ii  ghostscript             9.18~dfsg-4
> ii  gnupg                   1.4.20-4
> pn  mono-utils              <none>
> ii  pdftk                   2.02-3
> ii  poppler-utils           0.38.0-2
> ii  python3-debian          0.1.27
> pn  python3-guestfs         <none>
> ii  python3-rpm             4.12.0.1+dfsg1-3+b2
> ii  python3-tlsh            3.4.4+20151206-1+b1
> ii  rpm2cpio                4.12.0.1+dfsg1-3+b2
> ii  sng                     1.1.0-1
> ii  sqlite3                 3.11.1-1
> ii  squashfs-tools          1:4.3-3
> ii  unzip                   6.0-20
> ii  vim-common              2:7.4.963-1+b2
> ii  xz-utils                5.1.1alpha+20120614-2.1
>
> Versions of packages diffoscope suggests:
> ii  libjs-jquery  1.11.3+dfsg-4
>
> -- no debconf information
>
> --
> bye,
> pabs
>
> https://wiki.debian.org/PaulWise
>
>
>
> _______________________________________________
> Reproducible-builds mailing list
> Reproducible-builds@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[0001-fixed-issue-related-to-diffoscope-symlinks-crashing.patch (text/x-patch, attachment)]
[Message part 1 (text/plain, inline)]
Package: diffoscope
Version: 51
Severity: normal

Hi, Pabs!
Please try an attached patch. Hopefully, it'll solve this problem. :)

Thanks!
Satyam Zode
PICT, Pune

On Sun, Apr 3, 2016 at 2:02 AM, Satyam Zode <satyamzode@gmail.com> wrote:
> Hi, Pabs!
>
> Please try an attached patch! Hopefully, it'll solve this problem. :)
>
> Thank you!
> Satyam
> Satyam Zode
> PICT, Pune
>
>
> On Mon, Mar 21, 2016 at 8:02 AM, Paul Wise <pabs@debian.org> wrote:
>> Package: diffoscope
>> Version: 51
>> Severity: normal
>>
>> I found a way to crash diffoscope with broken symlinks:
>>
>> pabs@chianamo ~ $ mkdir foo bar
>> pabs@chianamo ~ $ touch foo/baz
>> pabs@chianamo ~ $ ln -s asdasfahguwbagpiuasbdifgsabdf bar/baz
>> pabs@chianamo ~ $ diffoscope foo bar
>> Traceback (most recent call last):
>>   File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 177, in main
>>     sys.exit(run_diffoscope(parsed_args))
>>   File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 148, in run_diffoscope
>>     parsed_args.file1, parsed_args.file2)
>>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 92, in compare_root_paths
>>     return compare_directories(path1, path2)
>>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 104, in compare_directories
>>     return FilesystemDirectory(path1).compare(FilesystemDirectory(path2))
>>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", line 157, in compare
>>     my_file, other_file, source=name)
>>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", line 102, in compare_files
>>     if file1.has_same_content_as(file2):
>>   File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 143, in tool_check
>>     return original_function(*args, **kwargs)
>>   File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", line 186, in has_same_content_as
>>     other_size = os.path.getsize(other.path)
>>   File "/usr/lib/python3.5/genericpath.py", line 50, in getsize
>>     return os.stat(filename).st_size
>> FileNotFoundError: [Errno 2] No such file or directory: 'bar/baz'
>>
>> -- System Information:
>> Debian Release: stretch/sid
>>   APT prefers testing
>>   APT policy: (900, 'testing'), (860, 'testing-proposed-updates'), (850, 'buildd-testing-proposed-updates'), (800, 'unstable'), (790, 'buildd-unstable'), (700, 'experimental'), (690, 'buildd-experimental'), (500, 'unstable-debug'), (1, 'experimental-debug')
>> Architecture: amd64 (x86_64)
>>
>> Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores)
>> Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8)
>> Shell: /bin/sh linked to /bin/dash
>> Init: systemd (via /run/systemd/system)
>>
>> Versions of packages diffoscope depends on:
>> ii  python3-libarchive-c   2.1-3
>> ii  python3-magic          1:5.25-2
>> ii  python3-pkg-resources  18.8-1
>> pn  python3:any            <none>
>>
>> Versions of packages diffoscope recommends:
>> ii  acl                     2.2.52-3
>> ii  binutils-multiarch      2.26-5
>> ii  bzip2                   1.0.6-8
>> ii  caca-utils              0.99.beta19-2+b1
>> ii  colord                  1.2.12-1
>> ii  cpio                    2.11+dfsg-5
>> pn  default-jdk | java-sdk  <none>
>> ii  enjarify                20151118-1
>> ii  fontforge-extras        0.3-4
>> pn  fp-utils                <none>
>> ii  genisoimage             9:1.1.11-3
>> ii  gettext                 0.19.7-2
>> ii  ghc                     7.10.3-7
>> ii  ghostscript             9.18~dfsg-4
>> ii  gnupg                   1.4.20-4
>> pn  mono-utils              <none>
>> ii  pdftk                   2.02-3
>> ii  poppler-utils           0.38.0-2
>> ii  python3-debian          0.1.27
>> pn  python3-guestfs         <none>
>> ii  python3-rpm             4.12.0.1+dfsg1-3+b2
>> ii  python3-tlsh            3.4.4+20151206-1+b1
>> ii  rpm2cpio                4.12.0.1+dfsg1-3+b2
>> ii  sng                     1.1.0-1
>> ii  sqlite3                 3.11.1-1
>> ii  squashfs-tools          1:4.3-3
>> ii  unzip                   6.0-20
>> ii  vim-common              2:7.4.963-1+b2
>> ii  xz-utils                5.1.1alpha+20120614-2.1
>>
>> Versions of packages diffoscope suggests:
>> ii  libjs-jquery  1.11.3+dfsg-4
>>
>> -- no debconf information
>>
>> --
>> bye,
>> pabs
>>
>> https://wiki.debian.org/PaulWise
>>
>>
>>
>> _______________________________________________
>> Reproducible-builds mailing list
>> Reproducible-builds@lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[0001-fixed-issue-related-to-diffoscope-symlinks-crashing.patch (text/x-patch, attachment)]
[Message part 1 (text/plain, inline)]
Hi Satyam!

Thanks for submitting a patch for this issue!

On Sun, Apr 03, 2016 at 02:08:52AM +0530, Satyam Zode wrote:
>          logger.debug('%s has_same_content %s', self, other)
>          # try comparing small files directly first
>          my_size = os.path.getsize(self.path)
> -        other_size = os.path.getsize(other.path)
> +        other_size = os.lstat(other.path).st_size
>          if my_size == other_size and my_size <= SMALL_FILE_THRESHOLD:
>              if open(self.path, 'rb').read() == open(other.path, 'rb').read():
>                  return True

Here you change only the other_size line, not my_size.
When other.path is a broken symlink, this will work (and diffoscope
doesn't crash).
But self.path could also be a broken symlink (if you reverse the
diffoscope arguments, `diffoscope bar foo`). In this case diffoscope
would crash with the same problem.

Can you please amend your patch to fix this also?

Kind regards,
  Reiner

[signature.asc (application/pgp-signature, inline)]
[Message part 1 (text/plain, inline)]
Hi Reiner!

Thank you for reviewing patch. I have made all the changes you
mentioned above. Please find an attachment :-)

Cheers,
Satyam Zode

[0001-Fixed-issue-related-to-diffoscope-symlinks-crashing.patch (text/x-patch, attachment)]
[Message part 1 (text/plain, inline)]
Hi Reiner!
I did some trivial changes. Please find an attached patch.
This patch closes #818856 and #796262.

Thanks!
Satyam Zode

[0001-Fixed-issue-related-to-diffoscope-symlinks-crashing-.patch (text/x-patch, attachment)]
[Message part 1 (text/plain, inline)]
On Thu, 2016-04-07 at 00:42 +0530, Satyam Zode wrote:

> Thank you for reviewing patch. I have made all the changes you
> mentioned above. Please find an attachment :-)

This will works better but will still give a crash when both symlinks
are broken and point to a filename of the same length; because open()
throws an IOError exception when it tries to open a broken symlink.

In addition, I think we need a test for this issue written before
fixing the issue, here are some test cases I can think of:

one broken symlink, one file

one file, one broken symlink
one broken symlink, one dir

one dir, one broken symlink
one working symlink to a file, one broken symlink
one broken symlink, one working symlink to a file
one working symlink to a dir, one broken symlink
one broken symlink, one
working symlink to a dir
two broken symlinks pointing at the same
location
two broken symlinks of the same size but different locations
two broken symlinks of different sizes

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[signature.asc (application/pgp-signature, inline)]
Source: diffoscope
Source-Version: 53

We believe that the bug you reported is fixed in the latest version of
diffoscope, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 818856@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Mattia Rizzolo <mattia@debian.org> (supplier of updated diffoscope package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Fri, 03 Jun 2016 14:23:16 +0000
Source: diffoscope
Binary: diffoscope
Architecture: source
Version: 53
Distribution: unstable
Urgency: medium
Maintainer: Reproducible builds folks <reproducible-builds@lists.alioth.debian.org>
Changed-By: Mattia Rizzolo <mattia@debian.org>
Description:
 diffoscope - in-depth comparison of files, archives, and directories
Closes: 818414 818856 820624 823874
Changes:
 diffoscope (53) unstable; urgency=medium
 .
   [ Reiner Herrmann ]
   * In the directory comparator, fall back to current directory instead of
     empty name. (Closes: #820624)
   * Improve deb(5) support. (Closes: #818414)
 .
   [ Mattia Rizzolo ]
   * debian/control: add myself to Uploaders.
   * Clean up temporary debuglink files.
   * Fix FileNotFoundError with broken symlinks. (Closes: #818856)
   * Decode Md5sumsFile in utf-8. (Closes: #823874)
   * Always suffix temporary files with '_diffoscope'.
   * Rewrite sanity check for the version to always run every time debian/rules
     is invoked, not only during a binary build.
 .
   [ Ed Maste ]
   * Add FreeBSD packages for required tools.
 .
   [ Ceridwen ]
   * Add README.rst to MANIFEST.in.
Checksums-Sha1:
 55d95d1bb6660fbf4a83524b40582ad01608c099 1956 diffoscope_53.dsc
 aac6e8078069ee98eb606416e3be1dbdbafb0a3e 287948 diffoscope_53.tar.xz
Checksums-Sha256:
 821000d0d1e83ab4373e66b3d0215bd3d99898e0249991d1b731c33e775706a2 1956 diffoscope_53.dsc
 f92d687c78e0a072ad9cf73f02b2977cf3272d698d1b30e8d0d8ca5519496e28 287948 diffoscope_53.tar.xz
Files:
 255870b59bd99d302defdcf360c780cd 1956 devel optional diffoscope_53.dsc
 670eb60a51dcb22aea779e4a49a3213e 287948 devel optional diffoscope_53.tar.xz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJXUaXPAAoJEEsEP825REVAILYP/RYS/lCXPBIHmLtAhHSiptlL
CmU7W82TSaUadVgYHvaJLAjTqhg/rrTIHVHJhFn7ZxIHBQDWhxIYaEk0sb35g3us
8YghnHFv5WgoJCsUEunT/Q08GKIJK/Ky1BJ49zsc/pCpaNmCWgTCyW/9Sf3L0KcY
5Ybj0mwbhL/7xhdY0QZv89BjVYYzxaMPGBiv/6/sK6rKnJAN4yWdyqwmrcDYFl6Q
M4CLpxRr+mwEenUueAaPh2f12n7IzOYo7OPTF8JzDQOLW343DBpBCceRVZCrHTAa
+8a1A/PuN3AUwbQy66tVE/J+Ap4CoORrfky33va6t+bd3ScOKHu9y0EPmlPLBrv7
Bp7lsfll/NmAV6ukN7PNxzxX8opFgcGw4xMi5svNSNMeFRQq3HqqLl73g29idJxl
jEoGHlIdQZAjRmW6sFEfJdLclbzOJdPhPBZdCfkY9FEKEIJ0/sbk9esKLyhIShbQ
0YYNJJYwBflD0cR8i6pDCoKy56lCGtBWunLOvMVJVS7UnA9taqFtMUtIgaxWHvBg
UM592iInYQVUksQJSlRrLBEXhMj1ht3+soY8BFkxNFqzbrwcRHLnZ2mH45nJKq1P
TnxpbQwxgFDXRvmIFDyE0drG5OmL6ku88c+4KOq6o+k4BtjnCY7HFx1+NVhHv04A
QNu1WpSpqdo/aDF/+pIo
=HnzJ
-----END PGP SIGNATURE-----

Debian Bug report logs - #818856 diffoscope: crashes on broken symlinks

Debian Bug report logs - #818856
diffoscope: crashes on broken symlinks