Debian Bug report logs - #516241
.haddock files need to be put in -dev

Package: haskell-devscripts; Maintainer for haskell-devscripts is Debian Haskell Group <pkg-haskell-maintainers@lists.alioth.debian.org>; Source for haskell-devscripts is src:haskell-devscripts.

Reported by: Joachim Breitner <nomeata@debian.org>

Date: Fri, 20 Feb 2009 00:18:04 UTC

Severity: grave

Done: Kari Pahula <kaol@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Kari Pahula <kaol@debian.org>:
Bug#516241; Package haddock. (Fri, 20 Feb 2009 00:18:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Joachim Breitner <nomeata@debian.org>:
New Bug report received and forwarded. Copy sent to Kari Pahula <kaol@debian.org>. (Fri, 20 Feb 2009 00:18:06 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Joachim Breitner <nomeata@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock
Date: Fri, 20 Feb 2009 01:18:17 +0100
Package: haddock
Version: 2.4.1-3
Severity: grave

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Kaol,

I was about to start packaging libraries for ghc6.10 (haskell-X11 to be
precise), using the just built haddock from incoming, when I this
this problem:

$ /usr/bin/haddock --read-interface=/usr/share/doc/ghc6-doc/libraries/base,/usr/share/doc/ghc6-doc/libraries/base/base.haddock 
haddock: out of memory (requested 6579890946048 bytes)
$

The out of memory comes instantanious.

A strace shows that it tries to mmap a large region, after opening that file
open("/usr/share/doc/ghc6-doc/libraries/base/base.haddock", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=652796, ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff1df5ce90) = -1 ENOTTY (Inappropriate ioctl for device)
fstat(3, {st_mode=S_IFREG|0644, st_size=652796, ...}) = 0
mmap(0x7f6c14600000, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6c14300000
select(4, [3], [], NULL, {0, 0})        = 1 (in [3], left {0, 0})
read(3, "\r\f\372\316\0\4\0\t\213\206\0\t$y\377\0\0\6u\0\0\0\0\0\0\0\0\0\0\0\1\0\0"..., 652796) = 652796
close(3)                                = 0
mmap(0x7f6c14500000, 6579890946048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

Note that this is on amd64, with the just built ghc packages:

ii  ghc6            6.10.1+dfsg1-10 GHC - the Glasgow Haskell Compilation system
ii  ghc6-doc        6.10.1+dfsg1-10 Documentation for the Glasgow Haskell Compilat
ii  ghc6-prof       6.10.1+dfsg1-10 Profiling libraries for the Glasgow Haskell Co


The file in question is this:
$ ls -l /usr/share/doc/ghc6-doc/libraries/base/base.haddock
- -rw-r--r-- 1 root root 652796 17. Feb 18:52 /usr/share/doc/ghc6-doc/libraries/base/base.haddock
$ md5sum /usr/share/doc/ghc6-doc/libraries/base/base.haddock
31b9337459e9ebf3d59ca24cd03c827c  /usr/share/doc/ghc6-doc/libraries/base/base.haddock

if this is interesting at all.

Is there anything else I can do to help you debug this problem?

Greetings,
Joachim


- -- System Information:
Debian Release: 5.0
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.28-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages haddock depends on:
ii  libc6                    2.9-1           GNU C Library: Shared libraries
ii  libedit2                 2.11~20080614-1 BSD editline and history libraries
ii  libffi5                  3.0.7-1         Foreign Function Interface library
ii  libgmp3c2                2:4.2.2+dfsg-3  Multiprecision arithmetic library
ii  libncurses5              5.7+20090214-1  shared libraries for terminal hand

haddock recommends no packages.

haddock suggests no packages.

- -- no debconf information

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkmd9sYACgkQ9ijrk0dDIGyiPACgrj1nz933aAwmhuJd+9qoM0A+
rDQAmwRJTuQ2Q5SSIDd6K2Sv09InSybN
=BSQ2
-----END PGP SIGNATURE-----




Information forwarded to debian-bugs-dist@lists.debian.org, Kari Pahula <kaol@debian.org>:
Bug#516241; Package haddock. (Fri, 20 Feb 2009 01:09:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Joachim Breitner <nomeata@debian.org>:
Extra info received and forwarded to list. Copy sent to Kari Pahula <kaol@debian.org>. (Fri, 20 Feb 2009 01:09:02 GMT) Full text and rfc822 format available.

Message #10 received at 516241@bugs.debian.org (full text, mbox):

From: Joachim Breitner <nomeata@debian.org>
To: 516241@bugs.debian.org
Cc: debian-haskell@lists.debian.org
Subject: Re: out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock
Date: Fri, 20 Feb 2009 02:07:11 +0100
[Message part 1 (text/plain, inline)]
[CC’ing d-haskell, this might be a larger annoyance]

Hi again,

Am Freitag, den 20.02.2009, 01:18 +0100 schrieb Joachim Breitner:
> I was about to start packaging libraries for ghc6.10 (haskell-X11 to be
> precise), using the just built haddock from incoming, when I this
> this problem:
>
> For the full bug report, see:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=516241

I’m wondering: Are haddock’s interface files arch independent, in
haddock 2? If not, then that could be a problem – and if I read the code
in /usr/share/doc/ghc6-doc/libraries/base/base.haddock correctly, it
seems to use Binary to put “Int”s (without any fixed bit number).

And Binary does indeed serialize these Ints according to machine size...

So these are the first bytes
of /usr/share/doc/ghc6-doc/libraries/base/base.haddock:
0000000: 0d0c face 0004 0000 45cf 0000 4486 1100  ........E...D...
         ^-------^  magic number
                   ^--^ version number
                        ^-------^ 32 bit pointer
                                  ^-------^ another 32 bit pointer

and here is what haddock reads in these lines (prints added by me) with
the this code in src/Haddock/InterfaceFile.hs:

 where
   get_dictionary bin_handle = liftIO $ do
      dict_p <- get bin_handle
      liftIO $ print ("dict_p",dict_p)
      data_p <- tellBin bin_handle
      liftIO $ print ("data_p",data_p)
      seekBin bin_handle dict_p
      liftIO $ print ()
      dict <- getDictionary bin_handle
      seekBin bin_handle data_p
      return dict

which results in this output on my amd64 machine:

("dict_p",BinPtr 76755360564358)
("data_p",BinPtr 14)
haddock: out of memory (requested 8310762766336 bytes)

Now 76755360564358 is 0x000045CF00004486, which, by looking at the
relevant parts of the file, is the concatenation of the two pointers.


So yes, I’m very confident that haddock’s interface files are not arch
independent. Which is quite bad, I guess.

I see two solutions:
 * We patch haddock to not store any arch dependent data.
   (Probably quite some work)
 * We mach all -doc packages arch any instead of all.
   (Easiler but less elegant)
 * We put the haddock interface files in the -dev packages.
   (Not sure about the implications)

Greetings,
Joachim
-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
[signature.asc (application/pgp-signature, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Kari Pahula <kaol@debian.org>:
Bug#516241; Package haddock. (Fri, 20 Feb 2009 01:48:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to ddssff@gmail.com:
Extra info received and forwarded to list. Copy sent to Kari Pahula <kaol@debian.org>. (Fri, 20 Feb 2009 01:48:03 GMT) Full text and rfc822 format available.

Message #15 received at 516241@bugs.debian.org (full text, mbox):

From: David Fox <david@seereason.com>
To: Joachim Breitner <nomeata@debian.org>
Cc: 516241@bugs.debian.org, debian-haskell@lists.debian.org
Subject: Re: out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock
Date: Thu, 19 Feb 2009 17:45:55 -0800
[Message part 1 (text/plain, inline)]
On Thu, Feb 19, 2009 at 5:07 PM, Joachim Breitner <nomeata@debian.org> wrote


> So yes, I'm very confident that haddock's interface files are not arch
> independent. Which is quite bad, I guess.
>
> I see two solutions:
>  * We patch haddock to not store any arch dependent data.
>   (Probably quite some work)
>  * We mach all -doc packages arch any instead of all.
>   (Easiler but less elegant)
>  * We put the haddock interface files in the -dev packages.
>   (Not sure about the implications)
>
>
They are most certainly architecture dependent, and must be set to
architecture any.  All the packages in our repository have this change, and
the cabal-debian tool generates a debianization with that architecture.
Putting them in the dev package seems a bit unfortunate.  What I am not sure
about is whether they are compiler dependent -- if they are I guess the
package name should be libghc6-foo-doc rather than haskell-foo-doc.
[Message part 2 (text/html, inline)]

Information stored :
Bug#516241; Package haddock. (Fri, 20 Feb 2009 09:18:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Kari Pahula <kaol@debian.org>:
Extra info received and filed, but not forwarded. (Fri, 20 Feb 2009 09:18:08 GMT) Full text and rfc822 format available.

Message #20 received at 516241-quiet@bugs.debian.org (full text, mbox):

From: Kari Pahula <kaol@debian.org>
To: 516241-quiet@bugs.debian.org, debian-haskell@lists.debian.org
Subject: Re: out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock
Date: Fri, 20 Feb 2009 11:09:39 +0200
reassign 516241 haskell-devscripts
retitle 516241 .haddock files need to be put in -dev
done

On Thu, Feb 19, 2009 at 05:45:55PM -0800, David Fox wrote:
> On Thu, Feb 19, 2009 at 5:07 PM, Joachim Breitner <nomeata@debian.org> wrote
> >  * We mach all -doc packages arch any instead of all.
> >   (Easiler but less elegant)

I'd rather not, since the bulk of those is still HTML.  If an average
-doc is 500k, that times ~10 arches is 5MB.  That times 100 and
counting Haskell packages is a bit.

> >  * We put the haddock interface files in the -dev packages.
> >   (Not sure about the implications)

The implications are that without having a .haddock file available in
-doc, ghc6-doc's indices won't get updated without having the -dev
package installed, too.

There's that 50k-100k or so that a .haddock file will grow the -dev
package's size, whether or not a -doc is installed.  But as far as
space is concerned, it's still a lesser evil than arch:any -doc, IMHO.

I'm reassigning this to haskell-devscripts since it still puts
.haddock files to -doc.

What does haddock even use from a .haddock file, to generate that
library index?  There's the module names, links to the package
specific docs and the cabal name of the package it's from, but
anything else?  IMHO it wouldn't be too difficult to put that much
into some plain old text file.  Do we even need to carry full .haddock
files around, as far as -doc packages are concerned?  They're
necessary for building -doc packages, but that's a separate issue.




Bug reassigned from package `haddock' to `haskell-devscripts'. Request was from Kari Pahula <kaol@debian.org> to control@bugs.debian.org. (Fri, 20 Feb 2009 09:21:02 GMT) Full text and rfc822 format available.

Changed Bug title to `.haddock files need to be put in -dev' from `out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock'. Request was from Kari Pahula <kaol@debian.org> to control@bugs.debian.org. (Fri, 20 Feb 2009 09:21:04 GMT) Full text and rfc822 format available.

Information stored :
Bug#516241; Package haskell-devscripts. (Fri, 20 Feb 2009 11:51:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Joachim Breitner <nomeata@debian.org>:
Extra info received and filed, but not forwarded. (Fri, 20 Feb 2009 11:51:02 GMT) Full text and rfc822 format available.

Message #29 received at 516241-quiet@bugs.debian.org (full text, mbox):

From: Joachim Breitner <nomeata@debian.org>
To: debian-haskell@lists.debian.org
Cc: 516241-quiet@bugs.debian.org
Subject: Re: out of memory when reading /usr/share/doc/ghc6-doc/libraries/base/base.haddock
Date: Fri, 20 Feb 2009 12:49:38 +0100
[Message part 1 (text/plain, inline)]
Hi,

Am Freitag, den 20.02.2009, 11:09 +0200 schrieb Kari Pahula:
> On Thu, Feb 19, 2009 at 05:45:55PM -0800, David Fox wrote:
> > On Thu, Feb 19, 2009 at 5:07 PM, Joachim Breitner
> <nomeata@debian.org> wrote
> > >  * We mach all -doc packages arch any instead of all.
> > >   (Easiler but less elegant)
> 
> I'd rather not, since the bulk of those is still HTML.  If an average
> -doc is 500k, that times ~10 arches is 5MB.  That times 100 and
> counting Haskell packages is a bit.
> 
> > >  * We put the haddock interface files in the -dev packages.
> > >   (Not sure about the implications)
> 
> The implications are that without having a .haddock file available in
> -doc, ghc6-doc's indices won't get updated without having the -dev
> package installed, too.
> 
> There's that 50k-100k or so that a .haddock file will grow the -dev
> package's size, whether or not a -doc is installed.  But as far as
> space is concerned, it's still a lesser evil than arch:any -doc, IMHO.
> 
> I'm reassigning this to haskell-devscripts since it still puts
> .haddock files to -doc.

Note that it also needs to be fixed in ghc6 itself, of course, as
ghc6-doc is not built with haskell-devscripts. It also means that I’ll
have to wait for a new ghc6 build to get started here on amd64 :-)

> What does haddock even use from a .haddock file, to generate that
> library index?  There's the module names, links to the package
> specific docs and the cabal name of the package it's from, but
> anything else?  IMHO it wouldn't be too difficult to put that much
> into some plain old text file.  Do we even need to carry full .haddock
> files around, as far as -doc packages are concerned?  They're
> necessary for building -doc packages, but that's a separate issue.

I’m not sure if we should hack something up there, that needs to be
maintained and that’s duplicating existing functionality.

Putting the .haddock file in -dev should be fine, IMHO. But does that
mean that the -doc package should depend on the -dev package? Versioned
or unversioned?

Greetings,
Joachim

-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
[signature.asc (application/pgp-signature, inline)]

Reply sent to Kari Pahula <kaol@debian.org>:
You have taken responsibility. (Wed, 25 Feb 2009 13:57:04 GMT) Full text and rfc822 format available.

Notification sent to Joachim Breitner <nomeata@debian.org>:
Bug acknowledged by developer. (Wed, 25 Feb 2009 13:57:04 GMT) Full text and rfc822 format available.

Message #34 received at 516241-done@bugs.debian.org (full text, mbox):

From: Kari Pahula <kaol@debian.org>
To: 516241-done@bugs.debian.org
Subject: Haddock files are (supposed to be) arch independent again
Date: Wed, 25 Feb 2009 15:56:18 +0200
I patched ghc6's Binary module.  Now it uses 4 bytes to store an Int
on all arches.  Trying to fit anything longer than that triggers a
runtime error on 64 bit architectures.

No need to try treat .haddock files as arch dependent anymore, just
put them under /usr/share in -doc.




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Thu, 26 Mar 2009 07:34:53 GMT) Full text and rfc822 format available.

Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Apr 20 15:53:29 2014; Machine Name: beach.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.