Debian Bug report logs - #767295
xl: apparent memory leak

version graph

Package: xen-utils-4.4; Maintainer for xen-utils-4.4 is (unknown);

Reported by: Gedalya <gedalya@gedalya.net>

Date: Wed, 29 Oct 2014 21:39:01 UTC

Severity: important

Found in version xen/4.4.1-3

Fixed in version xen/4.4.1-5

Done: Bastian Blank <waldi@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Wed, 29 Oct 2014 21:39:07 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
New Bug report received and forwarded. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Wed, 29 Oct 2014 21:39:07 GMT) (full text, mbox, link).


Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: xl: apparent memory leak
Date: Wed, 29 Oct 2014 17:36:39 -0400
Package: xen-utils-4.4
Version: 4.4.1-3

When booting domU's running amd64 jessie, I notice some memory problems 
with xl.

root@xen:~# pmap -x 4121
4121:   /usr/lib/xen-4.4/bin/xl create --quiet --defconfig 
/etc/xen/auto/mail_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     128       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
0000000000841000     288     240     240 rw---   [ anon ]
00007f9ca48b4000   23852   14464   14464 rw---   [ anon ]
00007f9ca8000000     132       8       8 rw---   [ anon ]
00007f9ca8021000   65404       0       0 -----   [ anon ]
00007f9cae639000     104       0       0 r-x-- libz.so.1.2.8
<< snip >>
00007ffff736e000     132     104     104 rw---   [ stack ]
00007ffff73f4000       8       8       0 r-x--   [ anon ]
00007ffff73f6000       8       0       0 r----   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB          119812   17124   15052

After init 6 within the VM:

root@xen:~# pmap -x 4121
4121:   /usr/lib/xen-4.4/bin/xl create --quiet --defconfig 
/etc/xen/auto/mail_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
0000000000841000     288     288     288 rw---   [ anon ]
0000000000889000    2292    2292    2292 rw---   [ anon ]
00007f9ca3169000   23852   14464   14464 rw---   [ anon ]
00007f9ca48b4000   23852   14464   14464 rw---   [ anon ]
00007f9ca8000000     132       8       8 rw---   [ anon ]
00007f9ca8021000   65404       0       0 -----   [ anon ]
00007f9cae639000     104      88       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB          145964   34620   31864

another init 6:

root@xen:~# pmap -x 4121
4121:   /usr/lib/xen-4.4/bin/xl create --quiet --defconfig 
/etc/xen/auto/mail_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
0000000000841000     288     288     288 rw---   [ anon ]
0000000000889000    2292    2292    2292 rw---   [ anon ]
00007f9ca1a1e000   23852   14464   14464 rw---   [ anon ]
00007f9ca3169000   23852   14464   14464 rw---   [ anon ]
00007f9ca48b4000   23852   14464   14464 rw---   [ anon ]
<< snip >>
---------------- ------- ------- -------
total kB          169816   49084   46328



---------------

For my older domUs running i386 wheezy, the initial size of xl is hugely 
smaller:

# pmap -x 10380
10380:   /usr/lib/xen-4.4/bin/xl create aptcache_deb70.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     128       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000015e3000     132     132     132 rw---   [ anon ]
00007fb484000000     132       8       8 rw---   [ anon ]
00007fb484021000   65404       0       0 -----   [ anon ]
00007fb4893d4000     104       0       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB           95808    2604     488

After first init 6:

# pmap -x 10380
10380:   /usr/lib/xen-4.4/bin/xl create aptcache_deb70.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000015e3000     132     132     132 rw---   [ anon ]
0000000001604000    5840    5780    5780 rw---   [ anon ]
00007fb484000000     132       8       8 rw---   [ anon ]
00007fb484021000   65404       0       0 -----   [ anon ]
00007fb4893d4000     104     100       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB          101656    8792    6276

And subsequent reboots don't make it grow any more.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Thu, 06 Nov 2014 12:18:10 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Thu, 06 Nov 2014 12:18:10 GMT) (full text, mbox, link).


Message #10 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@hellion.org.uk>
To: Gedalya <gedalya@gedalya.net>, 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: xl: apparent memory leak
Date: Thu, 06 Nov 2014 12:14:43 +0000
On Wed, 2014-10-29 at 17:36 -0400, Gedalya wrote:
> When booting domU's running amd64 jessie, I notice some memory problems 
> with xl.

I ran it (actually, upstream staging-4.4) under valgrind and it reported
after a reboot:

==5722== 24 bytes in 1 blocks are definitely lost in loss record 35 of 72
==5722==    at 0x4026B2D: malloc (vg_replace_malloc.c:299)
==5722==    by 0x41739FF: strdup (strdup.c:43)
==5722==    by 0x405F38E: libxl__device_disk_local_initiate_attach (libxl.c:2681)
==5722==    by 0x408D2A6: libxl__bootloader_run (libxl_bootloader.c:385)
==5722==    by 0x406B50C: initiate_domain_create (libxl_create.c:807)
==5722==    by 0x406B50C: do_domain_create (libxl_create.c:1354)
==5722==    by 0x406B6AF: libxl_domain_create_new (libxl_create.c:1377)
==5722==    by 0x8056182: create_domain (xl_cmdimpl.c:2254)
==5722==    by 0x8059F27: main_create (xl_cmdimpl.c:4407)
==5722==    by 0x804E119: main (xl.c:360)
==5722== 
==5722== 28 bytes in 1 blocks are definitely lost in loss record 39 of 72
==5722==    at 0x4026B2D: malloc (vg_replace_malloc.c:299)
==5722==    by 0x4279EBB: read_message (xs.c:1143)
==5722==    by 0x427A89B: read_thread (xs.c:1219)
==5722==    by 0x40E7953: start_thread (pthread_create.c:304)
==5722==    by 0x41CF95D: clone (clone.S:130)

And the bootloader one appears to increase with each reboot.

FTR I did:

        valgrind --leak-check=full xl cr -F /etc/xen/d32-1

(-F runs xl in the foreground, so this won't return to a prompt).

Then in another window:

        xl reboot d32-1
        
(equivalent to running init 6).

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Thu, 06 Nov 2014 14:15:04 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Thu, 06 Nov 2014 14:15:04 GMT) (full text, mbox, link).


Message #15 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@hellion.org.uk>
To: 767295@bugs.debian.org
Cc: Gedalya <gedalya@gedalya.net>
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Thu, 06 Nov 2014 14:12:33 +0000
On Thu, 2014-11-06 at 12:14 +0000, Ian Campbell wrote:
> On Wed, 2014-10-29 at 17:36 -0400, Gedalya wrote:
> > When booting domU's running amd64 jessie, I notice some memory problems 
> > with xl.
> 
> I ran it (actually, upstream staging-4.4) under valgrind and it reported
> after a reboot:
> 
> ==5722== 24 bytes in 1 blocks are definitely lost in loss record 35 of 72
> ==5722==    at 0x4026B2D: malloc (vg_replace_malloc.c:299)
> ==5722==    by 0x41739FF: strdup (strdup.c:43)
> ==5722==    by 0x405F38E: libxl__device_disk_local_initiate_attach (libxl.c:2681)
> ==5722==    by 0x408D2A6: libxl__bootloader_run (libxl_bootloader.c:385)
> ==5722==    by 0x406B50C: initiate_domain_create (libxl_create.c:807)
> ==5722==    by 0x406B50C: do_domain_create (libxl_create.c:1354)
> ==5722==    by 0x406B6AF: libxl_domain_create_new (libxl_create.c:1377)
> ==5722==    by 0x8056182: create_domain (xl_cmdimpl.c:2254)
> ==5722==    by 0x8059F27: main_create (xl_cmdimpl.c:4407)
> ==5722==    by 0x804E119: main (xl.c:360)
> ==5722== 
> ==5722== 28 bytes in 1 blocks are definitely lost in loss record 39 of 72
> ==5722==    at 0x4026B2D: malloc (vg_replace_malloc.c:299)
> ==5722==    by 0x4279EBB: read_message (xs.c:1143)
> ==5722==    by 0x427A89B: read_thread (xs.c:1219)
> ==5722==    by 0x40E7953: start_thread (pthread_create.c:304)
> ==5722==    by 0x41CF95D: clone (clone.S:130)

I've posted a fix for this upstream:
http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html

I've requested that it go into the next 4.4.x release. I also plan to
backport it to the package regardless once it is accepted into the dev
branch.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sat, 08 Nov 2014 02:45:04 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sat, 08 Nov 2014 02:45:04 GMT) (full text, mbox, link).


Message #20 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <ijc@hellion.org.uk>, 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Fri, 07 Nov 2014 21:32:04 -0500
[Message part 1 (text/plain, inline)]
On 11/06/2014 09:12 AM, Ian Campbell wrote:
> I've posted a fix for this upstream:
> http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html
>
> I've requested that it go into the next 4.4.x release. I also plan to
> backport it to the package regardless once it is accepted into the dev
> branch.
>
> Ian.

So I've tried building xen 4.4.1-3 with this patch (attached), and I 
replaced the binary packages libxen-4.4 and xen-utils-4.4 with the new ones.
The situation is the same: the xl process for jessie amd64 guests starts 
off at a resident size of 15mb as opposed to ~500kb for wheezy i386 
guests. At guest reboot it jumps to 17mb and then, after a few seconds, 
to ~34mb.
I also rebooted the host to help make sure the test is valid.

Thanks,

Gedalya

[0039-do-not-leak-diskpath-during-localdisk-attach.patch (text/x-patch, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sat, 08 Nov 2014 10:42:19 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sat, 08 Nov 2014 10:42:20 GMT) (full text, mbox, link).


Message #25 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@hellion.org.uk>
To: Gedalya <gedalya@gedalya.net>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sat, 08 Nov 2014 10:41:56 +0000
On Fri, 2014-11-07 at 21:32 -0500, Gedalya wrote:
> On 11/06/2014 09:12 AM, Ian Campbell wrote:
> > I've posted a fix for this upstream:
> > http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html
> >
> > I've requested that it go into the next 4.4.x release. I also plan to
> > backport it to the package regardless once it is accepted into the dev
> > branch.
> >
> > Ian.
> 
> So I've tried building xen 4.4.1-3 with this patch (attached), and I 
> replaced the binary packages libxen-4.4 and xen-utils-4.4 with the new ones.
> The situation is the same: the xl process for jessie amd64 guests starts 
> off at a resident size of 15mb as opposed to ~500kb for wheezy i386 
> guests. At guest reboot it jumps to 17mb and then, after a few seconds, 
> to ~34mb.
> I also rebooted the host to help make sure the test is valid.

Please can you try running xl under valgrind, something similar to what
I described earlier should work.

Could you also post your guest cfg file please.

Thanks,
Ian.





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sat, 08 Nov 2014 13:36:04 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sat, 08 Nov 2014 13:36:04 GMT) (full text, mbox, link).


Message #30 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <ijc@hellion.org.uk>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sat, 08 Nov 2014 08:32:58 -0500
On 11/08/2014 05:41 AM, Ian Campbell wrote:
> Please can you try running xl under valgrind, something similar to what
> I described earlier should work.
I guess it didn't find much..

# valgrind --leak-check=full xl cr -F /etc/xen/auto/asterisk_deb80.cfg
==6736== Memcheck, a memory error detector
==6736== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==6736== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==6736== Command: /usr/sbin/xl cr -F /etc/xen/auto/asterisk_deb80.cfg
==6736==
==6740==
==6740== HEAP SUMMARY:
==6740==     in use at exit: 11,663 bytes in 40 blocks
==6740==   total heap usage: 74 allocs, 34 frees, 44,570 bytes allocated
==6740==
==6740== LEAK SUMMARY:
==6740==    definitely lost: 0 bytes in 0 blocks
==6740==    indirectly lost: 0 bytes in 0 blocks
==6740==      possibly lost: 0 bytes in 0 blocks
==6740==    still reachable: 11,663 bytes in 40 blocks
==6740==         suppressed: 0 bytes in 0 blocks
==6740== Reachable blocks (those to which a pointer was found) are not 
shown.
==6740== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6740==
==6740== For counts of detected and suppressed errors, rerun with: -v
==6740== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==6739==
==6739== HEAP SUMMARY:
==6739==     in use at exit: 10,711 bytes in 35 blocks
==6739==   total heap usage: 63 allocs, 28 frees, 35,257 bytes allocated
==6739==
==6739== LEAK SUMMARY:
==6739==    definitely lost: 0 bytes in 0 blocks
==6739==    indirectly lost: 0 bytes in 0 blocks
==6739==      possibly lost: 0 bytes in 0 blocks
==6739==    still reachable: 10,711 bytes in 35 blocks
==6739==         suppressed: 0 bytes in 0 blocks
==6739== Reachable blocks (those to which a pointer was found) are not 
shown.
==6739== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6739==
==6739== For counts of detected and suppressed errors, rerun with: -v
==6739== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==6744==
==6744== HEAP SUMMARY:
==6744==     in use at exit: 4,827 bytes in 42 blocks
==6744==   total heap usage: 91 allocs, 49 frees, 38,343 bytes allocated
==6744==
==6744== LEAK SUMMARY:
==6744==    definitely lost: 0 bytes in 0 blocks
==6744==    indirectly lost: 0 bytes in 0 blocks
==6744==      possibly lost: 0 bytes in 0 blocks
==6744==    still reachable: 4,827 bytes in 42 blocks
==6744==         suppressed: 0 bytes in 0 blocks
==6744== Reachable blocks (those to which a pointer was found) are not 
shown.
==6744== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6744==
==6744== For counts of detected and suppressed errors, rerun with: -v
==6744== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==6745==
==6745== HEAP SUMMARY:
==6745==     in use at exit: 13,034 bytes in 44 blocks
==6745==   total heap usage: 97 allocs, 53 frees, 48,672 bytes allocated
==6745==
==6745== LEAK SUMMARY:
==6745==    definitely lost: 0 bytes in 0 blocks
==6745==    indirectly lost: 0 bytes in 0 blocks
==6745==      possibly lost: 0 bytes in 0 blocks
==6745==    still reachable: 13,034 bytes in 44 blocks
==6745==         suppressed: 0 bytes in 0 blocks
==6745== Reachable blocks (those to which a pointer was found) are not 
shown.
==6745== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6745==
==6745== For counts of detected and suppressed errors, rerun with: -v
==6745== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==6738==
==6738== HEAP SUMMARY:
==6738==     in use at exit: 11,017 bytes in 41 blocks
==6738==   total heap usage: 91 allocs, 50 frees, 48,555 bytes allocated
==6738==
==6738== LEAK SUMMARY:
==6738==    definitely lost: 0 bytes in 0 blocks
==6738==    indirectly lost: 0 bytes in 0 blocks
==6738==      possibly lost: 0 bytes in 0 blocks
==6738==    still reachable: 11,017 bytes in 41 blocks
==6738==         suppressed: 0 bytes in 0 blocks
==6738== Reachable blocks (those to which a pointer was found) are not 
shown.
==6738== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==6738==
==6738== For counts of detected and suppressed errors, rerun with: -v
==6738== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Parsing config from /etc/xen/auto/asterisk_deb80.cfg
Waiting for domain asterisk_deb08 (domid 8) to die [pid 6736]
Domain 8 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
Domain 8 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain asterisk_deb08 (domid 9) to die [pid 6736]
Domain 9 has shut down, reason code 0 0x0
Action for shutdown reason code 0 is destroy
Domain 9 needs to be cleaned up: destroying the domain
Done. Exiting now


> Could you also post your guest cfg file please.

bootloader="pygrub"
memory = 1024
name = "asterisk_deb08"
vcpus = 2
pvh = 1
vif = ['mac=00:16:3e:00:00:12, bridge=breth1', 'mac=00:16:3e:00:00:13, 
bridge=breth0']
disk = ['phy:/dev/rvg0/asterisk_deb80_rootfs,xvda,w',
    'phy:/dev/rvg0/asterisk_deb80_var,xvdb,w',
    'phy:/dev/rvg0/asterisk_deb80_rec,xvdc,w',
    'phy:/dev/rvg0/asterisk_deb80_swap,xvdd,w']

(The pvh part doesn't seem to make a difference)





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sun, 09 Nov 2014 12:06:05 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 09 Nov 2014 12:06:05 GMT) (full text, mbox, link).


Message #35 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@hellion.org.uk>
To: Gedalya <gedalya@gedalya.net>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sun, 09 Nov 2014 12:02:13 +0000
On Sat, 2014-11-08 at 08:32 -0500, Gedalya wrote:
> On 11/08/2014 05:41 AM, Ian Campbell wrote:
> > Please can you try running xl under valgrind, something similar to what
> > I described earlier should work.
> I guess it didn't find much..

Indeed not :-/

I managed to get a test box up and running, so I backported a couple of
additional upstream patches to help valgrind analyse Xen toolstacks and
even with those I'm not seeing leaks.

However with pmap I am seeing an additional

+00007f37f516a000   23852   14464   14464 rw---   [ anon ]

with each reboot. The size seems  to be the same each time. I can't
quite figure out what this new mapping is. I'll keep digging.

[...]

> > Could you also post your guest cfg file please.
> 

Thanks, nothing unusual here but worth confirming.

Ian.

[0] b1dfc531d8ad and 171c6d7ac17e, FWIW




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sun, 09 Nov 2014 12:21:05 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 09 Nov 2014 12:21:05 GMT) (full text, mbox, link).


Message #40 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <ijc@hellion.org.uk>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sun, 09 Nov 2014 07:18:17 -0500
On 11/09/2014 07:02 AM, Ian Campbell wrote:
> On Sat, 2014-11-08 at 08:32 -0500, Gedalya wrote:
>> On 11/08/2014 05:41 AM, Ian Campbell wrote:
>>> Please can you try running xl under valgrind, something similar to what
>>> I described earlier should work.
>> I guess it didn't find much..
> Indeed not :-/
>
> I managed to get a test box up and running, so I backported a couple of
> additional upstream patches to help valgrind analyse Xen toolstacks and
> even with those I'm not seeing leaks.
>
> However with pmap I am seeing an additional
>
> +00007f37f516a000   23852   14464   14464 rw---   [ anon ]
>
> with each reboot. The size seems  to be the same each time. I can't
> quite figure out what this new mapping is. I'll keep digging.
>
> [...]
Yea. That's precisely what I was seeing. And even the first one 
shouldn't be there. There is no such block on xl processes for squeeze 
or wheezy VMs.
That memory block is just a bit larger than the size of the initrd in 
the VM, could there be a connection?


>
>>> Could you also post your guest cfg file please.
> Thanks, nothing unusual here but worth confirming.
>
> Ian.
>
> [0] b1dfc531d8ad and 171c6d7ac17e, FWIW
>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sun, 09 Nov 2014 12:33:09 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 09 Nov 2014 12:33:09 GMT) (full text, mbox, link).


Message #45 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <ijc@hellion.org.uk>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sun, 09 Nov 2014 07:28:55 -0500
On 11/09/2014 07:18 AM, Gedalya wrote:
> That memory block is just a bit larger than the size of the initrd in 
> the VM, could there be a connection? 

Nope. I changed the initrd to 2.6mb and that memory block is still 
exactly at 23852 / 14464  / 14464.
Actually the process size is around 12 mb when pygrub is counting down 
to boot, then jumps up to 14+ mb





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Sun, 09 Nov 2014 15:00:10 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@hellion.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Sun, 09 Nov 2014 15:00:10 GMT) (full text, mbox, link).


Message #50 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@hellion.org.uk>
To: Gedalya <gedalya@gedalya.net>
Cc: 767295@bugs.debian.org
Subject: Re: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
Date: Sun, 09 Nov 2014 14:57:17 +0000
On Sun, 2014-11-09 at 07:28 -0500, Gedalya wrote:
> On 11/09/2014 07:18 AM, Gedalya wrote:
> > That memory block is just a bit larger than the size of the initrd in 
> > the VM, could there be a connection? 
> 
> Nope. I changed the initrd to 2.6mb and that memory block is still 
> exactly at 23852 / 14464  / 14464.
> Actually the process size is around 12 mb when pygrub is counting down 
> to boot, then jumps up to 14+ mb

Running under gdb and dumping the contents of the leak block it appears
to be the uncompressed payout of the vmlinuz

i.e. using bzexplode[0]:

        $ file../bzexplode /boot/vmlinuz-3.16-3-amd64 | xzcat | od -t x4 | less
        0000000 464c457f 00010102 00000000 00000000
        0000020 003e0002 00000001 01000000 00000000
        0000040 00000040 00000000 00e1f160 00000000
        
vs:
        (gdb) x/32xw 0x00007fffea8b5000
        0x7fffea8b5000:	0x00000000	0x00000000	0x0174b002	0x00000000
        0x7fffea8b5010:	0x464c457f	0x00010102	0x00000000	0x00000000
        0x7fffea8b5020:	0x003e0002	0x00000001	0x01000000	0x00000000

(the first 4 words are no doubt the allocator's metadata).

I think I can see the leak now, it seems to be present in the latest
upstream git tree too.

Ian.

[0]
http://lists.xen.org/archives/html/xen-devel/2012-02/txtVnQboj3Whe.txt




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 03:15:05 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 03:15:05 GMT) (full text, mbox, link).


Message #55 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, Ian Campbell <ian.campbell@citrix.com>
Cc: xen-devel@lists.xen.org, ian.jackson@eu.citrix.com, wei.liu2@citrix.com, 767295@bugs.debian.org
Subject: Re: [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Thu, 20 Nov 2014 22:13:20 -0500
On 11/20/2014 03:21 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 20, 2014 at 03:48:47PM +0000, Ian Campbell wrote:
>> The libxc xc_dom_* infrastructure uses a very simple malloc memory pool which
>> is freed by xc_dom_release. However the various xc_try_*_decode routines (other
>> than the gzip one) just use plain malloc/realloc and therefore the buffer ends
>> up leaked.
>>
>> The memory pool currently supports mmap'd buffers as well as a directly
>> allocated buffers, however the try decode routines make use of realloc and do
>> not fit well into this model. Introduce a concept of an external memory block
>> to the memory pool and provide an interface to register such memory.
>>
>> The mmap_ptr and mmap_len fields of the memblock tracking struct lose their
>> mmap_ prefix since they are now also used for external memory blocks.
>>
>> We are only seeing this now because the gzip decoder doesn't leak and it's only
>> relatively recently that kernels in the wild have switched to better
>> compression.
>>
>> This is https://bugs.debian.org/767295
>>
>> Reported by: Gedalya <gedalya@gedalya.net>
> Gedelya,
>
> Could you also test this patch to make sure it does fix the
> reported issue please?

So here's what happens now.
1. Starts up tiny
2. reboot: leak
3. reboot: freed (process larger, but the delta is all/mostly shared pages)
4. reboot: leak
5. reboot: freed
etc..

root@xen:~/xen-pkgs# xl cr /etc/xen/auto/asterisk_deb80.cfg
Parsing config from /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
root     22981  0.0  0.0  95968   588 ?        SLsl 21:55   0:00 
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# pmap -x 22981
22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     128       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000009a6000     288     240     240 rw---   [ anon ]
00007f14d4000000     132       8       8 rw---   [ anon ]
00007f14d4021000   65404       0       0 -----   [ anon ]
<< snip >>
---------------- ------- ------- -------
total kB           95968    2728     596

--- reboot domu ---

root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
root     22981  0.6  3.3 131652 20008 ?        SLsl 21:55   0:00 
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# pmap -x 22981
22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000009a6000     288     288     288 rw---   [ anon ]
00000000009ee000   35676   16772   16772 rw---   [ anon ]
00007f14d4000000     132       8       8 rw---   [ anon ]
00007f14d4021000   65404       0       0 -----   [ anon ]
00007f14d9ae0000     104     100       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB          131652   20072   17424

--- reboot domu ---

root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
root     22981  0.5  0.5  95876  3136 ?        SLsl 21:55   0:01 
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# pmap -x 22981
22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000009a6000     188     188     188 rw---   [ anon ]
00007f14d4000000     132       8       8 rw---   [ anon ]
00007f14d4021000   65404       0       0 -----   [ anon ]
00007f14d9ae0000     104     100       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB           95876    3200     552

--- reboot domu ---

root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
root     22981  0.6  3.4 134660 20548 ?        SLsl 21:55   0:02 
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# pmap -x 22981
22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000009a6000     188     188     188 rw---   [ anon ]
00000000009d5000   38784   17348   17348 rw---   [ anon ]
00007f14d4000000     132       8       8 rw---   [ anon ]
00007f14d4021000   65404       0       0 -----   [ anon ]
00007f14d9ae0000     104     100       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB          134660   20548   17900

--- reboot domu ---

root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
root     22981  0.6  0.5  95876  3200 ?        SLsl 21:55   0:03 
/usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
root@xen:~/xen-pkgs# pmap -x 22981
22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     144     144       0 r-x-- xl
0000000000623000       4       4       4 r---- xl
0000000000624000       8       8       8 rw--- xl
0000000000626000       4       4       4 rw---   [ anon ]
00000000009a6000     188     188     188 rw---   [ anon ]
00007f14d4000000     132       8       8 rw---   [ anon ]
00007f14d4021000   65404       0       0 -----   [ anon ]
00007f14d9ae0000     104     100       0 r-x-- libz.so.1.2.8
<< snip >>
---------------- ------- ------- -------
total kB           95876    3200     552




Tried valgrind, it doesn't look like it was able to see what was going on

root@xen:~# valgrind --leak-check=full xl cr -F 
/etc/xen/auto/asterisk_deb80.cfg
==24967== Memcheck, a memory error detector
==24967== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==24967== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==24967== Command: /usr/sbin/xl cr -F /etc/xen/auto/asterisk_deb80.cfg
==24967==
==24971==
==24971== HEAP SUMMARY:
==24971==     in use at exit: 11,695 bytes in 41 blocks
==24971==   total heap usage: 75 allocs, 34 frees, 44,602 bytes allocated
==24971==
==24971== LEAK SUMMARY:
==24971==    definitely lost: 0 bytes in 0 blocks
==24971==    indirectly lost: 0 bytes in 0 blocks
==24971==      possibly lost: 0 bytes in 0 blocks
==24971==    still reachable: 11,695 bytes in 41 blocks
==24971==         suppressed: 0 bytes in 0 blocks
==24971== Reachable blocks (those to which a pointer was found) are not 
shown.
==24971== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24971==
==24971== For counts of detected and suppressed errors, rerun with: -v
==24971== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==24970==
==24970== HEAP SUMMARY:
==24970==     in use at exit: 10,743 bytes in 36 blocks
==24970==   total heap usage: 64 allocs, 28 frees, 35,289 bytes allocated
==24970==
==24970== LEAK SUMMARY:
==24970==    definitely lost: 0 bytes in 0 blocks
==24970==    indirectly lost: 0 bytes in 0 blocks
==24970==      possibly lost: 0 bytes in 0 blocks
==24970==    still reachable: 10,743 bytes in 36 blocks
==24970==         suppressed: 0 bytes in 0 blocks
==24970== Reachable blocks (those to which a pointer was found) are not 
shown.
==24970== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24970==
==24970== For counts of detected and suppressed errors, rerun with: -v
==24970== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==24975==
==24975== HEAP SUMMARY:
==24975==     in use at exit: 4,859 bytes in 43 blocks
==24975==   total heap usage: 92 allocs, 49 frees, 38,375 bytes allocated
==24975==
==24975== LEAK SUMMARY:
==24975==    definitely lost: 0 bytes in 0 blocks
==24975==    indirectly lost: 0 bytes in 0 blocks
==24975==      possibly lost: 0 bytes in 0 blocks
==24975==    still reachable: 4,859 bytes in 43 blocks
==24975==         suppressed: 0 bytes in 0 blocks
==24975== Reachable blocks (those to which a pointer was found) are not 
shown.
==24975== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24975==
==24975== For counts of detected and suppressed errors, rerun with: -v
==24975== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==24976==
==24976== HEAP SUMMARY:
==24976==     in use at exit: 13,066 bytes in 45 blocks
==24976==   total heap usage: 98 allocs, 53 frees, 48,704 bytes allocated
==24976==
==24976== LEAK SUMMARY:
==24976==    definitely lost: 0 bytes in 0 blocks
==24976==    indirectly lost: 0 bytes in 0 blocks
==24976==      possibly lost: 0 bytes in 0 blocks
==24976==    still reachable: 13,066 bytes in 45 blocks
==24976==         suppressed: 0 bytes in 0 blocks
==24976== Reachable blocks (those to which a pointer was found) are not 
shown.
==24976== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24976==
==24976== For counts of detected and suppressed errors, rerun with: -v
==24976== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==24969==
==24969== HEAP SUMMARY:
==24969==     in use at exit: 11,049 bytes in 42 blocks
==24969==   total heap usage: 92 allocs, 50 frees, 48,587 bytes allocated
==24969==
==24969== LEAK SUMMARY:
==24969==    definitely lost: 0 bytes in 0 blocks
==24969==    indirectly lost: 0 bytes in 0 blocks
==24969==      possibly lost: 0 bytes in 0 blocks
==24969==    still reachable: 11,049 bytes in 42 blocks
==24969==         suppressed: 0 bytes in 0 blocks
==24969== Reachable blocks (those to which a pointer was found) are not 
shown.
==24969== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==24969==
==24969== For counts of detected and suppressed errors, rerun with: -v
==24969== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Parsing config from /etc/xen/auto/asterisk_deb80.cfg
Waiting for domain asterisk_deb80 (domid 31) to die [pid 24967]
Domain 31 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
Domain 31 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain asterisk_deb80 (domid 32) to die [pid 24967]
Domain 32 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
Domain 32 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain asterisk_deb80 (domid 33) to die [pid 24967]
Domain 33 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
Domain 33 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain asterisk_deb80 (domid 34) to die [pid 24967]
Domain 34 has shut down, reason code 0 0x0
Action for shutdown reason code 0 is destroy
Domain 34 needs to be cleaned up: destroying the domain
Done. Exiting now


>
>> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
>> ---
>> v2: Correct handling in xc_try_lzo1x_decode.
>>
>> This is a bug fix and should go into 4.5.
>>
>> I have a sneaking suspicion this is going to need to backport a very long way...
>> ---
>>   tools/libxc/include/xc_dom.h        |   10 ++++--
>>   tools/libxc/xc_dom_bzimageloader.c  |   20 ++++++++++++
>>   tools/libxc/xc_dom_core.c           |   61 +++++++++++++++++++++++++++--------
>>   tools/libxc/xc_dom_decompress_lz4.c |    5 +++
>>   4 files changed, 80 insertions(+), 16 deletions(-)
>>
>> diff --git a/tools/libxc/include/xc_dom.h b/tools/libxc/include/xc_dom.h
>> index 6ae6a9f..07d7224 100644
>> --- a/tools/libxc/include/xc_dom.h
>> +++ b/tools/libxc/include/xc_dom.h
>> @@ -33,8 +33,13 @@ struct xc_dom_seg {
>>   
>>   struct xc_dom_mem {
>>       struct xc_dom_mem *next;
>> -    void *mmap_ptr;
>> -    size_t mmap_len;
>> +    void *ptr;
>> +    enum {
>> +        XC_DOM_MEM_TYPE_MALLOC_INTERNAL,
>> +        XC_DOM_MEM_TYPE_MALLOC_EXTERNAL,
>> +        XC_DOM_MEM_TYPE_MMAP,
>> +    } type;
>> +    size_t len;
>>       unsigned char memory[0];
>>   };
>>   
>> @@ -298,6 +303,7 @@ void xc_dom_log_memory_footprint(struct xc_dom_image *dom);
>>   /* --- simple memory pool ------------------------------------------ */
>>   
>>   void *xc_dom_malloc(struct xc_dom_image *dom, size_t size);
>> +int xc_dom_register_external(struct xc_dom_image *dom, void *ptr, size_t size);
>>   void *xc_dom_malloc_page_aligned(struct xc_dom_image *dom, size_t size);
>>   void *xc_dom_malloc_filemap(struct xc_dom_image *dom,
>>                               const char *filename, size_t * size,
>> diff --git a/tools/libxc/xc_dom_bzimageloader.c b/tools/libxc/xc_dom_bzimageloader.c
>> index 2225699..964ebdc 100644
>> --- a/tools/libxc/xc_dom_bzimageloader.c
>> +++ b/tools/libxc/xc_dom_bzimageloader.c
>> @@ -161,6 +161,13 @@ static int xc_try_bzip2_decode(
>>   
>>       total = (((uint64_t)stream.total_out_hi32) << 32) | stream.total_out_lo32;
>>   
>> +    if ( xc_dom_register_external(dom, out_buf, total) )
>> +    {
>> +        DOMPRINTF("BZIP2: Error registering stream output");
>> +        free(out_buf);
>> +        goto bzip2_cleanup;
>> +    }
>> +
>>       DOMPRINTF("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx",
>>                 __FUNCTION__, *size, (long unsigned int) total);
>>   
>> @@ -305,6 +312,13 @@ static int _xc_try_lzma_decode(
>>           }
>>       }
>>   
>> +    if ( xc_dom_register_external(dom, out_buf, stream->total_out) )
>> +    {
>> +        DOMPRINTF("%s: Error registering stream output", what);
>> +        free(out_buf);
>> +        goto lzma_cleanup;
>> +    }
>> +
>>       DOMPRINTF("%s: %s decompress OK, 0x%zx -> 0x%zx",
>>                 __FUNCTION__, what, *size, (size_t)stream->total_out);
>>   
>> @@ -464,7 +478,13 @@ static int xc_try_lzo1x_decode(
>>   
>>           dst_len = lzo_read_32(cur);
>>           if ( !dst_len )
>> +        {
>> +            msg = "Error registering stream output";
>> +            if ( xc_dom_register_external(dom, out_buf, out_len) )
>> +                break;
>> +
>>               return 0;
>> +        }
>>   
>>           if ( dst_len > LZOP_MAX_BLOCK_SIZE )
>>           {
>> diff --git a/tools/libxc/xc_dom_core.c b/tools/libxc/xc_dom_core.c
>> index baa62a1..ecbf981 100644
>> --- a/tools/libxc/xc_dom_core.c
>> +++ b/tools/libxc/xc_dom_core.c
>> @@ -132,6 +132,7 @@ void *xc_dom_malloc(struct xc_dom_image *dom, size_t size)
>>           return NULL;
>>       }
>>       memset(block, 0, sizeof(*block) + size);
>> +    block->type = XC_DOM_MEM_TYPE_MALLOC_INTERNAL;
>>       block->next = dom->memblocks;
>>       dom->memblocks = block;
>>       dom->alloc_malloc += sizeof(*block) + size;
>> @@ -151,23 +152,45 @@ void *xc_dom_malloc_page_aligned(struct xc_dom_image *dom, size_t size)
>>           return NULL;
>>       }
>>       memset(block, 0, sizeof(*block));
>> -    block->mmap_len = size;
>> -    block->mmap_ptr = mmap(NULL, block->mmap_len,
>> -                           PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
>> -                           -1, 0);
>> -    if ( block->mmap_ptr == MAP_FAILED )
>> +    block->len = size;
>> +    block->ptr = mmap(NULL, block->len,
>> +                      PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
>> +                      -1, 0);
>> +    if ( block->ptr == MAP_FAILED )
>>       {
>>           DOMPRINTF("%s: mmap failed", __FUNCTION__);
>>           free(block);
>>           return NULL;
>>       }
>> +    block->type = XC_DOM_MEM_TYPE_MMAP;
>>       block->next = dom->memblocks;
>>       dom->memblocks = block;
>>       dom->alloc_malloc += sizeof(*block);
>> -    dom->alloc_mem_map += block->mmap_len;
>> +    dom->alloc_mem_map += block->len;
>>       if ( size > (100 * 1024) )
>>           print_mem(dom, __FUNCTION__, size);
>> -    return block->mmap_ptr;
>> +    return block->ptr;
>> +}
>> +
>> +int xc_dom_register_external(struct xc_dom_image *dom, void *ptr, size_t size)
>> +{
>> +    struct xc_dom_mem *block;
>> +
>> +    block = malloc(sizeof(*block));
>> +    if ( block == NULL )
>> +    {
>> +        DOMPRINTF("%s: allocation failed", __FUNCTION__);
>> +        return -1;
>> +    }
>> +    memset(block, 0, sizeof(*block));
>> +    block->ptr = ptr;
>> +    block->len = size;
>> +    block->type = XC_DOM_MEM_TYPE_MALLOC_EXTERNAL;
>> +    block->next = dom->memblocks;
>> +    dom->memblocks = block;
>> +    dom->alloc_malloc += sizeof(*block);
>> +    dom->alloc_mem_map += block->len;
>> +    return 0;
>>   }
>>   
>>   void *xc_dom_malloc_filemap(struct xc_dom_image *dom,
>> @@ -212,24 +235,25 @@ void *xc_dom_malloc_filemap(struct xc_dom_image *dom,
>>       }
>>   
>>       memset(block, 0, sizeof(*block));
>> -    block->mmap_len = *size;
>> -    block->mmap_ptr = mmap(NULL, block->mmap_len, PROT_READ,
>> +    block->len = *size;
>> +    block->ptr = mmap(NULL, block->len, PROT_READ,
>>                              MAP_SHARED, fd, 0);
>> -    if ( block->mmap_ptr == MAP_FAILED ) {
>> +    if ( block->ptr == MAP_FAILED ) {
>>           xc_dom_panic(dom->xch, XC_INTERNAL_ERROR,
>>                        "failed to mmap file: %s",
>>                        strerror(errno));
>>           goto err;
>>       }
>>   
>> +    block->type = XC_DOM_MEM_TYPE_MMAP;
>>       block->next = dom->memblocks;
>>       dom->memblocks = block;
>>       dom->alloc_malloc += sizeof(*block);
>> -    dom->alloc_file_map += block->mmap_len;
>> +    dom->alloc_file_map += block->len;
>>       close(fd);
>>       if ( *size > (100 * 1024) )
>>           print_mem(dom, __FUNCTION__, *size);
>> -    return block->mmap_ptr;
>> +    return block->ptr;
>>   
>>    err:
>>       if ( fd != -1 )
>> @@ -246,8 +270,17 @@ static void xc_dom_free_all(struct xc_dom_image *dom)
>>       while ( (block = dom->memblocks) != NULL )
>>       {
>>           dom->memblocks = block->next;
>> -        if ( block->mmap_ptr )
>> -            munmap(block->mmap_ptr, block->mmap_len);
>> +        switch ( block->type )
>> +        {
>> +        case XC_DOM_MEM_TYPE_MALLOC_INTERNAL:
>> +            break;
>> +        case XC_DOM_MEM_TYPE_MALLOC_EXTERNAL:
>> +            free(block->ptr);
>> +            break;
>> +        case XC_DOM_MEM_TYPE_MMAP:
>> +            munmap(block->ptr, block->len);
>> +            break;
>> +        }
>>           free(block);
>>       }
>>   }
>> diff --git a/tools/libxc/xc_dom_decompress_lz4.c b/tools/libxc/xc_dom_decompress_lz4.c
>> index 490ec56..b6a33f2 100644
>> --- a/tools/libxc/xc_dom_decompress_lz4.c
>> +++ b/tools/libxc/xc_dom_decompress_lz4.c
>> @@ -103,6 +103,11 @@ int xc_try_lz4_decode(
>>   
>>   		if (size == 0)
>>   		{
>> +			if ( xc_dom_register_external(dom, output, out_len) )
>> +			{
>> +				msg = "Error registering stream output";
>> +				goto exit_2;
>> +			}
>>   			*blob = output;
>>   			*psize = out_len;
>>   			return 0;
>> -- 
>> 1.7.10.4
>>




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 03:21:05 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 03:21:05 GMT) (full text, mbox, link).


Message #60 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, Ian Campbell <ian.campbell@citrix.com>
Cc: xen-devel@lists.xen.org, ian.jackson@eu.citrix.com, wei.liu2@citrix.com, 767295@bugs.debian.org
Subject: Re: [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Thu, 20 Nov 2014 22:19:16 -0500
[Message part 1 (text/plain, inline)]
On 11/20/2014 10:13 PM, Gedalya wrote:
> On 11/20/2014 03:21 PM, Konrad Rzeszutek Wilk wrote:
>> On Thu, Nov 20, 2014 at 03:48:47PM +0000, Ian Campbell wrote:
>>> The libxc xc_dom_* infrastructure uses a very simple malloc memory 
>>> pool which
>>> is freed by xc_dom_release. However the various xc_try_*_decode 
>>> routines (other
>>> than the gzip one) just use plain malloc/realloc and therefore the 
>>> buffer ends
>>> up leaked.
>>>
>>> The memory pool currently supports mmap'd buffers as well as a directly
>>> allocated buffers, however the try decode routines make use of 
>>> realloc and do
>>> not fit well into this model. Introduce a concept of an external 
>>> memory block
>>> to the memory pool and provide an interface to register such memory.
>>>
>>> The mmap_ptr and mmap_len fields of the memblock tracking struct 
>>> lose their
>>> mmap_ prefix since they are now also used for external memory blocks.
>>>
>>> We are only seeing this now because the gzip decoder doesn't leak 
>>> and it's only
>>> relatively recently that kernels in the wild have switched to better
>>> compression.
>>>
>>> This is https://bugs.debian.org/767295
>>>
>>> Reported by: Gedalya <gedalya@gedalya.net>
>> Gedelya,
>>
>> Could you also test this patch to make sure it does fix the
>> reported issue please?
>
> So here's what happens now.
> 1. Starts up tiny
> 2. reboot: leak
> 3. reboot: freed (process larger, but the delta is all/mostly shared 
> pages)
> 4. reboot: leak
> 5. reboot: freed
> etc..
>
For the record, I applied it again the xen package currently in debian, 
4.4.1-3, see attached patch.
The only problem was that tools/libxc/include/xc_dom.h is in 
tools/libxc/xc_dom.h, otherwise it applied fine.

[0039-libxc-dont-leak-buffer-containing-the-uncompressed-pv-kernel (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 11:15:05 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <Ian.Campbell@citrix.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 11:15:05 GMT) (full text, mbox, link).


Message #65 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Gedalya <gedalya@gedalya.net>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: <wei.liu2@citrix.com>, <767295@bugs.debian.org>, <xen-devel@lists.xen.org>, <ian.jackson@eu.citrix.com>
Subject: Re: [Xen-devel] [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Fri, 21 Nov 2014 11:12:13 +0000
On Fri, 2014-11-21 at 11:03 +0000, Ian Campbell wrote:
> http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
> various dynamic thresholds for growing and shrinking the heap. My guess
> is that we are bouncing up and down over some threshold with every other
> reboot.

IOW I'm not overly concerned with this apparent bi-modality, so long as
the amount isn't increasing in the long term...

I think the original patch should go in.

Ian.





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 11:15:08 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <Ian.Campbell@citrix.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 11:15:08 GMT) (full text, mbox, link).


Message #70 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Gedalya <gedalya@gedalya.net>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, <xen-devel@lists.xen.org>, <ian.jackson@eu.citrix.com>, <wei.liu2@citrix.com>, <767295@bugs.debian.org>
Subject: Re: [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Fri, 21 Nov 2014 11:03:17 +0000
On Thu, 2014-11-20 at 22:13 -0500, Gedalya wrote:
> On 11/20/2014 03:21 PM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 20, 2014 at 03:48:47PM +0000, Ian Campbell wrote:
> >> The libxc xc_dom_* infrastructure uses a very simple malloc memory pool which
> >> is freed by xc_dom_release. However the various xc_try_*_decode routines (other
> >> than the gzip one) just use plain malloc/realloc and therefore the buffer ends
> >> up leaked.
> >>
> >> The memory pool currently supports mmap'd buffers as well as a directly
> >> allocated buffers, however the try decode routines make use of realloc and do
> >> not fit well into this model. Introduce a concept of an external memory block
> >> to the memory pool and provide an interface to register such memory.
> >>
> >> The mmap_ptr and mmap_len fields of the memblock tracking struct lose their
> >> mmap_ prefix since they are now also used for external memory blocks.
> >>
> >> We are only seeing this now because the gzip decoder doesn't leak and it's only
> >> relatively recently that kernels in the wild have switched to better
> >> compression.
> >>
> >> This is https://bugs.debian.org/767295
> >>
> >> Reported by: Gedalya <gedalya@gedalya.net>
> > Gedelya,
> >
> > Could you also test this patch to make sure it does fix the
> > reported issue please?
> 
> So here's what happens now.
> 1. Starts up tiny
> 2. reboot: leak
> 3. reboot: freed (process larger, but the delta is all/mostly shared pages)
> 4. reboot: leak
> 5. reboot: freed
> etc..

WTF, how very strange!

> root@xen:~/xen-pkgs# xl cr /etc/xen/auto/asterisk_deb80.cfg
> Parsing config from /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.0  0.0  95968   588 ?        SLsl 21:55   0:00 /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     128       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     240     240 rw---   [ anon ]
> 00007f14d4000000     132       8       8 rw---   [ anon ]
> 00007f14d4021000   65404       0       0 -----   [ anon ]
> << snip >>
> ---------------- ------- ------- -------
> total kB           95968    2728     596
> 
> --- reboot domu ---
> 
> root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.6  3.3 131652 20008 ?        SLsl 21:55   0:00 /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     144       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     288     288 rw---   [ anon ]
> 00000000009ee000   35676   16772   16772 rw---   [ anon ]

This is the (temporarily) leaked mapping, right?

> Tried valgrind, it doesn't look like it was able to see what was going on

Indeed. The values for total heap usage at exist and still reachable etc
also don't seem to account for the ~3M of mapping on each iteration.

I don't know how glibc's allocator works, but I suppose it isn't
impossible that it is retaining some mappings of free regions and
collecting them to free later somehow, which just happens to only
trigger every other reboot (e.g. perhaps it is based on some threshold
of free memory).

...investigates...

So, http://man7.org/linux/man-pages/man3/malloc.3.html talks about
special behaviour using mmap for allocations above MMAP_THRESHOLD (128K
by default), which we will be hitting here I think. That explains the
anon mapping.

http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
various dynamic thresholds for growing and shrinking the heap. My guess
is that we are bouncing up and down over some threshold with every other
reboot.

Ian.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 20:21:09 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 20:21:09 GMT) (full text, mbox, link).


Message #75 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, xen-devel@lists.xen.org, ian.jackson@eu.citrix.com, wei.liu2@citrix.com, 767295@bugs.debian.org
Subject: Re: [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Fri, 21 Nov 2014 15:19:43 -0500
On 11/21/2014 06:03 AM, Ian Campbell wrote:
>
>> So here's what happens now.
>> 1. Starts up tiny
>> 2. reboot: leak
>> 3. reboot: freed (process larger, but the delta is all/mostly shared pages)
>> 4. reboot: leak
>> 5. reboot: freed
>> etc..
> WTF, how very strange!
:-)

>
>> --- reboot domu ---
>>
>> root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
>> root     22981  0.6  3.3 131652 20008 ?        SLsl 21:55   0:00 /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
>> root@xen:~/xen-pkgs# pmap -x 22981
>> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
>> Address           Kbytes     RSS   Dirty Mode  Mapping
>> 0000000000400000     144     144       0 r-x-- xl
>> 0000000000623000       4       4       4 r---- xl
>> 0000000000624000       8       8       8 rw--- xl
>> 0000000000626000       4       4       4 rw---   [ anon ]
>> 00000000009a6000     288     288     288 rw---   [ anon ]
>> 00000000009ee000   35676   16772   16772 rw---   [ anon ]
> This is the (temporarily) leaked mapping, right?
Yea that's the one that popped in after the reboot..
About 16 MB.

>
>> Tried valgrind, it doesn't look like it was able to see what was going on
> Indeed. The values for total heap usage at exist and still reachable etc
> also don't seem to account for the ~3M of mapping on each iteration.
>
> I don't know how glibc's allocator works, but I suppose it isn't
> impossible that it is retaining some mappings of free regions and
> collecting them to free later somehow, which just happens to only
> trigger every other reboot (e.g. perhaps it is based on some threshold
> of free memory).
>
> ...investigates...
>
> So, http://man7.org/linux/man-pages/man3/malloc.3.html talks about
> special behaviour using mmap for allocations above MMAP_THRESHOLD (128K
> by default), which we will be hitting here I think. That explains the
> anon mapping.
>
> http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
> various dynamic thresholds for growing and shrinking the heap. My guess
> is that we are bouncing up and down over some threshold with every other
> reboot.
>
> Ian.
OK this is way over my head.. I don't have a full and precise 
understanding of all of the above, but let me try to comment nevertheless.
There are two issues here. The added mappings (as I understand, files, 
non-anon, shared) do happen only on reboot, but they're not a real 
memory leak issue because they are shared with other processes, so no 
matter ho many xl processes we have, it's only another 2.6 or so MB 
added to the total memory usage of the server, right?
On the other hand we have this anon area, 16 MB, that pops in on one 
reboot and gets freed on the next. That's the real issue, as I see it..!
And all that only starts happening after the last line of valgrind 
output. Valgrind only had output up to the first boot of the VM, none later.
For a fresh, non-rebooted domu, the xl process shows up as in top as 
~588 KB RES, 0 SHR, how can the latter be anyway?? I don't understand.





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 21 Nov 2014 20:27:04 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 21 Nov 2014 20:27:04 GMT) (full text, mbox, link).


Message #80 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <Ian.Campbell@citrix.com>, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: wei.liu2@citrix.com, 767295@bugs.debian.org, xen-devel@lists.xen.org, ian.jackson@eu.citrix.com
Subject: Re: [Xen-devel] [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Fri, 21 Nov 2014 15:25:00 -0500
On 11/21/2014 06:12 AM, Ian Campbell wrote:
> On Fri, 2014-11-21 at 11:03 +0000, Ian Campbell wrote:
>> http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
>> various dynamic thresholds for growing and shrinking the heap. My guess
>> is that we are bouncing up and down over some threshold with every other
>> reboot.
> IOW I'm not overly concerned with this apparent bi-modality, so long as
> the amount isn't increasing in the long term...
>
> I think the original patch should go in.
>
> Ian.
>
>
It's an improvement, but consider this:
Someone has a xen host running wheezy, 40 domu's, with 768MB for dom0, 
worked fine so far. Tries upgrading to jessie, and lo, each domu process 
takes up only 588 KB on dom0, great!
Then a new kernel package is released, all domu's get rebooted once. All 
host memory is now full. Dude might have had other plans for that 
memory... This is dead memory so I guess it can be swapped out, not 
easily a scenario where the server totally crashes, but it's a bit ugly, 
we're talking about memory usage leaping from 0.6 to 16 MB per domu.




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Mon, 24 Nov 2014 10:45:10 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <Ian.Campbell@citrix.com>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Mon, 24 Nov 2014 10:45:10 GMT) (full text, mbox, link).


Message #85 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <Ian.Campbell@citrix.com>
To: Gedalya <gedalya@gedalya.net>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, <wei.liu2@citrix.com>, <767295@bugs.debian.org>, <xen-devel@lists.xen.org>, <ian.jackson@eu.citrix.com>
Subject: Re: [Xen-devel] [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Mon, 24 Nov 2014 10:37:47 +0000
On Fri, 2014-11-21 at 15:25 -0500, Gedalya wrote:
> On 11/21/2014 06:12 AM, Ian Campbell wrote:
> > On Fri, 2014-11-21 at 11:03 +0000, Ian Campbell wrote:
> >> http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
> >> various dynamic thresholds for growing and shrinking the heap. My guess
> >> is that we are bouncing up and down over some threshold with every other
> >> reboot.
> > IOW I'm not overly concerned with this apparent bi-modality, so long as
> > the amount isn't increasing in the long term...
> >
> > I think the original patch should go in.
> >
> > Ian.
> >
> >
> It's an improvement, but consider this:
> Someone has a xen host running wheezy, 40 domu's, with 768MB for dom0, 
> worked fine so far. Tries upgrading to jessie, and lo, each domu process 
> takes up only 588 KB on dom0, great!
> Then a new kernel package is released, all domu's get rebooted once. All 
> host memory is now full. Dude might have had other plans for that 
> memory... This is dead memory so I guess it can be swapped out, not 
> easily a scenario where the server totally crashes, but it's a bit ugly, 
> we're talking about memory usage leaping from 0.6 to 16 MB per domu.

Unfortunately this is down to the behaviour of the libc and not
something which appears to be under application control. 

The following program demonstrates the same behaviour and is certainly
not leaking anything. Notice that at "Freed block at XXXXX. Everything
is now freed, end of day" there is still an anon mapping of that
address. Notice also that the "in use" figures are zero.

If this concerns you then you should probably take a look at mallopt(3)
and/or be talking to the libc folks about it. It's not an xl issue
AFAICT.

Ian.

#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include <malloc.h>

#define KB 196

int main(int argc, char **argv)
{
	void *p;
	char buf[1000];

	snprintf(buf, 1000, "pmap -x %d", getpid());

	printf("Start of day\n");
	system(buf);
	malloc_stats();

	printf("\n=========================\n\n");

	p = malloc(KB*0x1000);

	printf("allocated %dKB block at %p\n", KB, p);
	system(buf);
	malloc_stats();

	printf("\n=========================\n\n");

	free(p);

	printf("Freed block at %p\n", p);
	system(buf);
	malloc_stats();

	printf("\n=========================\n\n");

	p = malloc(KB*0x1000);

	printf("Allocated another %dKB block at %p\n", KB, p);
	system(buf);
	malloc_stats();

	printf("\n=========================\n\n");

	free(p);

	printf("Freed block at %p. Everything is now freed, end of day\n", p);
	system(buf);
	malloc_stats();

	printf("\n=========================\n\n");

	return 0;

}





Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Tue, 25 Nov 2014 07:18:04 GMT) (full text, mbox, link).


Acknowledgement sent to Gedalya <gedalya@gedalya.net>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Tue, 25 Nov 2014 07:18:05 GMT) (full text, mbox, link).


Message #90 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Gedalya <gedalya@gedalya.net>
To: Ian Campbell <Ian.Campbell@citrix.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>, wei.liu2@citrix.com, 767295@bugs.debian.org, xen-devel@lists.xen.org, ian.jackson@eu.citrix.com
Subject: Re: [Xen-devel] [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel
Date: Tue, 25 Nov 2014 02:14:57 -0500
On 11/24/2014 05:37 AM, Ian Campbell wrote:
> Unfortunately this is down to the behaviour of the libc and not
> something which appears to be under application control.
>
> The following program demonstrates the same behaviour and is certainly
> not leaking anything. Notice that at "Freed block at XXXXX. Everything
> is now freed, end of day" there is still an anon mapping of that
> address. Notice also that the "in use" figures are zero.
>
> If this concerns you then you should probably take a look at mallopt(3)
> and/or be talking to the libc folks about it. It's not an xl issue
> AFAICT.
Firstly, thank you very much for explaining this in such clear detail, 
above all this has been quite educational for me :-)
After reading the man page, it looks like glibc's behavior here is 
indeed by design. I'm unable to form, much less advocate an opinion 
about how libc should behave, any discussion about libc must be very broad.
Stepping away from the technical details, I still think that any future 
enhancement to make xl go out of its way to free this memory would 
definitely be nice. Right now we have memory that is allocated for a 
single, momentary use and it can stay allocated for the lifetime of a 
domu unnecessarily, taking the size of the xl process to another order 
of magnitude. Even if no longer technically a bug / memory leak, and the 
implied priority, this could still merit someone's attention.




Severity set to 'important' from 'normal' Request was from Bastian Blank <bastian.blank@credativ.de> to control@bugs.debian.org. (Wed, 26 Nov 2014 07:51:09 GMT) (full text, mbox, link).


Reply sent to Bastian Blank <waldi@debian.org>:
You have taken responsibility. (Thu, 27 Nov 2014 21:24:43 GMT) (full text, mbox, link).


Notification sent to Gedalya <gedalya@gedalya.net>:
Bug acknowledged by developer. (Thu, 27 Nov 2014 21:24:43 GMT) (full text, mbox, link).


Message #97 received at 767295-close@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>
To: 767295-close@bugs.debian.org
Subject: Bug#767295: fixed in xen 4.4.1-4
Date: Thu, 27 Nov 2014 21:24:27 +0000
Source: xen
Source-Version: 4.4.1-4

We believe that the bug you reported is fixed in the latest version of
xen, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 767295@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bastian Blank <waldi@debian.org> (supplier of updated xen package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Thu, 27 Nov 2014 20:17:36 +0100
Source: xen
Binary: libxen-4.4 libxenstore3.0 libxen-dev xenstore-utils xen-utils-common xen-utils-4.4 xen-hypervisor-4.4-amd64 xen-system-amd64 xen-hypervisor-4.4-arm64 xen-system-arm64 xen-hypervisor-4.4-armhf xen-system-armhf
Architecture: source all amd64
Version: 4.4.1-4
Distribution: unstable
Urgency: medium
Maintainer: Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>
Changed-By: Bastian Blank <waldi@debian.org>
Description:
 libxen-4.4 - Public libs for Xen
 libxen-dev - Public headers and libs for Xen
 libxenstore3.0 - Xenstore communications library for Xen
 xen-hypervisor-4.4-amd64 - Xen Hypervisor on AMD64
 xen-hypervisor-4.4-arm64 - Xen Hypervisor on ARM64
 xen-hypervisor-4.4-armhf - Xen Hypervisor on ARMHF
 xen-system-amd64 - Xen System on AMD64 (meta-package)
 xen-system-arm64 - Xen System on ARM64 (meta-package)
 xen-system-armhf - Xen System on ARMHF (meta-package)
 xen-utils-4.4 - XEN administrative tools
 xen-utils-common - Xen administrative tools - common files
 xenstore-utils - Xenstore command line utilities for Xen
Closes: 759384 767295 769543 770460
Changes:
 xen (4.4.1-4) unstable; urgency=medium
 .
   [ Bastian Blank ]
   * Make operations pre-emptible.
     CVE-2014-5146, CVE-2014-5149
   * Don't allow page table updates from non-PV page tables.
     CVE-2014-8594
   * Enforce privilege level while loading code segment.
     CVE-2014-8595
   * Fix reference counter leak.
     CVE-2014-9030
   * Use linux 3.16.0-4 stuff.
   * Fix memory leak in xl. (closes: #767295)
 .
   [ Ian Campbell ]
   * Add licensing for tools/python/logging to debian/copyright.
     (Closes: #759384)
   * Correctly include xen-init-name in xen-utils-common. (Closes: #769543)
   * xen-utils recommends grub-xen-host package (Closes: #770460)
Checksums-Sha1:
 23075ae54e1a8ea49864c871be036f7c9fc88e12 2600 xen_4.4.1-4.dsc
 598a7e9f0c4279bad0765e5964f13f1c266a30aa 59772 xen_4.4.1-4.debian.tar.xz
 7fdd633935a4ad26294929e345f74c69f63a2336 120666 xen-utils-common_4.4.1-4_all.deb
 902851a1eec687232b068effc4770db8cf95e586 1671010 xen-hypervisor-4.4-amd64_4.4.1-4_amd64.deb
 0f5016527f38126a6687f755c9fcda904148ffc7 19586 xen-system-amd64_4.4.1-4_amd64.deb
 add47ed1b2f26940e181b347d79e6424723f63c0 30458 libxenstore3.0_4.4.1-4_amd64.deb
 a4a19ae136ffddd6b1cf0124102abe1e3d287650 295062 libxen-4.4_4.4.1-4_amd64.deb
 bb97828b8e38b608f14fa0565cb1d570fe1f190e 476442 libxen-dev_4.4.1-4_amd64.deb
 b107f1d59c8d4b76856a21840c9d8f4867b37537 393300 xen-utils-4.4_4.4.1-4_amd64.deb
 a91952017a6a419e53e62c39e91914aa08da0abd 26120 xenstore-utils_4.4.1-4_amd64.deb
Checksums-Sha256:
 28da1bed741e73d0646e90eed19fe189599d1cf1e293d6e5c2c5b42947514596 2600 xen_4.4.1-4.dsc
 8b81cde7cd1e0066ee0d2b4225dbc997e42991f6242b4d37754faa04d14d3333 59772 xen_4.4.1-4.debian.tar.xz
 e47df6cc38c8c7368478022b0c562b41f5a7e12cb90511a8069b7f4d1f403acf 120666 xen-utils-common_4.4.1-4_all.deb
 9b34a6bc3195559ff27c4d8430cd1961cd0717a50070f210ccfe93f2b3f28e43 1671010 xen-hypervisor-4.4-amd64_4.4.1-4_amd64.deb
 9e27b24f7858f1c4e2d8ebd4b5f6026cfa1edfec64523d9d58b806398d7d857e 19586 xen-system-amd64_4.4.1-4_amd64.deb
 24735aaf1754f5adfe24d282e913ece79c12cdbfbb38a77bbc4417da5597d133 30458 libxenstore3.0_4.4.1-4_amd64.deb
 c5ab637175197592c8f46490ec6629850e818e597193e1867dbaeb28fdb7e858 295062 libxen-4.4_4.4.1-4_amd64.deb
 26463a88f4e2543d7980777cbb50f52dd599c082111fb976594f47b5dd8aa065 476442 libxen-dev_4.4.1-4_amd64.deb
 74e1c08489418db06d736df02cb24d71bbe254627b0d7a91a42e121b778ac250 393300 xen-utils-4.4_4.4.1-4_amd64.deb
 f4228ff85519ddf9b014349c12e24e950b9dc9a81fda8f57432317e9588f7064 26120 xenstore-utils_4.4.1-4_amd64.deb
Files:
 509ecc20c9d5fb8eb14fabfcf2d9ac23 2600 kernel optional xen_4.4.1-4.dsc
 aa51a2398499174dc462a7bfd639dedf 59772 kernel optional xen_4.4.1-4.debian.tar.xz
 882642769cd3e6763cddf07980d4cecf 120666 kernel optional xen-utils-common_4.4.1-4_all.deb
 ce3749fbc82c2537cad6fe440ed795cc 1671010 kernel optional xen-hypervisor-4.4-amd64_4.4.1-4_amd64.deb
 774d0ea604b46ca6b9784d6639d7ef6f 19586 kernel optional xen-system-amd64_4.4.1-4_amd64.deb
 f0d445b9bd059f3cfbc3d9325eb191c5 30458 libs optional libxenstore3.0_4.4.1-4_amd64.deb
 f5a1a954eb030711ba25c82b78b3000f 295062 libs optional libxen-4.4_4.4.1-4_amd64.deb
 c5ad87daae3a9c894c21d2046e07cc69 476442 libdevel optional libxen-dev_4.4.1-4_amd64.deb
 249c3d7b27ed2601b429b3252d4dc564 393300 kernel optional xen-utils-4.4_4.4.1-4_amd64.deb
 f9c006b04fb36b5ff1783b8af56b4246 26120 admin optional xenstore-utils_4.4.1-4_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUd4TMAAoJEG2TiIWKaf5RAXsH+wY8YlPwdnGwZKl+UBP6j6uG
w53j6d6SEv3lAOIGYsQwwtByBpFJzE/Gt7AzsSXrS7Lv9pcHnd7+wgIROeWRDJ9a
iT2ui/z9e2G3eYP1pj9CgyQzfXfl/WbV01V7YivzD3QHDCHSb03PX7IsOFtBsomE
+Ja6BbPSxGYG+kqG4og3bfMrllFf6tVTc3Nw4LUOgEkct1uhVK619xvzHJNPk4Yk
4Yj1gopSQjGzTpkA3htxa6IE3/IjvvQu9QBxcGwehSK4dEjJqTGgkw5ZwWEEDiSK
EPSg3MdZ5MK7CFuWe72oN4P24DNThC4ySbL5QuiUtBS6xBPicJL4dGy1Pe/ED6s=
=zRox
-----END PGP SIGNATURE-----




Information forwarded to debian-bugs-dist@lists.debian.org, Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>:
Bug#767295; Package xen-utils-4.4. (Fri, 28 Nov 2014 13:27:05 GMT) (full text, mbox, link).


Acknowledgement sent to Ian Campbell <ijc@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>. (Fri, 28 Nov 2014 13:27:05 GMT) (full text, mbox, link).


Message #102 received at 767295@bugs.debian.org (full text, mbox, reply):

From: Ian Campbell <ijc@debian.org>
To: 767295@bugs.debian.org
Subject: Re: Bug#767295: xl: apparent memory leak
Date: Fri, 28 Nov 2014 13:24:26 +0000
reopen 767295
thanks

The 4.4.1-4 release only included one related fix there are actually two
more:

        commit 379b351889a8f02abe30a06e2ce9ba8b381b91ab
        Author: Ian Campbell <ian.campbell@citrix.com>
        Date:   Thu Nov 6 13:00:31 2014 +0000
        
            tools: libxl: do not leak diskpath during local disk attach
            
            libxl__device_disk_local_initiate_attach is assigning dls->diskpath with a
            strdup of the device path. This is then passed to the callback, e.g.
            parse_bootloader_result but bootloader_cleanup will not free it.
            
            Since the callback is within the scope of the (e)gc and therefore doesn't need
            to be malloc'd, a gc'd alloc will do. All other assignments to this field use
            the gc.
            
            https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=767295
            
            Reported-by: Gedalya <gedalya@gedalya.net>
            Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
            Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
            Acked-by: Wei Liu <wei.liu2@citrix.com>
        
        commit 8f4023dd7d77de7b2c1af77e86637202a33f948a
        Author: Ian Campbell <ian.campbell@citrix.com>
        Date:   Thu Nov 20 15:48:47 2014 +0000
        
            libxc: don't leak buffer containing the uncompressed PV kernel
            
            The libxc xc_dom_* infrastructure uses a very simple malloc memory pool which
            is freed by xc_dom_release. However the various xc_try_*_decode routines (other
            than the gzip one) just use plain malloc/realloc and therefore the buffer ends
            up leaked.
            
            The memory pool currently supports mmap'd buffers as well as a directly
            allocated buffers, however the try decode routines make use of realloc and do
            not fit well into this model. Introduce a concept of an external memory block
            to the memory pool and provide an interface to register such memory.
            
            The mmap_ptr and mmap_len fields of the memblock tracking struct lose their
            mmap_ prefix since they are now also used for external memory blocks.
            
            We are only seeing this now because the gzip decoder doesn't leak and it's only
            relatively recently that kernels in the wild have switched to better
            compression.
            
            This is https://bugs.debian.org/767295
            
            Reported by: Gedalya <gedalya@gedalya.net>
            Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
            Reviewed-by: Wei Liu <wei.liu2@citrix.com>

Which the actual leaks discovered here (of which the second is the far
larger).





Bug reopened Request was from Ian Campbell <ijc@debian.org> to control@bugs.debian.org. (Fri, 28 Nov 2014 13:27:11 GMT) (full text, mbox, link).


No longer marked as fixed in versions xen/4.4.1-4. Request was from Ian Campbell <ijc@debian.org> to control@bugs.debian.org. (Fri, 28 Nov 2014 13:27:12 GMT) (full text, mbox, link).


Reply sent to Bastian Blank <waldi@debian.org>:
You have taken responsibility. (Sun, 30 Nov 2014 19:51:14 GMT) (full text, mbox, link).


Notification sent to Gedalya <gedalya@gedalya.net>:
Bug acknowledged by developer. (Sun, 30 Nov 2014 19:51:14 GMT) (full text, mbox, link).


Message #111 received at 767295-close@bugs.debian.org (full text, mbox, reply):

From: Bastian Blank <waldi@debian.org>
To: 767295-close@bugs.debian.org
Subject: Bug#767295: fixed in xen 4.4.1-5
Date: Sun, 30 Nov 2014 19:49:21 +0000
Source: xen
Source-Version: 4.4.1-5

We believe that the bug you reported is fixed in the latest version of
xen, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 767295@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Bastian Blank <waldi@debian.org> (supplier of updated xen package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Sun, 30 Nov 2014 20:13:32 +0100
Source: xen
Binary: libxen-4.4 libxenstore3.0 libxen-dev xenstore-utils xen-utils-common xen-utils-4.4 xen-hypervisor-4.4-amd64 xen-system-amd64 xen-hypervisor-4.4-arm64 xen-system-arm64 xen-hypervisor-4.4-armhf xen-system-armhf
Architecture: source all amd64
Version: 4.4.1-5
Distribution: unstable
Urgency: medium
Maintainer: Debian Xen Team <pkg-xen-devel@lists.alioth.debian.org>
Changed-By: Bastian Blank <waldi@debian.org>
Description:
 libxen-4.4 - Public libs for Xen
 libxen-dev - Public headers and libs for Xen
 libxenstore3.0 - Xenstore communications library for Xen
 xen-hypervisor-4.4-amd64 - Xen Hypervisor on AMD64
 xen-hypervisor-4.4-arm64 - Xen Hypervisor on ARM64
 xen-hypervisor-4.4-armhf - Xen Hypervisor on ARMHF
 xen-system-amd64 - Xen System on AMD64 (meta-package)
 xen-system-arm64 - Xen System on ARM64 (meta-package)
 xen-system-armhf - Xen System on ARMHF (meta-package)
 xen-utils-4.4 - XEN administrative tools
 xen-utils-common - Xen administrative tools - common files
 xenstore-utils - Xenstore command line utilities for Xen
Closes: 767295
Changes:
 xen (4.4.1-5) unstable; urgency=medium
 .
   * Fix excessive checks of hypercall arguments.
     CVE-2014-8866
   * Fix boundary checks of emulated MMIO access.
     CVE-2014-8867
   * Fix additional memory leaks in xl. (closes: #767295)
Checksums-Sha1:
 224033f6dca82c3717349ad2b882d4d13ff0faa3 2598 xen_4.4.1-5.dsc
 5cb42d61c090e17c8e1d07abb4eec370402e7abf 64792 xen_4.4.1-5.debian.tar.xz
 310d9cec42512d3fc4fd8f68bb6e17adba3c7154 120772 xen-utils-common_4.4.1-5_all.deb
 d50cd6debff16965ff9914cfee862f5eb1751c5a 1671186 xen-hypervisor-4.4-amd64_4.4.1-5_amd64.deb
 9134dcbd64b9204a33ea09201221f6399a61d236 19660 xen-system-amd64_4.4.1-5_amd64.deb
 c84c686aaa82a5f142db0fabb94b55cb37157a54 30572 libxenstore3.0_4.4.1-5_amd64.deb
 a5850c6bb093e915664086a99a6555985a785f40 295086 libxen-4.4_4.4.1-5_amd64.deb
 37e9aa25ad1f8f1398bf1fcc9210a063d1a4c93f 476346 libxen-dev_4.4.1-5_amd64.deb
 3d215f6a52ef0db6afc4bfb6454ff41194f3cc4a 393280 xen-utils-4.4_4.4.1-5_amd64.deb
 f8661ece2b83acfd1cac6d5a565022319b05e983 26198 xenstore-utils_4.4.1-5_amd64.deb
Checksums-Sha256:
 fc66cdd908f3bb12e02327bd29a887b06d7b21ab38edacd3c9e712494c56996c 2598 xen_4.4.1-5.dsc
 5f2d2137fa68ff63545a5e8a2e45d836fea481cb0f7a854bcff5f882cbeb881f 64792 xen_4.4.1-5.debian.tar.xz
 1bfb6b49e713f2807fddf7dbb0c3c8ee556a963fc940bf25e22f75263aa4ab08 120772 xen-utils-common_4.4.1-5_all.deb
 959dcce1cfc70d823a7f84adb1f1d2ccbbed042f5d2160d5e910bd7152008c54 1671186 xen-hypervisor-4.4-amd64_4.4.1-5_amd64.deb
 5e414a9f08df0e701e73576020aedf3029490a20407f83c27a51432cf9b9b1bf 19660 xen-system-amd64_4.4.1-5_amd64.deb
 93f593202d5d1b3e4ad059c5930b9cd94e4f4ff7724081d60a6f6c5277422cff 30572 libxenstore3.0_4.4.1-5_amd64.deb
 b9340b5e6b9378b557402a2201f8e2068a78bcd948cfe2991bccc312d064bde1 295086 libxen-4.4_4.4.1-5_amd64.deb
 c17513d46a9f8c32f6bfd0f83ff2e243142e93b10bf3a4d4d0706c284e3f48f0 476346 libxen-dev_4.4.1-5_amd64.deb
 77bead8f2f6414068ada22218f306cb5b2d974dc82744043330bbfc0c6e696b2 393280 xen-utils-4.4_4.4.1-5_amd64.deb
 098abd59aa380e18ec8ff1c28e8944cf1b2e340d3331986395390efad0639053 26198 xenstore-utils_4.4.1-5_amd64.deb
Files:
 f1a2af9f9651dbeabff5d8fc062b6412 2598 kernel optional xen_4.4.1-5.dsc
 34f0a6c8c8d8cff1903789ce68f30f51 64792 kernel optional xen_4.4.1-5.debian.tar.xz
 0e8be9db756ed5614109fc1a23063506 120772 kernel optional xen-utils-common_4.4.1-5_all.deb
 1804a09b1aec022711b0dadc1d95cacb 1671186 kernel optional xen-hypervisor-4.4-amd64_4.4.1-5_amd64.deb
 a73268649f67307496ff5257ea4027f9 19660 kernel optional xen-system-amd64_4.4.1-5_amd64.deb
 39a901402a540ddb1a02050abd46d90e 30572 libs optional libxenstore3.0_4.4.1-5_amd64.deb
 3e5672de5fe4240e210f5e49310f7b8f 295086 libs optional libxen-4.4_4.4.1-5_amd64.deb
 7a22ff82b1c665afc7a9bcced6babb2b 476346 libdevel optional libxen-dev_4.4.1-5_amd64.deb
 5c15d6d7ec7d2d0fa9158465cf9e5b22 393280 kernel optional xen-utils-4.4_4.4.1-5_amd64.deb
 4e80677ec103a2fccfbeca1c3dda847a 26198 admin optional xenstore-utils_4.4.1-5_amd64.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUe3GMAAoJEG2TiIWKaf5RjOsH/0NTgWrDPBhffzY0/0+pjphr
rjpsw/W1merwZgO5mx+BUxTLwmxSbVOkqBxU46QyxNlw/QoErolezKO06zJ/jepP
A4+2z8un6wyvMRKwWznsNVwkGYozwA14rpPH74O9h4aswrjZB4p/BeS2jkJKRH8b
jLoy0TdpLpSZdeRHVcEQKUakZYmST49MAwB8bvH/UKgd/PGwezfGEXhrrayvNSoM
b0hdChatSTufnjeX7qtOCR89Rt+vFwcO91upvjwih8a9STGsboGGhdLw3Jc9S6tj
8kwEY45k3J2+hnsFTDMqD1bV1O4Wdee+KqitbUtd67+Wx3azkl12tSoatdFWAdc=
=lBub
-----END PGP SIGNATURE-----




Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Sat, 03 Jan 2015 07:35:31 GMT) (full text, mbox, link).


Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Sun Jul 2 04:12:55 2023; Machine Name: bembo

Debian Bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.