Debian Bug report logs - #757037
liblzo2-2: Upgrading liblzo2-2 to 2.08 crashes openvpn

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Thomas Herrmann <tobox@gmx.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: liblzo2-2: Upgrading liblzo2-2 to 2.08 crashes openvpn

Date: Mon, 04 Aug 2014 21:02:13 +0200

Package: liblzo2-2
Version: 2.08-1
Severity: important

Dear Maintainer,

after upgrading to 2.08-1, openvpn (version 2.3.2-9) fails with the following error message:

Cannot initialize LZO compression library
Exiting due to fatal error

Downgrading to 2.06 fixes the problem for me.



-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: armel (armv5tel)

Kernel: Linux 3.14-2-kirkwood
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages liblzo2-2 depends on:
ii  libc6              2.19-7
ii  multiarch-support  2.19-7

liblzo2-2 recommends no packages.

liblzo2-2 suggests no packages.

-- no debconf information

Message #10 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Peter Eisentraut <petere@debian.org>

To: Thomas Herrmann <tobox@gmx.de>, 757037@bugs.debian.org

Subject: Re: Bug#757037: liblzo2-2: Upgrading liblzo2-2 to 2.08 crashes openvpn

Date: Sun, 17 Aug 2014 22:57:57 -0400

On Mon, 2014-08-04 at 21:02 +0200, Thomas Herrmann wrote:
> after upgrading to 2.08-1, openvpn (version 2.3.2-9) fails with the following error message:
> 
> Cannot initialize LZO compression library
> Exiting due to fatal error
> 
> Downgrading to 2.06 fixes the problem for me.

Do you have a reproducible test case for that?

(It doesn't happen for me if I just run /usr/sbin/openvpn without
anything else.)

Message #15 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Thomas Herrmann <tobox@gmx.de>

To: 757037@bugs.debian.org

Subject: Re: Bug#757037: liblzo2-2: Upgrading liblzo2-2 to 2.08 crashes openvpn

Date: Mon, 18 Aug 2014 13:26:43 +0200

On 08/18/2014 04:57 AM, Peter Eisentraut wrote:
> Do you have a reproducible test case for that?

Yes, just created one:

root@qnap:/etc/openvpn# cat test.conf
local 192.168.242.215
port 1194
proto udp
dev tun
ca ca.crt
cert qnap.crt
key qnap.key
dh dh1024.pem
server 192.168.54.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.242.0 255.255.255.0"
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3

root@qnap:/etc/openvpn# openvpn --cd /etc/openvpn --config /etc/openvpn/test.conf      Mon Aug 18 13:15:35 2014 OpenVPN 2.3.2 arm-unknown-linux-gnueabi [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Mar 18 2014
Mon Aug 18 13:15:35 2014 Diffie-Hellman initialized with 1024 bit key
Mon Aug 18 13:15:35 2014 Socket Buffers: R=[163840->131072] S=[163840->131072]
Mon Aug 18 13:15:35 2014 ROUTE_GATEWAY 192.168.242.1/255.255.255.0 IFACE=eth0 HWADDR=00:08:9b:c6:12:56
Mon Aug 18 13:15:35 2014 TUN/TAP device tun0 opened
Mon Aug 18 13:15:35 2014 TUN/TAP TX queue length set to 100
Mon Aug 18 13:15:35 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Mon Aug 18 13:15:35 2014 /sbin/ip link set dev tun0 up mtu 1500
Mon Aug 18 13:15:35 2014 /sbin/ip addr add dev tun0 local 192.168.54.1 peer 192.168.54.2
Mon Aug 18 13:15:35 2014 /sbin/ip route add 192.168.54.0/24 via 192.168.54.2
Mon Aug 18 13:15:35 2014 GID set to nogroup
Mon Aug 18 13:15:35 2014 UID set to nobody
Mon Aug 18 13:15:35 2014 UDPv4 link local (bound): [AF_INET]192.168.242.215:1194
Mon Aug 18 13:15:35 2014 UDPv4 link remote: [undef]
Mon Aug 18 13:15:35 2014 MULTI: multi_init called, r=256 v=256
Mon Aug 18 13:15:35 2014 IFCONFIG POOL: base=192.168.54.4 size=62, ipv6=0
Mon Aug 18 13:15:35 2014 IFCONFIG POOL LIST
Mon Aug 18 13:15:35 2014 Initialization Sequence Completed
Mon Aug 18 13:15:35 2014 109.91.169.175:38150 Cannot initialize LZO compression library
Mon Aug 18 13:15:35 2014 109.91.169.175:38150 Exiting due to fatal error


Downgrade to 2.06:

root@qnap:/etc/openvpn# dpkg -i ~/liblzo2-2_2.06-1+deb7u1_armel.deb dpkg: warning: downgrading liblzo2-2:armel from 2.08-1 to 2.06-1+deb7u1
(Reading database ... 99277 files and directories currently installed.)
Preparing to unpack .../liblzo2-2_2.06-1+deb7u1_armel.deb ...
Unpacking liblzo2-2:armel (2.06-1+deb7u1) over (2.08-1) ...
Setting up liblzo2-2:armel (2.06-1+deb7u1) ...
Processing triggers for libc-bin (2.19-7) ...


root@qnap:/etc/openvpn# openvpn --cd /etc/openvpn --config /etc/openvpn/test.conf
Mon Aug 18 13:16:26 2014 OpenVPN 2.3.2 arm-unknown-linux-gnueabi [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Mar 18 2014
Mon Aug 18 13:16:26 2014 Diffie-Hellman initialized with 1024 bit key
Mon Aug 18 13:16:26 2014 Socket Buffers: R=[163840->131072] S=[163840->131072]
Mon Aug 18 13:16:26 2014 ROUTE_GATEWAY 192.168.242.1/255.255.255.0 IFACE=eth0 HWADDR=00:08:9b:c6:12:56
Mon Aug 18 13:16:26 2014 TUN/TAP device tun0 opened
Mon Aug 18 13:16:26 2014 TUN/TAP TX queue length set to 100
Mon Aug 18 13:16:26 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Mon Aug 18 13:16:26 2014 /sbin/ip link set dev tun0 up mtu 1500
Mon Aug 18 13:16:26 2014 /sbin/ip addr add dev tun0 local 192.168.54.1 peer 192.168.54.2
Mon Aug 18 13:16:26 2014 /sbin/ip route add 192.168.54.0/24 via 192.168.54.2
Mon Aug 18 13:16:26 2014 GID set to nogroup
Mon Aug 18 13:16:26 2014 UID set to nobody
Mon Aug 18 13:16:26 2014 UDPv4 link local (bound): [AF_INET]192.168.242.215:1194
Mon Aug 18 13:16:26 2014 UDPv4 link remote: [undef]
Mon Aug 18 13:16:26 2014 MULTI: multi_init called, r=256 v=256
Mon Aug 18 13:16:26 2014 IFCONFIG POOL: base=192.168.54.4 size=62, ipv6=0
Mon Aug 18 13:16:26 2014 IFCONFIG POOL LIST
Mon Aug 18 13:16:26 2014 Initialization Sequence Completed
Mon Aug 18 13:16:31 2014 109.91.169.175:59973 TLS: Initial packet from [AF_INET]109.91.169.175:59973, sid=265834d9 280eeafc
Mon Aug 18 13:16:31 2014 109.91.169.175:59973 VERIFY OK: depth=1, C=DE, ST=XX, L=XX, O=Home, OU=None, CN=Home CA, emailAddress=XX
Mon Aug 18 13:16:31 2014 109.91.169.175:59973 VERIFY OK: depth=0, C=DE, ST=XX, L=XX, O=Home, OU=None, CN=XX, emailAddress=XX


I have just realized that the problem only occurrs, when a client connects. So to reproduce the problem a fully working openvpn setup (including keys an certificates) is necessary, including a client.

Regards
Thomas

Message #22 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Marc Kleine-Budde <mkl@blackshift.org>

To: 757037@bugs.debian.org

Subject: new liblzo breaking things

Date: Sun, 07 Sep 2014 13:58:40 +0200

[Message part 1 (text/plain, inline)]

The vpn program tinc has the same problem (See:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=760646) on armel.

You need an armel (x86_64 doesn't have the problem), but no special
setup to reproduce the problem:

    touch /tmp/tinc.conf
    tincd -D -c /tmp

It will fail with:

    Error initializing LZO compressor!

In case liblzo gets initialized correctly, you'll see this error message:

    Couldn't write pid file /var/run/tinc.pid: Permission denied

The the following code snipped from the tinc's "src/tincd.c" shows that
the lzo initialization fails:

        if(lzo_init() != LZO_E_OK) {
                logger(LOG_ERR, "Error initializing LZO compressor!");
                return 1;
        }

BTW: even "lzop" fails with the new liblzo library:

$ lzop
                          Lempel-Ziv-Oberhumer Packer
                           Copyright (C) 1996 - 2010
lzop v1.03         Markus Franz Xaver Johannes Oberhumer          Nov
1st 2010

lzo_init() failed - check your LZO installation !
library version conflict (2050, 2080) - check your LZO installation !

regards,
Marc

[signature.asc (application/pgp-signature, attachment)]

Message #27 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: 757037@bugs.debian.org

Subject: Re: Bug#757037: new liblzo breaking things

Date: Mon, 3 Nov 2014 00:19:07 +0000

On Sun, 07 Sep 2014 at 13:58:40 +0200, Marc Kleine-Budde wrote:
> BTW: even "lzop" fails with the new liblzo library:
> 
> $ lzop
>                           Lempel-Ziv-Oberhumer Packer
>                            Copyright (C) 1996 - 2010
> lzop v1.03         Markus Franz Xaver Johannes Oberhumer          Nov
> 1st 2010
> 
> lzo_init() failed - check your LZO installation !

This bit me on a recently-upgraded jessie system where I'm somewhat
reliant on openvpn, so after working around it by downgrading liblzo2-2
to the wheezy-security version, I looked further into it.

You don't actually need to build packages at all: the same failure occurs
in a freshly compiled lzo2 tree (just ./configure && make) using
"LD_LIBRARY_PATH=src/.libs gdb lzop", "run".

When rebuilt with CFLAGS="-O0 -g" it works. So, some optimization is
to blame for this.

When rebuilt with CFLAGS="-Og -g" it fails, and this time it's possible
to see *where* it fails, because the assignment to r isn't optimized out:

(gdb) break _lzo_config_check
...
_lzo_config_check () at src/lzo_init.c:100
(gdb) watch r
(gdb) s
...
_lzo_config_check () at src/lzo_init.c:117
117	    r &= UA_GET_NE16(p) == 0;
(gdb) 
lzo_memops_get_ne16 (ss=0xbefff1f1) at src/lzo_func.h:387
387	    LZO_MEMOPS_COPY2(&v, ss);
(gdb) 
Watchpoint 4: r

Old value = 1
New value = 0

My next try was CPPFLAGS="-DLZO_CFG_NO_UNALIGNED" to stop lzo doing
"clever" pointer dereferences in LZO_MEMOPS_COPY2, and sure enough,
that works. So we have a workaround: define that macro on at least armel.
It might slow lzo2 down, but working slowly is better than failing rapidly.
(My next idea was -fno-strict-aliasing, but unfortunately that didn't work.)

This optimization stuff made me suspicious about new compiler optimizations
since wheezy's (working) lzo2 was compiled. It's getting late now, but the
next thing I try is going to be rebuilding wheezy's lzo2 on a jessie system
to see whether it has the same failure conditions.

I'd really like to see this fixed in jessie; I'm tempted to say it
should be RC. I'll try to produce and test a complete patch, either
using CPPFLAGS="-DLZO_CFG_NO_UNALIGNED" on armel or preferably turning off
some specific optimization.

Hope this helps,
    S

Message #32 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Marc Kleine-Budde <mkl@blackshift.org>

To: 757037@bugs.debian.org

Subject: Bug#757037: liblzop alignment problem

Date: Sun, 09 Nov 2014 21:26:37 +0100

[Message part 1 (text/plain, inline)]

Hey Simon,

indeed it's an alignment problem. On ARM <= v5 you cannot access data
with unaligned pointers. However you can ask the kernel to fixup that
problem, which has a veeery high runtime overhead. This options is
switched off by default on debian.

You can switch it on via:

    echo 3 | sudo tee /proc/cpu/alignment

and voila, lzop works. And gives a nice warning in dmesg:

> [3671050.938449] Alignment trap: lzop (4437) PC=0xb6f371a0 Instr=0xe1d030b0 Address=0xbee013a9 FSR 0x001
> [3671050.951027] Alignment trap: lzop (4437) PC=0xb6f371bc Instr=0xe1d030b0 Address=0xbee013a9 FSR 0x001
> [3671050.962950] Alignment trap: lzop (4437) PC=0xb6f371d8 Instr=0xe1d020b0 Address=0xbee013a9 FSR 0x001
> [3671050.974746] Alignment trap: lzop (4437) PC=0xb6f371f8 Instr=0xe5902000 Address=0xbee013a9 FSR 0x001
> [3671050.987114] Alignment trap: lzop (4437) PC=0xb6f37214 Instr=0xe5902000 Address=0xbee013a9 FSR 0x001
> [3671050.999138] Alignment trap: lzop (4437) PC=0xb6f37234 Instr=0xe5900000 Address=0xbee013a9 FSR 0x001

And you can compare the output of

    cat /proc/cpu/alignment

before running lzop and after. Here the "Word" counter increases by 3.

As you see address in the dmesg output ends with "9" which is an
unaligned address. So from my point of view switching unaligned access
off in libzop with -DLZO_CFG_NO_UNALIGNED should be done on at least
ARMel. Are there other alignment sensitive platforms in debian?

Marc

[signature.asc (application/pgp-signature, attachment)]

Message #37 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Marc Kleine-Budde <mkl@blackshift.org>, 757037@bugs.debian.org

Subject: Re: Bug#757037: liblzop alignment problem

Date: Wed, 12 Nov 2014 00:20:56 +0000

[Message part 1 (text/plain, inline)]

Control: severity 757037 grave
Control: tags 757037 + patch

I'm bumping this bug up to grave severity, because it seems it makes
this liblzo2 version entirely useless on a fairly common armel CPU (my armel
has a Marvell Kirkwood armv5tel, as used in the SheevaPlug and its
derivatives) in the default Debian configuration. It also makes liblzo2
FTBFS on such machines, because the tests fail.

On Sun, 09 Nov 2014 at 21:26:37 +0100, Marc Kleine-Budde wrote:
> So from my point of view switching unaligned access
> off in libzop with -DLZO_CFG_NO_UNALIGNED should be done on at least
> ARMel. Are there other alignment sensitive platforms in debian?

(Please Cc me, I am not subscribed to this bug. Non-maintainer contributors
to Debian bugs usually aren't.)

I think alignment-sensitivity is the common case, and quietly accepting
unaligned accesses (and giving the correct answer!) as is done on x86 is the
exceptional case...

According to http://www.ibm.com/developerworks/library/pa-dalign/
the mips family are alignment-sensitive, and so is powerpc sometimes,
although it might do automatic fixup.

According to https://bugs.launchpad.net/ubuntu/+source/zmqpp/+bug/1256886
Ubuntu armhf is alignment-sensitive (presumably Debian armhf is the same?)

I've also seen references to sparc not liking unaligned accesses.

s390 can do unaligned accesses fine according to
http://lxr.free-electrons.com/source/include/asm-s390/unaligned.h?v=2.2.26,
but realistically, who's going to test that? Not me :-)

I think the safest thing would be to use -DLZO_CFG_NO_UNALIGNED
on all Debian architectures except i386, amd64, and maybe the powerpc
family if someone can test those. I attach a patch that restricts
use of unaligned accesses to i386 and amd64 CPUs (any kernel);
it's easy to extend to whitelist other CPUs where unaligned
accesses are OK.

liblzo2 does actually try to avoid using unsafe unaligned accesses
by using a structure

    typedef struct lzo_memops_TU2_struct { unsigned char a[2]; } lzo_memops_TU2;

and only doing a type-punning assignment similar to[1]

    *(lzo_memops_TU2 *) (void *) dest = *(const lzo_memops_TU2 *) (void *) src;

if that structure has required alignment 1... but it seems (recent?) gcc
might be optimizing that into a single 2-byte operation which requires
alignment. This might be a gcc bug, but then, I suspect this might be
a violation of strict aliasing (hence undefined behaviour) if you get all
language-lawyer about it. Pessimistically assuming that we have to access
byte-by-byte on miscellaneous architectures seems both quicker and safer
than working out whether this is the compiler's fault.

tl;dr: the attached patch works, I would suggest using it.

Regards,
    S

[1] Actually it's a little more complicated than that; there are another
    couple of layers of typedef between the source code and the reality

[lzo2_2.08-1.1.diff (text/x-diff, attachment)]

Message #46 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Marc Kleine-Budde <mkl@blackshift.org>

To: Simon McVittie <smcv@debian.org>, 757037@bugs.debian.org

Subject: Re: Bug#757037: liblzop alignment problem

Date: Wed, 12 Nov 2014 11:51:43 +0100

[Message part 1 (text/plain, inline)]

On 11/12/2014 01:20 AM, Simon McVittie wrote:
> Control: severity 757037 grave
> Control: tags 757037 + patch
> 
> I'm bumping this bug up to grave severity, because it seems it makes
> this liblzo2 version entirely useless on a fairly common armel CPU (my armel
> has a Marvell Kirkwood armv5tel, as used in the SheevaPlug and its
> derivatives) in the default Debian configuration. It also makes liblzo2
> FTBFS on such machines, because the tests fail.

Thanks.

> On Sun, 09 Nov 2014 at 21:26:37 +0100, Marc Kleine-Budde wrote:
>> So from my point of view switching unaligned access
>> off in libzop with -DLZO_CFG_NO_UNALIGNED should be done on at least
>> ARMel. Are there other alignment sensitive platforms in debian?
> 
> (Please Cc me, I am not subscribed to this bug. Non-maintainer contributors
> to Debian bugs usually aren't.)

Done

> I think alignment-sensitivity is the common case, and quietly accepting
> unaligned accesses (and giving the correct answer!) as is done on x86 is the
> exceptional case...

Looking at the linux Kernel, there we have the
HAVE_EFFICIENT_UNALIGNED_ACCESS kconfig symbol. When an unaligned access
occurs in the kernel it's always fixed up. So the bad code will run, but
not as fast as it could.

> arch/arm/Kconfig:44:	select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU

Translated to debian this mean:
armel -> unaligned access will silently fail on <= ARMv5
         (unless configured otherwise in /proc/cpu/alignment)
         will work if run on => ARMv6, e.g. Raspberry Pi, Beagle Bone
         But usually you would use armhf on such a system.
armhf -> ok with unaligned access

> arch/arm64/Kconfig:50:	select HAVE_EFFICIENT_UNALIGNED_ACCESS

arm64 -> ok

> arch/powerpc/Kconfig:103:	select HAVE_EFFICIENT_UNALIGNED_ACCESS if !CPU_LITTLE_ENDIAN

ppc, ppc64 -> ok
ppcel -> not ok

> arch/x86/Kconfig:64:	select HAVE_EFFICIENT_UNALIGNED_ACCESS

i386, amd64 -> ok

> According to http://www.ibm.com/developerworks/library/pa-dalign/
> the mips family are alignment-sensitive, and so is powerpc sometimes,
> although it might do automatic fixup.
> 
> According to https://bugs.launchpad.net/ubuntu/+source/zmqpp/+bug/1256886
> Ubuntu armhf is alignment-sensitive (presumably Debian armhf is the same?)

The just "starting lzop" testcase works on my ARMv7, current debian
testing, both with armhf and armel root filesystem. /proc/cpu/alignment
shows now Problems with the user space.

> I've also seen references to sparc not liking unaligned accesses.
> 
> s390 can do unaligned accesses fine according to
> http://lxr.free-electrons.com/source/include/asm-s390/unaligned.h?v=2.2.26,
> but realistically, who's going to test that? Not me :-)
> 
> I think the safest thing would be to use -DLZO_CFG_NO_UNALIGNED
> on all Debian architectures except i386, amd64, and maybe the powerpc
> family if someone can test those. I attach a patch that restricts
> use of unaligned accesses to i386 and amd64 CPUs (any kernel);
> it's easy to extend to whitelist other CPUs where unaligned
> accesses are OK.
> 
> liblzo2 does actually try to avoid using unsafe unaligned accesses
> by using a structure
> 
>     typedef struct lzo_memops_TU2_struct { unsigned char a[2]; } lzo_memops_TU2;
> 
> and only doing a type-punning assignment similar to[1]
> 
>     *(lzo_memops_TU2 *) (void *) dest = *(const lzo_memops_TU2 *) (void *) src;

Which AFAIK should boil down to some kind of memcpy() with compile time
constant length.

> if that structure has required alignment 1... but it seems (recent?) gcc
> might be optimizing that into a single 2-byte operation which requires
> alignment. This might be a gcc bug, but then, I suspect this might be
> a violation of strict aliasing (hence undefined behaviour) if you get all
> language-lawyer about it. Pessimistically assuming that we have to access
> byte-by-byte on miscellaneous architectures seems both quicker and safer
> than working out whether this is the compiler's fault.
> 
> tl;dr: the attached patch works, I would suggest using it.
> 
> Regards,
>     S
> 
> [1] Actually it's a little more complicated than that; there are another
>     couple of layers of typedef between the source code and the reality
> 

regards,
Marc

[signature.asc (application/pgp-signature, attachment)]

Message #51 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: Marc Kleine-Budde <mkl@blackshift.org>, 757037@bugs.debian.org

Subject: Re: Bug#757037: liblzop alignment problem

Date: Wed, 12 Nov 2014 13:30:46 +0000

On 12/11/14 10:51, Marc Kleine-Budde wrote:
> On 11/12/2014 01:20 AM, Simon McVittie wrote:

> Translated to debian this mean: armel -> unaligned access will
> silently fail on <= ARMv5 (unless configured otherwise in
> /proc/cpu/alignment) will work if run on => ARMv6, e.g. Raspberry
> Pi, Beagle Bone But usually you would use armhf on such a system. 
> armhf -> ok with unaligned access

If the lzo2 maintainer would prefer, I'd be happy to adapt my patch to
allow unaligned accesses on additional architectures, or to swap the
assumption around so that it only forbids unaligned architectures on
specific architectures (which should include armel, and I don't
personally care about any others, but the mips* porters might).

If we need to differentiate between armel and armhf then it will be
necessary to use DEB_HOST_ARCH instead of DEB_HOST_ARCH_CPU (both
armel and armhf have "arm" as their DEB_HOST_ARCH_CPU), with something
like:

# whitelist approach
ifeq ($(filter amd64 %-amd64 i386 %-i386 armhf,$(DEB_HOST_ARCH)),)
# neither x86 family nor armhf: unaligned accesses are not good
endif

or

# blacklist approach
ifeq ($(filter-out armel mips%,$(DEB_HOST_ARCH)),)
# armel or mips-family: unaligned accesses are not good
endif

However, I will not be able to test on a baseline specimen of each
Debian architecture, because Debian porterboxes are not guaranteed to
match the baseline for the architecture: armel porterboxes are likely
to be better than armv5 in practice, i386 porterboxes are definitely
better than i586, and so on.

The only non-porterbox machines that I can personally test are an
armv5tel and a modern x86-64.

>> liblzo2 does actually try to avoid using unsafe unaligned
>> accesses by using a structure
>> 
>> typedef struct lzo_memops_TU2_struct { unsigned char a[2]; }
>> lzo_memops_TU2;
>> 
>> and only doing a type-punning assignment similar to[1]
>> 
>> *(lzo_memops_TU2 *) (void *) dest = *(const lzo_memops_TU2 *)
>> (void *) src;
> 
> Which AFAIK should boil down to some kind of memcpy() with compile
> time constant length.

That's what I'd expect too, but judging by the failure we're seeing,
it looks as though the compiler might be producing a single "load
16-bit" instruction, which is considerably cheaper but assumes alignment.

    S

Message #56 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: 757037@bugs.debian.org

Cc: Marc Kleine-Budde <mkl@blackshift.org>

Subject: lzo2: diff for NMU version 2.08-1.1

Date: Sun, 23 Nov 2014 22:08:24 +0000

[Message part 1 (text/plain, inline)]

Control: tags 757037 + patch pending

I've prepared an NMU for lzo2 (versioned as 2.08-1.1) based on
feedback on debian-devel and further testing, and
uploaded it to DELAYED/7. Please feel free to tell me if I
should delay it longer.

This has the same performance as 2.08-1 on amd64, and builds / passes
tests / works in OpenVPN on a Marvell Kirkwood (arm5vtel) CPU
running in Debian's default configuration, which previously exhibited
this bug when using or rebuilding 2.08-1.

Regards,
    S

[lzo2-2.08-1.1-nmu.diff (text/x-diff, attachment)]

Message #63 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: 757037@bugs.debian.org

Cc: Marc Kleine-Budde <mkl@blackshift.org>

Subject: Re: Bug#757037: lzo2: diff for NMU version 2.08-1.1

Date: Sun, 23 Nov 2014 23:34:18 +0000

[Message part 1 (text/plain, inline)]

On Sun, 23 Nov 2014 at 22:08:24 +0000, Simon McVittie wrote:
> I've prepared an NMU for lzo2 (versioned as 2.08-1.1) based on
> feedback on debian-devel and further testing

I have cancelled this delayed upload, because Julian Taylor pointed out
on debian-devel that there's a much simpler way to do this:
we know that the compiler used on Debian optimizes memcpy() calls
with constant n into optimal inlined instructions for the appropriate
CPU, without actually emitting a function call. So we can just define
LZO_MEMOPS_COPY2(dd, ss) to be memcpy(dd, ss, 1) and so on.

As a patch for upstream this would probably have to be guarded by
-DLZO_CFG_TRUST_THE_STDLIB or something, because they seem to be targeting
liblzo2 to be functional and fast even on the most naive compiler/libc
combinations possible... but on Debian, where we control libc and the
compiler, we know they're good.

It might be desirable to replace LZO_MEMOPS_SETn, LZO_MEMOPS_MOVEn
with memset and memmove calls too, since they also seem to violate
strict aliasing in ways that I suspect might lead an optimizing compiler
to emit the wrong code.

Sorry for the noise, I'll put together another patch shortly.

Using memcpy() like the attached does seem to work and have
equivalent performance, but I'm slightly concerned that
lzo2 might not guarantee not to pass overlapping arguments to
LZO_MEMOPS_COPYn; if they can overlap, memmove() would be
necessary. Some of the fallback implementations for the LZO_MEMOPS_MOVEn
family are not overlapping-argument-safe in any case, so memcpy()
is *probably* OK, but that might just mean nobody has noticed
the bug before.

    S

[0001-Use-memcpy-instead-of-reinventing-it.patch (text/x-diff, attachment)]

Message #70 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: 757037@bugs.debian.org

Cc: Marc Kleine-Budde <mkl@blackshift.org>, Thomas Herrmann <tobox@gmx.de>

Subject: Re: Bug#757037: lzo2: diff for NMU version 2.08-1.1

Date: Wed, 17 Dec 2014 00:08:57 +0000

[Message part 1 (text/plain, inline)]

On Sun, 23 Nov 2014 at 23:34:18 +0000, Simon McVittie wrote:
> As a patch for upstream this would probably have to be guarded by
> -DLZO_CFG_TRUST_THE_STDLIB or something, because they seem to be targeting
> liblzo2 to be functional and fast even on the most naive compiler/libc
> combinations possible... but on Debian, where we control libc and the
> compiler, we know they're good.
> 
> It might be desirable to replace LZO_MEMOPS_SETn, LZO_MEMOPS_MOVEn
> with memset and memmove calls too, since they also seem to violate
> strict aliasing in ways that I suspect might lead an optimizing compiler
> to emit the wrong code.

Here is an updated patch and a proposed nmudiff. I'm not NMUing it
right now because I want to let my armel run OpenVPN for a while first,
and make sure it keeps working... but I'll NMU it if I don't see
objections on the bug or OpenVPN issues on my armel.

Reviews welcome, particularly from the lzo2 maintainer.

Regards,
    S

[lzo2_2.08-1.1.diff (text/x-diff, attachment)]

[0001-Conditionally-replace-reinvention-of-memcpy-with-cal.patch (text/x-diff, attachment)]

Message #75 received at 757037-close@bugs.debian.org (full text, mbox, reply):

From: Simon McVittie <smcv@debian.org>

To: 757037-close@bugs.debian.org

Subject: Bug#757037: fixed in lzo2 2.08-1.1

Date: Fri, 19 Dec 2014 09:49:05 +0000

Source: lzo2
Source-Version: 2.08-1.1

We believe that the bug you reported is fixed in the latest version of
lzo2, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 757037@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Simon McVittie <smcv@debian.org> (supplier of updated lzo2 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Tue, 16 Dec 2014 23:35:36 +0000
Source: lzo2
Binary: liblzo2-dev liblzo2-2 liblzo2-2-udeb
Architecture: source
Version: 2.08-1.1
Distribution: unstable
Urgency: low
Maintainer: Peter Eisentraut <petere@debian.org>
Changed-By: Simon McVittie <smcv@debian.org>
Description:
 liblzo2-2  - data compression library
 liblzo2-2-udeb - data compression library (udeb)
 liblzo2-dev - data compression library (development files)
Closes: 757037
Changes:
 lzo2 (2.08-1.1) unstable; urgency=low
 .
   * Non-maintainer upload.
   * Replace liblzo's reinvention of memcpy() with calls to memcpy().
     gcc already knows how to inline memcpy calls with constant n,
     and also gets the alignment constraints right, avoiding incorrect
     unaligned accesses on armel (Closes: #757037)
Checksums-Sha1:
 da1f07537e7ba9c5432fa3f55f28919effd29433 1804 lzo2_2.08-1.1.dsc
 e4ec0f441875acab53a971066aad8d1c60108095 5752 lzo2_2.08-1.1.debian.tar.xz
Checksums-Sha256:
 00f1603d931a1dde5968bdbc5a93ea0bc4b18cc43c9bb39cb3306fe47eba1459 1804 lzo2_2.08-1.1.dsc
 72ca5fe8847b699304cc63aae1c633a34e350af48d0182631678cb95bef267f4 5752 lzo2_2.08-1.1.debian.tar.xz
Files:
 aae70607111cc7974fa9fbaa5da5799d 1804 libs optional lzo2_2.08-1.1.dsc
 afa248ea980357f980937d1d0bdfb59f 5752 libs optional lzo2_2.08-1.1.debian.tar.xz

-----BEGIN PGP SIGNATURE-----

iQIVAwUBVJPxaU3o/ypjx8yQAQgsNQ/9E5worfg41dbth9jnhmE9fuz1EVg3lKFJ
1/dqh1VHxaBEg1fSc302Nt/NQvVN205P5NOWZUitVom7aqX8dKW3dShjnXz8E+FW
Mct4YbC7637j+OmlG5ZRP4feRRT9Ktjk3Qcig2h1A/BToPVyqtYQuf41DyRBy4e7
1RNzL72IDLpcklowi0DiQ3y5b4KUktvKHgpDFZO6/lskVEhc3Zy6jCL8gkuVXHF+
7UghqbSmzYdE6PYCRhS19f7Mv/vHcvzgW1PW9jLwlymljIPQarKd2dCX1Ah3aoRD
c45qX+T+T6TVuiFQaWGkP59BTtewNXWmvxSkr9L9+af2FxXJ/nzn8IM2fKuez6E5
P1plzyqYWXAh4/0dcK3M/okNdEo2OQGLhcpblHYFlk6km6V6uhPUu24ARMRr7jGZ
SnSxvTxP3GQqI21fAF95QJ6INT9atNr0F4RKMvh9GtEoLrsQGaqA93kmmB1uAJCI
HeuiwvG2XVJRoSRPKQ77uGQ8eVduVtfGtzZF7wYTfz1aASG1esOVwmWA/gDblY1q
dcgfUhKJtKpTKqdA6w2vANFf2t6TuZwRcRu0xVqIq/0+Poj1HBsbatXeGAGws+0b
hMFuAyQW6DDzmxxh6D+KsHgZOvKnhx3YSwzUk3rrW/5seyVDr/M8vgp60JeSKtne
Vn00oCv7iZ0=
=Gyv8
-----END PGP SIGNATURE-----

Message #84 received at 757037@bugs.debian.org (full text, mbox, reply):

From: Peter Eisentraut <petere@debian.org>

To: 757037@bugs.debian.org

Subject: forwarded upstream

Date: Sun, 04 Jan 2015 09:49:25 -0500

Control: forwarded 757037 markus@oberhumer.com

Thanks for everyone's help on this.  I have forwarded the patch
upstream.

Send a report that this bug log contains spam.