Debian Bug report logs - #987916
openssh: Segfault or malloc_consolidate(): invalid chunk size + Aborted with GSSAPITrustDns yes

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Christoph Anton Mitterer <calestyo@scientia.net>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: openssh: Segfault or malloc_consolidate(): invalid chunk size + Aborted with GSSAPITrustDns yes

Date: Sun, 02 May 2021 03:30:07 +0200

Source: openssh
Version: 1:8.4p1-5
Severity: important


Hey.

This is from https://bugzilla.mindrot.org/show_bug.cgi?id=3307:

Hey there.

I've noted the two errors, with the following setup:

Locally, I have:
OpenSSH_8.4p1 Debian-5, OpenSSL 1.1.1k  25 Mar 2021

from which I connect to some internal node at CERN (hammercloud-ai-11.cern.ch) via some publicly available node (lxplus.cern.ch) which all have:
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

The lxplus.cern.ch is actually a round robin DNS name, but all nodes behind have the same ssh server key.


Since CERN uses AFS, I have to do GSSAPI auth.
Locally I have a keytab file created with ktuil, which even works out of the box with SSH - that is, if I don't have a krb ticket yet, it automatically creates one.


My SSH config looks like the following:
Host hammercloud-ai-11.cern.ch
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials yes
        GSSAPIRenewalForcesRekey yes
        GSSAPITrustDns yes
        ProxyJump       lxplus.cern.ch


Host lxplus.cern.ch
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials yes
        GSSAPIRenewalForcesRekey yes
        GSSAPITrustDns yes
#       ControlMaster   auto
#       ControlPersist  10s
#       ControlPath     ~/.ssh/channel-mux/%r@%h:%p

Host *.cern.ch
        User someUser
        IdentityFile    ~/.ssh/id_ed25519
        SetEnv "LANG=en_US.UTF-8"


Further, I do have a custom locale which is basically en_US.UTF-8, but with some international stuff like "," as decimal separator.

Now that works to login to lxplus, and from there (within an interactive session) to hammercloud-ai-11.

When I use the ProxyJump however and directly go to hammercloud-ai-11, I start to see errors.


1) with LANG=en_DE.UTF-8 it segfaults:
$ ssh hammercloud-ai-11.cern.ch -v
...
Authenticated to hammercloud-ai-11.cern.ch (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_DE.UTF-8
Segmentation fault
$ debug1: stdio forwarding: done

Interestingly it seems to still try to send "my" locale instead what I've configured above with:
        SetEnv "LANG=en_US.UTF-8"



2) the same with LANG=C
$ export LANG=C
$ ssh hammercloud-ai-11.cern.ch -v
...
Authenticated to hammercloud-ai-11.cern.ch (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = C
malloc_consolidate(): invalid chunk size
                                        Aborted
$ debug1: stdio forwarding: done


Whether or not using a Control Channel doesn't seem to matter.


When I comment the
Host hammercloud-ai-11.cern.ch
...
#       GSSAPITrustDns yes


It works in both cases.

Commeting the same for lxplus (the proxy node), doesn't solve the issue.


Any ideas?

Cheers,
Chris.



forgot:

May 01 16:38:39 heisenberg kernel: ssh[16368]: segfault at 7e00000008 ip 00007f646525a86c sp 00007ffd72b5fb30 error 4 in libc-2.31.so[7f64651f9000+14b000]
May 01 16:38:39 heisenberg kernel: Code: 43 28 00 00 00 00 48 8b 54 24 08 48 89 ef 48 89 43 10 48 83 cf 01 48 89 7b 08 48 89 53 18 48 89 2c 2b 48 85 c9 74 87 48 89 cb <48> 8b 43 08 89 c1 c1 e9 04 83 e9 02 49 8d 4c cc 10 49 39 cd 0f 85
May 01 16:38:50 heisenberg kernel: ssh[16375]: segfault at 7e00000008 ip 00007fe602caa86c sp 00007fff2ac78150 error 4 in libc-2.31.so[7fe602c49000+14b000]
May 01 16:38:50 heisenberg kernel: Code: 43 28 00 00 00 00 48 8b 54 24 08 48 89 ef 48 89 43 10 48 83 cf 01 48 89 7b 08 48 89 53 18 48 89 2c 2b 48 85 c9 74 87 48 89 cb <48> 8b 43 08 89 c1 c1 e9 04 83 e9 02 49 8d 4c cc 10 49 39 cd 0f 85




-- System Information:
Debian Release: 11.0
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-6-amd64 (SMP w/4 CPU threads)
Locale: LANG=en_DE.UTF-8, LC_CTYPE=en_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Message #10 received at 987916@bugs.debian.org (full text, mbox, reply):

From: Bernhard Übelacker <bernhardu@mailbox.org>

To: Christoph Anton Mitterer <calestyo@scientia.net>

Cc: 987916@bugs.debian.org

Subject: Re: Bug#987916: openssh: Segfault or malloc_consolidate(): invalid chunk size + Aborted with GSSAPITrustDns yes

Date: Tue, 18 May 2021 10:11:14 +0200

Hello Chris,
I am not involved in packaging, just trying to give some
pointers to get better information for the maintainers.

In [1] are several possible actions listed, that could be
used to get more informations.

Just to clarify, host heisenberg is your local system,
from which the connection starts?

If yes, I would propose these actions:

- If possible, install systemd-coredump. Then in the journalctl
  output there should already appear a basic backtrace with
  a crashing process, plus a core gets collected of it and
  stored for some time (e.g. "coredumpctl list" or "coredumpctl gdb").

- Try connecing setting following environment before:
    export MALLOC_CHECK_=3
  That should make stricter checks in the allocator
  and maybe fail earlier.

- You might install debug symbols too, also described in [1].
  With that you could also start the connection inside a debugger:
    gdb -q --args ssh hammercloud-ai-11.cern.ch -v
  And do following actions at the gdb prompt:
      run
      bt
      detach
      quit
  That way the 'bt' command should print a backtrace
  that might help to reproduce the issue.

Kind regards,
Bernhard

[1] https://wiki.debian.org/HowToGetABacktrace

Send a report that this bug log contains spam.