Debian Bug report logs - #109717
ssh: scripted ssh sessions tend to hang

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Ryan Murray <rmurray@cyberhqz.com>

To: submit@bugs.debian.org

Subject: ssh: scripted ssh sessions tend to hang

Date: Wed, 22 Aug 2001 21:51:24 -0700

Package: ssh
Version: 1:2.3.0p1-1.13
Severity: important

Scripted ssh sessions tend to "hang" from time to time.  The source end
thinks it has a socket open in the ESTABLISHED state, and the remote end
shows the socket in TIME_WAIT.  A message like this is seen
in the log on the remote end:

sshd[27823]: error: channel 0: internal error: we do not read, but chan_read_failed for istate 8

This error message comes up quite soon after the publickey is accepted, with
both ssh2 and ssh protocols (3 seconds in the case I have handy).

I've reported the ssh server version above, clients have been the same version
(for sure), and possibly some of the 2.9 series (not as sure).

Message #10 received at 109717@bugs.debian.org (full text, mbox, reply):

From: Matthew Vernon <matthew@sel.cam.ac.uk>

To: Ryan Murray <rmurray@cyberhqz.com>, 109717@bugs.debian.org

Cc: control@bugs.debian.org

Subject: Bug#109717: ssh: scripted ssh sessions tend to hang

Date: Fri, 24 Aug 2001 02:24:31 +0100 (BST)

severity 109717 normal
quit

 > Scripted ssh sessions tend to "hang" from time to time.  The source end
 > thinks it has a socket open in the ESTABLISHED state, and the remote end
 > shows the socket in TIME_WAIT.  A message like this is seen
 > in the log on the remote end:
 
I need to know:

a) what exactly you're doing with ssh
b) if you can replicate this with the latest version of ssh (uploaded
yesterday) 
c) the output off ssh -v when this occurs

Thanks,

Matthew 

-- 
Rapun.sel - outermost outpost of the Pick Empire
http://www.pick.ucam.org

Message #17 received at 109717@bugs.debian.org (full text, mbox, reply):

From: Ryan Murray <rmurray@debian.org>

To: Matthew Vernon <matthew@sel.cam.ac.uk>

Cc: 109717@bugs.debian.org

Subject: Re: Bug#109717: ssh: scripted ssh sessions tend to hang

Date: Tue, 16 Oct 2001 00:47:44 -0700

[Message part 1 (text/plain, inline)]

On Fri, Aug 24, 2001 at 02:24:31AM +0100, Matthew Vernon wrote:
> severity 109717 normal
> quit
> 
>  > Scripted ssh sessions tend to "hang" from time to time.  The source end
>  > thinks it has a socket open in the ESTABLISHED state, and the remote end
>  > shows the socket in TIME_WAIT.  A message like this is seen
>  > in the log on the remote end:
>  
> I need to know:
> 
> a) what exactly you're doing with ssh

scripted sessions, there's lots of examples, here's two that I can find hung
right now.

ssh to auric used by the build-daemons to run wanna-build
eg: ssh -l buildd auric wanna-build -v --no-down-propagation --database=i386/build-db --user=buildd --dist=stable lapack_3.0-5.4

rsync over ssh usage by all .debian.org machines to transfer the ldap database
eg: rsync -e ssh -rp sshdist samosa /var/cache/userdir-ldap/hosts/gluck.debian.org .
    ssh -l sshdist samosa rsync --server --sender -pr . /var/cache/userdir-ldap/hosts/gluck.debian.org

poor network routing makes the problem happen far more often.

> b) if you can replicate this with the latest version of ssh (uploaded
> yesterday) 

The build daemons are using 2.9p2.  Most of the rest of the boxes
involved are running the 2.3 in use on all .debian.org boxes

> c) the output off ssh -v when this occurs

Server or client?

auric (buildd server):  (all of the servers and most of the clients
are this version)

SSH Version OpenSSH_2.3.0p1, protocol versions 1.5/2.0.
Compiled with SSL (0x0090600f).

reconfig (buildd client):
OpenSSH_2.9p2, SSH protocols 1.5/2.0, OpenSSL 0x0090602f

-- 
Ryan Murray, Debian Developer (rmurray@cyberhqz.com, rmurray@debian.org)
The opinions expressed here are my own.

[Message part 2 (application/pgp-signature, inline)]

Message #22 received at 109717@bugs.debian.org (full text, mbox, reply):

From: Pete Harlan <harlan@artselect.com>

To: 109717@bugs.debian.org

Subject: ssh connections hanging

Date: Tue, 4 Dec 2001 18:44:01 -0600

Hello,

Any progress with this?  It has broken a lot of our sysadmin scripts.

I've created a simple case that produces the problem, and a
workaround (that works in my simple case and one of the cases that has
been biting us (since the upgrade to ssh2.))

First, I'm using the latest Debian Unstable ssh (also tried 3.0.2p1
from openssh.org); all scripts run on this machine:

	% ssh -V
	OpenSSH_3.0.1p1, SSH protocols 1.5/2.0, OpenSSL 0x0090602f

	% dpkg -s ssh
	Package: ssh
	Status: install ok installed
	[...]
	Version: 1:3.0.1p1-1.2
	[...]

I have the following two scripts in my home dir:

	% cat bgtask.pl 
	#!/usr/bin/perl -w

	use strict;

	while (1)
	{
		sleep 10;
	}

and

	% cat launchit.sh 
	#!/bin/bash

	nohup bgtask.pl &

	echo launchit.sh is done.

To reproduce the hang:

	% ssh localhost launchit.sh 
	launchit.sh is done.

The ssh session doesn't return.  I Ctrl-C it and it comes back with:

	Killed by signal 2.
	% 

If I instead run the script directly, not over ssh:

	% launchit.sh
	launchit.sh is done.
	% nohup: appending output to `nohup.out'

It comes back to the prompt immediately (with nohup's usual warning
about appending output to nohup.out.)

To make the hang go away, I can change launchit.sh to direct all
output from the 'nohup' line to /dev/null:

	% cat launchit.sh 
	#!/bin/bash

	nohup bgtask.pl &>/dev/null &

	echo launchit.sh is done.

Now the script returns as expected:

	% ssh localhost launchit.sh 
	launchit.sh is done.
	%

It's as if ssh is aware that there are processes that could write to
the tty, so it's waiting until the processes go away.  But that's not
what bash does, it's not what Potato's ssh did, and it's not what a
lot of our scripts are designed for.

(The script that bit us the most here is the mysql.server startup
script for MySQL.  It launches MySQL in the background, without
redirecting all output from the backgrounded process to /dev/null.
This means that we can't write a script that stops and starts our db
server without modifying the mysql.server scripts with the &>/dev/null.)

Note that non-free ssh2 doesn't do this; it just produces:

	% ssh localhost ./launchit.sh 
	launchit.sh is done.
	Connection to localhost closed.
	%

Also, the free ssh _client_ doesn't have a problem when the server is
the non-free ssh2 server, so it appears to be a server issue, not a
client issue.

I downloaded the latest (3.0.2p1) source from openssh.org and it had
this problem too, so it's probably an upstream thing.

--Pete

Message #27 received at 109717@bugs.debian.org (full text, mbox, reply):

From: rmurray@cyberhqz.com (Ryan Murray)

To: 109717@bugs.debian.org

Subject: this has gotten worse

Date: Thu, 6 Dec 2001 15:14:32 -0800

After upgrading both server and client to 3.0.1p1-1.1 (and -1.2), this problem
has gotten worse.  The problem described by pete is slightly different from
the one I am running into, in that I am not backgrounding the process on
the remote machine, but am waiting for all of the output and the connection
to close before continuing.

Message #32 received at 109717@bugs.debian.org (full text, mbox, reply):

From: Pete Harlan <harlan@artselect.com>

To: 109717@bugs.debian.org

Subject: openssh folks discussing it

Date: Fri, 14 Dec 2001 17:46:21 -0600

FYI,

There are a lot of messages about this issue on the openssh mailing
list archives from the past couple of days.  The issues are much more
involved than I'd imagined, but it looks like solutions (to what many
believe is not a problem) are in sight.

	http://marc.theaimsgroup.com/?l=openssh-unix-dev&r=1&w=2

--Pete

Message #37 received at 109717@bugs.debian.org (full text, mbox, reply):

From: Pete Harlan <harlan@artselect.com>

To: 109717@bugs.debian.org

Subject: Haven't seen random hanging in a while

Date: Tue, 18 Dec 2001 11:52:43 -0600

FYI,

In addition to the "waits for backgrounded processes with open
descriptors to finish" issue, we'd seen random hanging of sessions
where there was no backgrounding, similar to what the other people in
this bugreport were reporting.

Those hangs have gone away---we never see stale cron-initiated ssh
hangs all over the place anymore.  Thanks!

--Pete

Message #40 received at 109717-submitter@bugs.debian.org (full text, mbox, reply):

From: Jonathan David Amery <jdamery@ysolde.ucam.org>

To: 109717-submitter@bugs.debian.org

Subject: Does this bug still exist?

Date: Fri, 19 Apr 2002 18:40:45 +0100

 We have a report from Pete Harlan that bug #109717 (ssh: scripted ssh
sessions tend to hang) has stopped occuring.  Do you concur with this?

 Thanks,

 Jonathan.

Message #45 received at 109717-submitter@bugs.debian.org (full text, mbox, reply):

From: Justin Pryzby <justinpryzby@users.sourceforge.net>

To: 109717-submitter@bugs.debian.org, rmurray@debian.org

Cc: control@bugs.debian.org

Subject: your bug/ssh: scripted ssh sessions tend to hang

Date: Fri, 14 Apr 2006 12:54:34 -0400

tag 109717 moreinfo
thanks

Hello Ryan,

Do you still see this bug?

http://bugs.debian.org/109717
ssh: scripted ssh sessions tend to hang

Thanks
Justin

Message #50 received at 109717-done@bugs.debian.org (full text, mbox, reply):

From: Ryan Murray <rmurray@cyberhqz.com>

To: 109717-done@bugs.debian.org

Subject: scripted ssh sessions

Date: Thu, 10 Sep 2020 09:46:35 -0700

Hi,

This was an occasional bug that was hard to reproduce, but I don't think 
anyone has seen it since the upstream fixes in 2001, so (finally) closing.

Send a report that this bug log contains spam.