Debian Bug report logs - #606821
mksh: bash-style process substitution

version graph

Package: mksh; Maintainer for mksh is Thorsten Glaser <tg@mirbsd.de>; Source for mksh is src:mksh.

Reported by: Jonathan Nieder <jrnieder@gmail.com>

Date: Sat, 11 Dec 2010 23:45:05 UTC

Severity: wishlist

Tags: upstream

Found in version mksh/39.3.20101101-1

Forwarded to http://thread.gmane.org/gmane.os.miros.general/9338

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Thorsten Glaser <tg@mirbsd.de>:
Bug#606821; Package mksh. (Sat, 11 Dec 2010 23:45:08 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
New Bug report received and forwarded. Copy sent to Thorsten Glaser <tg@mirbsd.de>. (Sat, 11 Dec 2010 23:45:08 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: submit@bugs.debian.org
Subject: mksh: bash-style process substitution
Date: Sat, 11 Dec 2010 17:40:23 -0600
Package: mksh
Version: 39.3.20101101-1
Severity: wishlist
Tags: upstream

Hi Thorsten,

Recently I found myself debugging a pipeline with "tee".
Unfortunately the logs grew large very quickly, so as a stopgap
measure I used a compressor.

	mkfifo backflow

	... <backflow |
	tee >(xz -1 >log-one.xz) |
	... |
	tee >(xz -1 >log-two.xz) |
	... |
	tee >(xz -1 >log-three.xz) >backflow

This construct is also handy when one wants to update the current
environment downstream from some other process.

	while read -r line
	do
		... something interesting with line ...
		accum=$(updated result)
	done < <(upstream process)

	: print result
	printf "%s\n" "$accum"

The semantics:

      <( ... )    - runs ... in a subshell in the background, with its
                    output connected to a pipe.  The "<( ... )"
                    expression evaluates to a filename that can be
                    opened to read from that pipe.

      >( ... )    - runs ... in a subshell in the background, with its
                    input connected to a pipe.  The ">( ... )"
                    evaluates to a filename that can be opened to write
                    to that pipe.

On Linux and similar OSes this is typically implemented using
/proc/fd.  A more portable implementation would use FIFOs.

What do you think?  Is it worth implementing in mksh?




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#606821; Package mksh. (Sun, 12 Dec 2010 00:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thorsten Glaser <tg@mirbsd.de>:
Extra info received and forwarded to list. (Sun, 12 Dec 2010 00:09:03 GMT) Full text and rfc822 format available.

Message #10 received at 606821@bugs.debian.org (full text, mbox):

From: Thorsten Glaser <tg@mirbsd.de>
To: 606821@bugs.debian.org
Subject: Re: Bug#606821: mksh: bash-style process substitution
Date: Sun, 12 Dec 2010 00:00:33 +0000 (UTC)
Jonathan Nieder dixit:

>This construct is also handy when one wants to update the current
>environment downstream from some other process.

	process |&
	while IFS= read -pr line; do
		…
	done

↑ is the proper ksh idiom for that.


>      <( ... )    - runs ... in a subshell in the background, with its
>                    output connected to a pipe.

Ah, interesting. I would have thought it ran the stuff first,
wrote it to a temporary file and then passed that to the
other process, sort of like a dynamic here document, from
when I first read about it. (Didn’t know >(…) existed.)

>On Linux and similar OSes this is typically implemented using
>/proc/fd.  A more portable implementation would use FIFOs.

I don’t know whether this works on all mksh target OSes, but
that can surely be investigated.

>What do you think?  Is it worth implementing in mksh?

<(…) is already on the wishlist – if you want to submit a
patch, you’re welcome ;-) it should work on DEC ULTRIX 4.5,
DEC OSF/1 V2.0, Minix 3, Haiku, Cygwin, UWIN, etc. though,
and, if possible, Plan 9. ☺

A side note, I’d like to track upstream feature requests
*not* inside the Debian BTS. We usually use the MirBSD
mailing list for that, although there’s now a Launchpad
project associated with mksh which can be used if desired.

bye,
//mirabilos
-- 
<ch> you introduced a merge commit        │<mika> % g rebase -i HEAD^^
<mika> sorry, no idea and rebasing just fscked │<mika> Segmentation
<ch> should have cloned into a clean repo      │  fault (core dumped)
<ch> if I rebase that now, it's really ugh     │<mika:#grml> wuahhhhhh




Information forwarded to debian-bugs-dist@lists.debian.org, Thorsten Glaser <tg@mirbsd.de>:
Bug#606821; Package mksh. (Mon, 13 Dec 2010 21:24:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Thorsten Glaser <tg@mirbsd.de>. (Mon, 13 Dec 2010 21:24:03 GMT) Full text and rfc822 format available.

Message #15 received at 606821@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: miros-discuss@mirbsd.org
Subject: Re: mksh: bash-style process substitution
Date: Mon, 13 Dec 2010 15:20:17 -0600
Thorsten Glaser wrote:
> Jonathan Nieder dixit[1]:

>>      <( ... )    - runs ... in a subshell in the background, with its
>>                    output connected to a pipe.
[...]
>> On Linux and similar OSes this is typically implemented using
>> /proc/fd.  A more portable implementation would use FIFOs.
>
> I don’t know whether this works on all mksh target OSes, but
> that can surely be investigated.
[...]
> <(…) is already on the wishlist – if you want to submit a
> patch, you’re welcome ;-) it should work on DEC ULTRIX 4.5,
> DEC OSF/1 V2.0, Minix 3, Haiku, Cygwin, UWIN, etc. though,
> and, if possible, Plan 9. ☺

Ha.  mkfifo was part of POSIX.1-1988, so it should be fairly
portable.

 - Ultrix and OSF/1 2.0:	has mkfifo.
 - Minix 3:	ought to have mkfifo (claims POSIX compliance).
 - Haiku:	has mkfifo
 - Cygwin:	has /proc/self/fd
 - UWIN:	complicated?  The ast-ast library supports /dev/fd,
        	but native Windows apps obviously don't.
 - Plan 9:	has devdup[2].  "bind #d /dev/fd" and you're good to go.

It's on my list now. :)

>> This construct is also handy when one wants to update the current
>> environment downstream from some other process.
>
> 	process |&
> 	while IFS= read -pr line; do
> 		…
> 	done

Nice.  I had been hoping for something like that.

> A side note, I’d like to track upstream feature requests
> *not* inside the Debian BTS. We usually use the MirBSD
> mailing list

Thanks for a pointer.  I'll report straight here from now on (perhaps
along with reports to Debian with an appropriate "Forwarded:"
pseudo-header).

[1] http://bugs.debian.org/606821
[2] http://plan9.bell-labs.com/magic/man2html/3/dup




Set Bug forwarded-to-address to 'http://thread.gmane.org/gmane.os.miros.general/9338'. Request was from Jonathan Nieder <jrnieder@gmail.com> to control@bugs.debian.org. (Mon, 13 Dec 2010 22:09:03 GMT) Full text and rfc822 format available.

Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#606821; Package mksh. (Sat, 26 Nov 2011 20:42:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thorsten Glaser <tg@mirbsd.de>:
Extra info received and forwarded to list. (Sat, 26 Nov 2011 20:42:06 GMT) Full text and rfc822 format available.

Message #22 received at 606821@bugs.debian.org (full text, mbox):

From: Thorsten Glaser <tg@mirbsd.de>
To: 606821@bugs.debian.org
Cc: miros-discuss@mirbsd.org
Subject: mksh process substitution
Date: Sat, 26 Nov 2011 20:37:11 +0000 (UTC)
Hi,

apparently (from a #grml discussion in IRC), zsh has two:

<(…) uses /dev/fd (or pipes, I guess)
=(…) uses temporary files (to aid applications to lseek on it)

Maybe we should just take on the latter (reuse the heredoc
infrastructure) and make it available as <(…) (since nobody
even knows about =(…) and so it's pointless)?

bye,
//mirabilos
-- 
  “Having a smoking section in a restaurant is like having
          a peeing section in a swimming pool.”
						-- Edward Burr




Information forwarded to debian-bugs-dist@lists.debian.org, Thorsten Glaser <tg@mirbsd.de>:
Bug#606821; Package mksh. (Sat, 26 Nov 2011 22:45:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Thorsten Glaser <tg@mirbsd.de>. (Sat, 26 Nov 2011 22:45:03 GMT) Full text and rfc822 format available.

Message #27 received at 606821@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: 606821@bugs.debian.org
Cc: miros-discuss@mirbsd.org
Subject: Re: mksh process substitution
Date: Sat, 26 Nov 2011 16:41:59 -0600
Thorsten Glaser wrote:

> apparently (from a #grml discussion in IRC), zsh has two:
>
> <(…) uses /dev/fd (or pipes, I guess)
> =(…) uses temporary files (to aid applications to lseek on it)
>
> Maybe we should just take on the latter (reuse the heredoc
> infrastructure) and make it available as <(…) (since nobody
> even knows about =(…) and so it's pointless)?

A common use of <(...) is to parse output from a command and let it
influence the current environment:

	while read key value
	do
		if test "$key" = thegoodone
		then
			val=$value
			break
		fi
	done < <(producer)

The "producer" command here could produce a potentially infinite
stream of output.

The application that led me to mention process substitution to you
also involved concurrent processes.

	producer |
	tee >(xz >log.xz) |
	consumer

Hope that helps,
Jonathan




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#606821; Package mksh. (Sat, 26 Nov 2011 22:51:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thorsten Glaser <tg@mirbsd.de>:
Extra info received and forwarded to list. (Sat, 26 Nov 2011 22:51:03 GMT) Full text and rfc822 format available.

Message #32 received at 606821@bugs.debian.org (full text, mbox):

From: Thorsten Glaser <tg@mirbsd.de>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: 606821@bugs.debian.org, miros-discuss@mirbsd.org
Subject: Re: mksh process substitution
Date: Sat, 26 Nov 2011 22:46:27 +0000 (UTC)
Jonathan Nieder dixit:

>A common use of <(...) is to parse output from a command and let it
>influence the current environment:
>
>	while read key value

We use coprocesses for that.

>The "producer" command here could produce a potentially infinite
>stream of output.

I think this case is best be met with a coprocess then.

>The application that led me to mention process substitution to you
>also involved concurrent processes.
>
>	producer |
>	tee >(xz >log.xz) |
>	consumer

True, there’s >(…) which I hadn’t thought of. So the complex
fd/pipe logic must still be there. GNU bash doesn’t mention
=(…) though so I guess we should take this to a cross-shell-
developers discussion instead.

(Are you ok with closing the Debian BTS part of this? If you
want, I can move it to Launchpad, if you’d prefer to have it
not just on a mailing list.)

bye,
//mirabilos
-- 
  “Having a smoking section in a restaurant is like having
          a peeing section in a swimming pool.”
						-- Edward Burr




Information forwarded to debian-bugs-dist@lists.debian.org:
Bug#606821; Package mksh. (Wed, 05 Dec 2012 19:51:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Thorsten Glaser <tg@mirbsd.de>:
Extra info received and forwarded to list. (Wed, 05 Dec 2012 19:51:03 GMT) Full text and rfc822 format available.

Message #37 received at 606821@bugs.debian.org (full text, mbox):

From: Thorsten Glaser <tg@mirbsd.de>
To: 606821@bugs.debian.org
Subject: mksh: bash-style process substitution
Date: Wed, 5 Dec 2012 19:46:19 +0000 (UTC)
Hi,

do these have to be pipes? That got me wondering.

Otherwise, I could do a tree-level transform (except the IOREDIR
cases are again slightly more complex). Something like this:

• if <(command2) is contained in command1:
  ‣ allocate a tempfile
    ⇒ attach the tempfile to the list of cleanups of command1
  ‣ run the command2
    ⇒ redirect its stdout to the tempfile
    ⇒ keep stderr and (?) stdin
    ⇒ throw away $? (?)
  ‣ replace <(command2) with the name of the tempfile
  ‣ repeat for any further occurrences
• if >(command3) is contained in command1:
  ‣ allocate a tempfile
    ⇒ attach the tempfile to the list of cleanups of command3
  ‣ replace >(command3) with the name of the tempfile on command1
  ‣ change the TCOM tree command1 to a sequence
    { command1 ; tmpvar=$? ; command3 ; return $tmpvar ; }
  ‣ for any further occurrences command4, … append to the new
    command sequence before the return

Of course, this needs much more careful checking. What does
GNU bash do to stdin, stderr, and the exit codes of the
substituted processes? (Though, strictly spoken, we could
do the above and test it quickly, then change the tempfile
mechanism to spit out FIFOs instead; I’ve done the last step
in an experimental branch already somewhere, so it’s easy.)

bye,
//mirabilos
-- 
I want one of these. They cost 720 € though… good they don’t have the HD hole,
which indicates 3½″ floppies with double capacity… still. A tad too much, atm.
‣ http://www.floppytable.com/floppytable-images-1.html



Information forwarded to debian-bugs-dist@lists.debian.org, Thorsten Glaser <tg@mirbsd.de>:
Bug#606821; Package mksh. (Thu, 06 Dec 2012 07:09:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Thorsten Glaser <tg@mirbsd.de>. (Thu, 06 Dec 2012 07:09:03 GMT) Full text and rfc822 format available.

Message #42 received at 606821@bugs.debian.org (full text, mbox):

From: Jonathan Nieder <jrnieder@gmail.com>
To: Thorsten Glaser <tg@mirbsd.de>
Cc: 606821@bugs.debian.org
Subject: Re: mksh: bash-style process substitution
Date: Wed, 5 Dec 2012 23:06:04 -0800
Thorsten Glaser wrote:

> Hi,
>
> do these have to be pipes? That got me wondering.

Yes, for most use cases I have run into they do need to be pipes and
not tempfiles.  I do think =(...) is neat, too, but it is a distinct
feature.

Thanks,
Jonathan



Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Thu Apr 17 16:02:34 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.