Debian Bug report logs - #935115
bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX

Date: Mon, 19 Aug 2019 17:47:08 +0200

Package: bash
Version: 5.0-4
Severity: critical
Justification: breaks unrelated software

Found this gem in #934027:

tglase@tglase:~ $ cat testscript
dbc_mysql_createdb(){
	local ret l_dbname _dbc_nodb

	echo "dbc_mysql_createdb: _dbc_nodb(1)=$_dbc_nodb"
	_dbc_nodb="yes" dbc_mysql_exec_command "CREATE DATABASE \`$dbc_dbname\`${extrasql:-}"
	ret=$?
	echo "dbc_mysql_createdb: _dbc_nodb(2)=$_dbc_nodb"	# POSIX: unspecified
	/usr/bin/env | fgrep -c _dbc_nodb | sed 's/^/after: /'	# POSIX: unspecified
	echo dbc_mysql_createdb:$ret
}

dbc_mysql_exec_command(){
	local statement l_sqlfile l_retval
	statement="$@"
	l_retval=0
	echo "dbc_mysql_exec_command: _dbc_nodb=$_dbc_nodb"	# POSIX: required
	/usr/bin/env | fgrep -c _dbc_nodb | sed 's/^/inner: /'	# POSIX: unspecified
	return $l_retval
}

dbc_mysql_createdb
tglase@tglase:~ $ dash testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 0
dbc_mysql_createdb: _dbc_nodb(2)=yes
after: 0
dbc_mysql_createdb:0
tglase@tglase:~ $ ksh -c 'alias local=typeset; . ./testscript'
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 0
dbc_mysql_createdb: _dbc_nodb(2)=yes
after: 0
dbc_mysql_createdb:0
tglase@tglase:~ $ lksh -o posix testscript  # native mksh +o posix is identical
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=yes
after: 1
dbc_mysql_createdb:0
tglase@tglase:~ $ posh testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0
dbc_mysql_createdb:0
tglase@tglase:~ $ yash testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0
dbc_mysql_createdb:0
tglase@tglase:~ $ zsh -c 'emulate sh; . ./testscript'
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=yes
after: 1
dbc_mysql_createdb:0
tglase@tglase:~ $ bash --version 2>&1 | head -1
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnux32)
tglase@tglase:~ $ bash testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0
dbc_mysql_createdb:0
tglase@tglase:~ $ bash --posix testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 1
dbc_mysql_createdb:0
tglase@tglase:~ $ schroot -prc stretch
(stretch-i386)tglase@tglase:~ $ bash --version 2>&1 | head -1
GNU bash, version 4.4.12(1)-release (i686-pc-linux-gnu)
(stretch-i386)tglase@tglase:~ $ bash testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0
dbc_mysql_createdb:0
(stretch-i386)tglase@tglase:~ $ bash --posix testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=yes
after: 1
dbc_mysql_createdb:0


The expected output is:

dbc_mysql_createdb: _dbc_nodb(1)=	# initially not set / empty
dbc_mysql_exec_command: _dbc_nodb=yes	# MUST be visible inside the function
inner: 0 or 1				# MAY be exported, does not need to
dbc_mysql_createdb: _dbc_nodb(2)=[yes]	# MAY be visible afterwards, optional
after: 0 or 1				# if visible afterwards MAY be exported
dbc_mysql_createdb:0

POSIX reference:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
specifically:

 * If the command name is a function that is not a standard utility
   implemented as a function, variable assignments shall affect the
   current execution environment during the execution of the function.
   It is unspecified:
   + Whether or not the variable assignments persist after the
     completion of the function
   + Whether or not the variables gain the export attribute during the
     execution of the function
   + Whether or not export attributes gained as a result of the variable
     assignments persist after the completion of the function (if
     variable assignments persist after the completion of the function)

Tested shells:

┌──────────────┬────────────────┬──────────┬────────────────────┬──────────┐
│ shell \ what │ inside visible │ exported │ afterwards visible │ exported │
├──────────────┼────────────────┼──────────┼────────────────────┼──────────┤
│ bash4        │ ✔ as required  │ ✓ (good) │ ✗ no, permitted    │ –        │
│ bash4 posix  │ ✔ as required  │ ✓ (good) │ ✓ yes, permitted   │ ✓ (ok)   │
│ bash5        │ ✔ as required  │ ✓ (good) │ ✗ no, permitted    │ –        │
│ bash5 posix  │ ✘  !!FAIL!! ⚠  │ ✓ (huh?) │ ✗ empty            │ ✓ (huh?) │
│ dash         │ ✔ as required  │ ✗ (ok)   │ ✗ no, permitted    │ –        │
│ ksh93        │ ✔ as required  │ ✗ (ok)   │ ✗ no, permitted    │ –        │
│ mksh ±posix  │ ✔ as required  │ ✓ (good) │ ✓ yes, permitted   │ ✓ (ok)   │
│ posh (old)   │ ✔ as required  │ ✓ (good) │ ✗ no, permitted    │ –        │
│ yash         │ ✔ as required  │ ✓ (good) │ ✗ no, permitted    │ –        │
│ zsh as sh    │ ✔ as required  │ ✓ (good) │ ✓ yes, permitted   │ ✓ (ok)   │
└──────────────┴────────────────┴──────────┴────────────────────┴──────────┘

Out of all tested shells, only bash5 in POSIX mode is completely broken:
it makes an empty, but exported (huh?) variable available to both the
function’s body and afterwards but does not pass on the content.

Out of all nonbroken shells, all shells that do make the variable
visible afterwards export it, both inside and afterwards, which is
consistent: bash4 in POSIX mode, mksh in native and POSIX modes,
zsh in sh mode.

Out of all nonbroken shells, only dash and AT&T ksh93 do not export
the variable during execution of the function body, which, while
perhaps not expected by the programmer, is valid POSIX.

The “afterwards visible” property is dividing. Hiding it would be
consistent related to how assignments are handled running non-functions,
which is what the majority does (bash4/5 in nōn-POSIX mode, dash,
AT&T ksh93, posh and yash), only a very interesting minority, namely
bash4 in POSIX mode, mksh and zsh in “sh” (POSIX) mode, keep the variable
alive. (With mksh upstream hat on, and looking at pdksh, antecessor of
both posh and mksh: pdksh keeps it visible (so, this is a deliberate(?)
change in posh) but did not export it, which apparently got fixed later
in both shells.)

-- System Information:
Debian Release: bullseye/sid
  APT prefers unreleased
  APT policy: (500, 'unreleased'), (500, 'buildd-unstable'), (500, 'unstable'), (100, 'experimental')
Architecture: x32 (x86_64)
Foreign Architectures: i386, amd64

Kernel: Linux 4.19.0-5-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=C, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/lksh
Init: sysvinit (via /sbin/init)

Versions of packages bash depends on:
ii  base-files   11
ii  debianutils  4.8.6.3
ii  libc6        2.28-10
ii  libtinfo6    6.1+20190803-1

Versions of packages bash recommends:
pn  bash-completion  <none>

Versions of packages bash suggests:
pn  bash-doc  <none>

-- no debconf information

Message #12 received at 935115-quiet@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@mirbsd.de>

To: miros-mksh@mirbsd.org

Cc: 935115-quiet@bugs.debian.org, svashisht@redhat.com

Subject: passing variable assignments to functions (was Re: Bug#935115: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX)

Date: Mon, 19 Aug 2019 15:59:42 +0000 (UTC)

Dixi quod…

>┌──────────────┬────────────────┬──────────┬────────────────────┬──────────┐
>│ shell \ what │ inside visible │ exported │ afterwards visible │ exported │
>├──────────────┼────────────────┼──────────┼────────────────────┼──────────┤
>│ ksh93        │ ✔ as required  │ ✗ (ok)   │ ✗ no, permitted    │ –        │
>│ mksh ±posix  │ ✔ as required  │ ✓ (good) │ ✓ yes, permitted   │ ✓ (ok)   │
>│ posh (old)   │ ✔ as required  │ ✓ (good) │ ✗ no, permitted    │ –        │
>└──────────────┴────────────────┴──────────┴────────────────────┴──────────┘

>The “afterwards visible” property is dividing. Hiding it would be
>consistent related to how assignments are handled running non-functions,

Looking at POSIX again, same place:

 * If the command name is a standard utility implemented as a function
   (see XBD [142]Utility), the effect of variable assignments shall be
   as if the utility was not implemented as a function.

This is easier, not to mention more reliable to do, if all function
invocations are handled as if the callee is not a function, that is,
inside visible plus exported, outside untouched.

I’m strongly considering changing mksh to match; note this was not
changed deliberately before / from pdksh/oksh.

bye,
//mirabilos
-- 
This space for rent.

https://paypal.me/mirabilos to support my work.

Message #17 received at submit@bugs.debian.org (full text, mbox, reply):

From: Chet Ramey <chet.ramey@case.edu>

To: Thorsten Glaser <tg@mirbsd.de>, 935115@bugs.debian.org, Debian Bug Tracking System <submit@bugs.debian.org>

Cc: chet.ramey@case.edu

Subject: Re: Bug#935115: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX

Date: Mon, 19 Aug 2019 16:06:34 -0400

On 8/19/19 11:47 AM, Thorsten Glaser wrote:
> Package: bash
> Version: 5.0-4
> Severity: critical
> Justification: breaks unrelated software
> 

> The expected output is:
> 
> dbc_mysql_createdb: _dbc_nodb(1)=	# initially not set / empty
> dbc_mysql_exec_command: _dbc_nodb=yes	# MUST be visible inside the function
> inner: 0 or 1				# MAY be exported, does not need to
> dbc_mysql_createdb: _dbc_nodb(2)=[yes]	# MAY be visible afterwards, optional
> after: 0 or 1				# if visible afterwards MAY be exported
> dbc_mysql_createdb:0
> 
> POSIX reference:
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
> specifically:
> 
>  * If the command name is a function that is not a standard utility
>    implemented as a function, variable assignments shall affect the
>    current execution environment during the execution of the function.
>    It is unspecified:
>    + Whether or not the variable assignments persist after the
>      completion of the function
>    + Whether or not the variables gain the export attribute during the
>      execution of the function
>    + Whether or not export attributes gained as a result of the variable
>      assignments persist after the completion of the function (if
>      variable assignments persist after the completion of the function)

This behavior hasn't changed substantially since bash-4.4: it still
implements the previous version of the Posix standard, which says that
assignment statements preceding functions are supposed to be treated like
special builtins.

$ cat x15b
# test whether or not temporary environment assignments are exported
# in posix mode
#set -o posix
showfoo()
{
        printf %s "foo=${foo-<unset>}"
        echo -n ' environment foo='
        printenv foo || echo
}
unset foo
showfoo
foo=foo showfoo
showfoo
$ ./bash -o posix ./x15b
foo=<unset> environment foo=
foo=foo environment foo=foo
foo=foo environment foo=foo
$ ../bash-5.0-patched/bash -o posix ./x15b
foo=<unset> environment foo=
foo=foo environment foo=foo
foo=foo environment foo=foo
$ ../bash-4.4-patched/bash -o posix ./x15b
foo=<unset> environment foo=
foo=foo environment foo=foo
foo=foo environment foo=foo

(The first test uses the current development version.)

I don't see substantial differences when I interpose another function in
between and call showfoo from that function with a preceding assignment
statement.

There is a problem with bash-5.0 when the variable is declared local in
the interposed function, and I need to fix that, but that's not a posix
issue because posix doesn't have anything to say about local variables.

So thanks for the report, and I'll look at the local variable issue.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

Message #27 received at 935115@bugs.debian.org (full text, mbox, reply):

From: Thorsten Glaser <tg@debian.org>

To: Chet Ramey <chet.ramey@case.edu>

Cc: 935115@bugs.debian.org

Subject: Re: Bug#935115: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX

Date: Mon, 19 Aug 2019 21:59:43 +0000 (UTC)

Hi Chet,

>There is a problem with bash-5.0 when the variable is declared local in
>the interposed function, and I need to fix that, but that's not a posix

ah, okay, I didn’t test what removing the “local” would do.

Debian is a special beast: it requires of a /bin/sh to behave
like a POSIX shell *plus* “local” and a couple other things
(echo -n, test -a/-o, kill -signal and trap with numbers) that
are reasonable. Currently, bash, dash and mksh fulfil these
requirements, as ksh93 is missing “local”; posh is too outdated
wrt mksh development and thus too buggy, and I don’t know how
well yash or *shudder* zsh would do as /bin/sh.

>issue because posix doesn't have anything to say about local variables.

It does not, agreed, but you probably realise that’s besides
the point here ;-)

>So thanks for the report, and I'll look at the local variable issue.

Thanks!

bye,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
	-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

Message #32 received at 935115@bugs.debian.org (full text, mbox, reply):

From: Usama Makhzoum <osmakh1@gmail.com>

To: 935115@bugs.debian.org

Subject: passing variable assignments to functions (was Re: Bug#935115: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX)

Date: Thu, 15 Oct 2020 21:22:54 +0300

looks like it has been fixed, i can't reproduce in either unstable, 
testing or stable by the moment of now.

Message #37 received at 935115-done@bugs.debian.org (full text, mbox, reply):

From: Ivo De Decker <ivodd@debian.org>

To: 935115-done@bugs.debian.org

Subject: Re: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX

Date: Sat, 6 Feb 2021 19:49:30 +0100

Version: 5.1-2

Hi,

It seems this is fixed in the current version of bash (I didn't check which
exact version introduced the fix).

On Mon, Aug 19, 2019 at 05:47:08PM +0200, Thorsten Glaser wrote:
> tglase@tglase:~ $ bash --version 2>&1 | head -1
> GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnux32)
> tglase@tglase:~ $ bash testscript
> dbc_mysql_createdb: _dbc_nodb(1)=
> dbc_mysql_exec_command: _dbc_nodb=yes
> inner: 1
> dbc_mysql_createdb: _dbc_nodb(2)=
> after: 0
> dbc_mysql_createdb:0
> tglase@tglase:~ $ bash --posix testscript
> dbc_mysql_createdb: _dbc_nodb(1)=
> dbc_mysql_exec_command: _dbc_nodb=
> inner: 1
> dbc_mysql_createdb: _dbc_nodb(2)=
> after: 1
> dbc_mysql_createdb:0

With bash 5.1-2:

(sid_amd64-dchroot)ivodd@barriere:~/src$ bash --version 2>&1 | head -1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
(sid_amd64-dchroot)ivodd@barriere:~/src$ bash testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0
dbc_mysql_createdb:0
(sid_amd64-dchroot)ivodd@barriere:~/src$ bash --posix testscript
dbc_mysql_createdb: _dbc_nodb(1)=
dbc_mysql_exec_command: _dbc_nodb=yes
inner: 1
dbc_mysql_createdb: _dbc_nodb(2)=
after: 0


Cheers,

Ivo

Send a report that this bug log contains spam.