Debian Bug report logs - #189332
flex: unput(char) not valid outside body of grammar

Display info messages

Message #5 received at maintonly@bugs.debian.org (full text, mbox, reply):

From: Peter Muir <iyhi@yahoo.com>

To: Debian Bug Tracking System <maintonly@bugs.debian.org>

Subject: flex: unput(char) not valid outside body of grammar

Date: Wed, 16 Apr 2003 16:55:13 -0400

Package: flex
Version: 2.5.31-3
Severity: minor
Tags: sid patch

PROBLEM

The patch provides a test case directory
`tests/test-unput-footer'. This problem also
occurs in the latest CVS version of bison.

cd /tmp/flex-2.5.31/tests/test-unput-footer/
make test
gcc -c -o scanner.o -I. -I. -I../.. -I../..  -O2 -Wall -D_REENTRANT -g scanner.c
scanner.l: In function `f':
scanner.l:47: `yytext_ptr' undeclared (first use in this function)
scanner.l:47: (Each undeclared identifier is reported only once
scanner.l:47: for each function it appears in.)
make: *** [scanner.o] Error 1

Compilation exited abnormally with code 2 at Wed Apr 16 16:03:23


CAUSE

flex undefines yytext_ptr at the end of the flex body.

>From scanner.c:
166-
167:#define unput(c) yyunput( c, (yytext_ptr)  )
168-
--
340-extern char *testtext;
341:#define yytext_ptr testtext
342-
--
1688-#undef yy_set_interactive
1689:#undef yytext_ptr
1690-#undef YY_DO_BEFORE_ACTION
1691-


SOLUTIONS

1. Fix documentation with something like:

   unput(char) is only valid within the
   body. All unput calls
   must be in the body. Putting back a string
   or the return value from a function
   call will require C macros. For example
   the following macro will put back a NUL
   terminated string.

   #define unput_string(s)\
     { const char *x, *y; x = (s); \
        if (x) { y=x+strlen(x); while (x < y) { y--; unput(*y); } } }

   ...
   .         {  unput_string("?"); }

2. Fix template.


PATCH TO PROVIDE TEST CASE

diff -r -u -N flex-2.5.31/configure.in flex-2.5.31.unput/configure.in
--- flex-2.5.31/configure.in	2003-04-01 11:17:51.000000000 -0500
+++ flex-2.5.31.unput/configure.in	2003-04-16 15:13:20.000000000 -0400
@@ -126,6 +126,7 @@
 tests/test-reject/Makefile
 tests/test-c++-multiple-scanners/Makefile
 tests/test-top/Makefile
+tests/test-unput-footer/Makefile
 dnl --new-test-here-- This line is processed by tests/create-test.
 )
 
diff -r -u -N flex-2.5.31/tests/Makefile.am flex-2.5.31.unput/tests/Makefile.am
--- flex-2.5.31/tests/Makefile.am	2003-03-27 13:02:59.000000000 -0500
+++ flex-2.5.31.unput/tests/Makefile.am	2003-04-16 14:59:11.000000000 -0400
@@ -26,6 +26,7 @@
 	create-test
 
 DIST_SUBDIRS = \
+	test-unput-footer \
 	test-basic-nr \
 	test-basic-r \
 	test-bison-yylloc \
@@ -67,6 +68,7 @@
 	test-table-opts
 
 SUBDIRS = \
+	test-unput-footer \
 	test-basic-nr \
 	test-basic-r \
 	test-bison-yylloc \
diff -r -u -N flex-2.5.31/tests/test-unput-footer/.cvsignore flex-2.5.31.unput/tests/test-unput-footer/.cvsignore
--- flex-2.5.31/tests/test-unput-footer/.cvsignore	1969-12-31 19:00:00.000000000 -0500
+++ flex-2.5.31.unput/tests/test-unput-footer/.cvsignore	2003-04-16 16:03:09.000000000 -0400
@@ -0,0 +1,7 @@
+Makefile
+Makefile.in
+OUTPUT
+parser.y
+scanner.c
+scanner.o
+test-unput-footer
diff -r -u -N flex-2.5.31/tests/test-unput-footer/Makefile.am flex-2.5.31.unput/tests/test-unput-footer/Makefile.am
--- flex-2.5.31/tests/test-unput-footer/Makefile.am	1969-12-31 19:00:00.000000000 -0500
+++ flex-2.5.31.unput/tests/test-unput-footer/Makefile.am	2003-04-16 14:58:50.000000000 -0400
@@ -0,0 +1,50 @@
+# This file is part of flex.
+
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+
+# Neither the name of the University nor the names of its contributors
+# may be used to endorse or promote products derived from this software
+# without specific prior written permission.
+
+# THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE.
+
+FLEX = $(top_builddir)/flex
+
+builddir = @builddir@
+
+EXTRA_DIST = scanner.l parser.y test.input
+CLEANFILES = scanner.c scanner.h parser.c parser.h test-unput-footer OUTPUT $(OBJS)
+OBJS = scanner.o # parser.o
+
+AM_CPPFLAGS = -I$(srcdir) -I$(builddir) -I$(top_srcdir) -I$(top_builddir)
+#LDFLAGS = $(top_srcdir)/libfl.a 
+#LFLAGS = --header="scanner.h"
+#YFLAGS = --defines --output=parser.c
+
+testname = test-unput-footer
+
+scanner.c: $(srcdir)/scanner.l
+	$(FLEX) $(LFLAGS) $<
+
+parser.c: $(srcdir)/parser.y
+	$(BISON) $(YFLAGS) $<
+
+$(testname)$(EXEEXT): $(OBJS)
+	$(CC) -o $@ $(LDFLAGS) $(OBJS) $(LOADLIBES)
+
+test: $(testname)$(EXEEXT)
+	./$(testname)$(EXEEXT) < $(srcdir)/test.input
+
+.c.o:
+	$(CC) -c -o $@ $(AM_CPPFLAGS) $(CPPFLAGS) $(CFLAGS) $<
diff -r -u -N flex-2.5.31/tests/test-unput-footer/scanner.l flex-2.5.31.unput/tests/test-unput-footer/scanner.l
--- flex-2.5.31/tests/test-unput-footer/scanner.l	1969-12-31 19:00:00.000000000 -0500
+++ flex-2.5.31.unput/tests/test-unput-footer/scanner.l	2003-04-16 16:01:52.000000000 -0400
@@ -0,0 +1,58 @@
+/*
+ * This file is part of flex.
+ * 
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ * 
+ * Neither the name of the University nor the names of its contributors
+ * may be used to endorse or promote products derived from this software
+ * without specific prior written permission.
+ * 
+ * THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
+ * IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE.
+ */
+
+%{
+/* A template scanner file to build "scanner.c". */
+#include <stdio.h>
+#include <stdlib.h>
+#include "config.h"
+/*#include "parser.h" */
+
+static void f(const char *x);
+
+%}
+
+%option 8bit outfile="scanner.c" prefix="test"
+%option nomain noyywrap 
+%option warn
+
+%%
+
+[0-9]              { unput('?'); }
+[a-z]              { f("?"); }
+.
+%%
+
+void f(const char *x) { unput(*x); }
+
+int main(void);
+
+int
+main ()
+{
+    yyin = stdin;
+    yyout = stdout;
+    yylex();
+    printf("TEST RETURNING OK.\n");
+    return 0;
+}
diff -r -u -N flex-2.5.31/tests/test-unput-footer/test.input flex-2.5.31.unput/tests/test-unput-footer/test.input
--- flex-2.5.31/tests/test-unput-footer/test.input	1969-12-31 19:00:00.000000000 -0500
+++ flex-2.5.31.unput/tests/test-unput-footer/test.input	2003-04-16 14:58:50.000000000 -0400
@@ -0,0 +1,2 @@
+0000 foo 1111 foo 0000 bar
+0000 foo 1111 foo 0000 bar


-- System Information:
Debian Release: testing/unstable
Architecture: i386
Kernel: Linux paul 2.5.66 #2 Tue Mar 25 19:23:50 EST 2003 i686
Locale: LANG=en_US, LC_CTYPE=en_US

Versions of packages flex depends on:
ii  libc6                         2.3.1-16   GNU C Library: Shared libraries an
ii  m4                            1.4-15     a macro processing language

-- no debconf information

Message #10 received at 189332-done@bugs.debian.org (full text, mbox, reply):

From: Manoj Srivastava <srivasta@debian.org>

To: Peter Muir <iyhi@yahoo.com>

Cc: 189332-done@bugs.debian.org

Subject: Re: Bug#189332: flex: unput(char) not valid outside body of grammar

Date: Wed, 16 Apr 2003 17:40:13 -0500

>> On Wed, 16 Apr 2003 16:55:13 -0400,
>> Peter Muir <iyhi@yahoo.com> said: 

 > PROBLEM
   ...
 > CAUSE

 > flex undefines yytext_ptr at the end of the flex body.


	This is a new feature of flex. It cleans up after it self;
 there are a number of assets present in the rules section that
 are not present outside it. This is helpful since flex nmow no longer
 pollutes the user namespace as much, and helps in allowing for
 multiple scanners, with potentialy different options, in the same
 program. 

	Please consider the following modification to the flex file
 you submitted in th patch; there are no macros used, just the
 function definitions are, umm, relocated.

	Please note that the documentation already documents unput as
 a special directive available only in actions; there is never any
 representation that these special directives are available outside
 actions. 

	manoj

----------------------------------------------------------------------
/*
 * This file is part of flex.
 * 
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * 
 * Neither the name of the University nor the names of its contributors
 * may be used to endorse or promote products derived from this software
 * without specific prior written permission.
 * 
 * THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
 * IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
 * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE.
 */

%{
/* A template scanner file to build "scanner.c". */
#include <stdio.h>
#include <stdlib.h>
#include "config.h"
/*#include "parser.h" */

static void f(const char *x);

%}

%option 8bit outfile="scanner.c" prefix="test"
%option nomain noyywrap 
%option warn

%%
%{
 void f(const char *x) { unput(*x); }
%}

[0-9]              { unput('?'); }
[a-z]              { f("?"); }
.
%%


int main(void);

int
main ()
{
    yyin = stdin;
    yyout = stdout;
    yylex();
    printf("TEST RETURNING OK.\n");
    return 0;
}
----------------------------------------------------------------------


-- 
Date: 1 Jul 90 22:55:10 GMT From: merlyn@iwarp.intel.com (Randal
Schwartz) $x=25;print substr(',rekcah lreP rehtona tsuJ',$x,1) while
--$x >= 0
Manoj Srivastava   <srivasta@debian.org>  <http://www.debian.org/%7Esrivasta/>
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Message #15 received at 189332@bugs.debian.org (full text, mbox, reply):

From: Paul Eggert <eggert@CS.UCLA.EDU>

To: 189332@bugs.debian.org

Cc: lex-help@lists.sourceforge.net, Peter Muir <iyhi@yahoo.com>, bug-bison@gnu.org

Subject: flex: unput(char) not valid outside body of grammar

Date: Fri, 18 Apr 2003 00:50:08 -0700 (PDT)

This email is following up to a Debian bug report about flex
<http://bugs.debian.org/189332>, which reports that flex test version
2.5.31 breaks Bison 1.875.

> From: Manoj Srivastava <srivasta@debian.org>
> Date: Date: Wed, 16 Apr 2003 17:40:13 -0500

> This is a new feature of flex. It cleans up after it self; there are
> a number of assets present in the rules section that are not present
> outside it. This is helpful since flex nmow no longer pollutes the
> user namespace as much, and helps in allowing for multiple scanners,
> with potentialy different options, in the same program.

This change means that flex no longer conforms to POSIX 1003.1-2001.
The POSIX specification for lex
<http://www.opengroup.org/onlinepubs/007904975/utilities/lex.html>
says that the functions yylex, yymore, yyless, input, and unput are
all accessible to user code included in the lex input.  As far as I
can tell, POSIX imposes no restriction that the functions are
accessible only to user actions.

The change to flex may be necessary for reentrant scanners, but it
shouldn't be necessary for traditional scanners such as those
specified by POSIX.


> Please consider the following modification to the flex file
> you submitted in th patch

I tried doing something like that for Bison (along the lines suggested
by Peter Muir in
<http://mail.gnu.org/archive/html/bison-patches/2003-04/msg00002.html>),
but this had other problems: it called yyunput before it was declared,
and this violates C99.

I shall rework Bison so that it doesn't use unput at all.  This will
avoid the portability hassles, at the cost of making the Bison scanner
a bit less elegant.  However, I suggest that the flex maintainers fix
the incompatibility with POSIX, as people will be using flex to build
older versions of Bison for quite some time.  Also, people will be
using flex to build other scanners that may assume the POSIX-required
behavior.

Message #20 received at 189332@bugs.debian.org (full text, mbox, reply):

From: Manoj Srivastava <srivasta@debian.org>

To: Paul Eggert <eggert@CS.UCLA.EDU>

Cc: 189332@bugs.debian.org, lex-help@lists.sourceforge.net, Peter Muir <iyhi@yahoo.com>, bug-bison@gnu.org, debian-devel@lists.debian.org

Subject: Re: Bug#189332: flex: unput(char) not valid outside body of grammar

Date: Fri, 18 Apr 2003 04:10:55 -0500

>> On Fri, 18 Apr 2003 00:50:08 -0700 (PDT),
>> Paul Eggert <eggert@CS.UCLA.EDU> said: 

 > This email is following up to a Debian bug report about flex
 >> http://bugs.debian.org/189332>, which reports that flex test
 >> http://bugs.debian.org/189332>version
 > 2.5.31 breaks Bison 1.875.

 >> From: Manoj Srivastava <srivasta@debian.org> Date: Date: Wed, 16
 >> Apr 2003 17:40:13 -0500

 > This change means that flex no longer conforms to POSIX
 > 1003.1-2001.  The POSIX specification for lex
 >> http://www.opengroup.org/onlinepubs/007904975/utilities/lex.html>
 > says that the functions yylex, yymore, yyless, input, and unput are
 > all accessible to user code included in the lex input.  As far as I
 > can tell, POSIX imposes no restriction that the functions are
 > accessible only to user actions.

 > The change to flex may be necessary for reentrant scanners, but it
 > shouldn't be necessary for traditional scanners such as those
 > specified by POSIX.

	Indeed. The full POSIX references are:
 ISO/IEX 9945-2: 1993(3)   Information Technology -- POSIX
 IEEE Std 1003.2-1992      Part 2: Shell and Utilities
  A.2.7.5 lex Actions

	It also says that it is unspecified whether the functions or
 macros appear in the C code output of lex, or are accessible only
 through the -l l operand og the c compiler. 


	I have, however, noticed something bizarre: adding a %nounput
 options defines unput, but never undefines it, this seems like a bug
 somewhere. 
----------------------------------------------------------------------
%{
/* A template scanner file to build "scanner.c". */
#include <stdio.h>
#include <stdlib.h>
#include "config.h"
/*#include "parser.h" */

static void f(const char *x);

%}

%option 8bit outfile="scanner.c" prefix="test"
%option nomain noyywrap nounput
%option warn

%%
%{
 void f(const char *x) { unput(*x); }
%}

[0-9]              { unput('?'); }
[a-z]              { f("?"); }
.
%%

int main(void);

int
main ()
{
    yyin = stdin;
    yyout = stdout;
    yylex();
    printf("TEST RETURNING OK.\n");
    return 0;
}
----------------------------------------------------------------------
 

>> Please consider the following modification to the flex file
 >> you submitted in th patch

 > I tried doing something like that for Bison (along the lines suggested
 > by Peter Muir in
 > <http://mail.gnu.org/archive/html/bison-patches/2003-04/msg00002.html>),
 > but this had other problems: it called yyunput before it was declared,
 > and this violates C99.


	Umm, the modification I provided was tested, and it did not
 call yyunput before it was declared (the trick is to place the
 functions in the _rules_ section in %{ %} stanzas. 

	manoj
-- 
God gave you that gifted tongue of yours, and set it between your
teeth, to make known your true meaning to us, not be rattled like a
muffin man's bell. Carlyle

Manoj Srivastava   <srivasta@debian.org>  <http://www.debian.org/%7Esrivasta/>
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Message #25 received at 189332@bugs.debian.org (full text, mbox, reply):

From: Paul Eggert <eggert@CS.UCLA.EDU>

To: 189332@bugs.debian.org

Cc: lex-help@lists.sourceforge.net, Peter Muir <iyhi@yahoo.com>, bug-bison@gnu.org, debian-devel@lists.debian.org

Subject: Re: Bug#189332: flex: unput(char) not valid outside body of grammar

Date: 18 Apr 2003 03:21:25 -0700

Manoj Srivastava <srivasta@debian.org> writes:

>  [POSIX] also says that it is unspecified whether the functions or
>  macros appear in the C code output of lex, or are accessible only
>  through the -l l operand of the c compiler. 

Yes.  This means that if Bison were trying to be portable to all lex
variants, Bison would have to link itself with -ll for the lex
variants that define unput etc. in the lex library.  Bison assumes
flex, though, so it doesn't have to worry about that possibility.

One other problem I've observed is that flex 2.5.31 defines
yylex_destroy even if the application doesn't need yylex_destroy.
There is no %option noyylex_destroy.


>  the modification I provided was tested, and it did not
>  call yyunput before it was declared (the trick is to place the
>  functions in the _rules_ section in %{ %} stanzas. 

That modification relies on nested functions, which is not portable.
Bison is supposed to be portable to any ANSI C compiler, so it can't
use that technique.

Send a report that this bug log contains spam.