Debian Bug report logs - #748962
texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Zephyrus C <zephyrus8080@gmail.com>

To: "To: Debian Bug Tracking System" <submit@bugs.debian.org>

Cc: Norbert Preining <preining@logic.at>, Danai Sae-Han <danai.sae-han@edpnet.be>, Zephyrus C <zephyrus8080@gmail.com>

Subject: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 23 May 2014 02:01:40 +0900

[Message part 1 (text/plain, inline)]

Package: texlive-base
Version: 2013.20140408-1
Severity: important
Tags: d-i upstream patch

Dear Maintainer,


   * What led up to the situation?
     There was a strange failure of tex installation that was likely to be
caused by low free space condition.
It was not clear what failure could lead to such strange condition.

   * What exactly did you do (or not do) that was effective (or
     ineffective)?
     I tried to figure out which command failed and at the same time failed
to
    return proper exit value to the invoking programs.

   * What was the outcome of this action?
      I am attaching a patch to fix a slightly buggy function and
      tighten up checks in fmtutil.sh

   * What outcome did you expect instead?

     I think the installer will catch more errors due to low
free-space conditions during the installation with the patch, and this
should save support man-hours later.

Description

"fmtutil-sys --all" may not return error exit code under low free space
condition.

This submission is an extension to 748271, but now with a much more
sharper focus.
Bug#748271: Info received (Bug#748271: [latex-cjk-common] latex command did
not
work until I ran it once under superuser!?)

Thank you for the help so far regarding 748271.

Now, I *may* have identified WHY the error condition (under low space
condition) was not propagated to debian aptitude/apt-get installation
command under a very rage circumstance, and that could be the reason
why the incorrect TeX* installation was not caught earlier in my previous
case.
(It may not be the root cause, but still the problem noticed in this bug
report is
valid and can be fixed by the attached patch.)

I found that fmtutil.sh has a slightly buggy function.
(fmtutil.sh is part of texlive-base and is called eventually when
fmtutil-sys --all is called during installation to create .fmt files.)

I am attaching a patch to re-define the problematic function and to
replace log_warning with log_failure to signal the calling program
that a failure occurred.

Short Summary:

There is one shell-function "verbose()" which is used to invoke a
passed command and redirect stderr based on the exit value of
$mktexfmtMode (which is either "true" or "false").

Here is the definition of  buggy verbose()
from fmtutil.sh ( linked from /usr/bin/fmtutil to
/usr/share/texlive/texmf-dist/scripts/texlive/fmtutil.sh)

###############################################################################
# verbose (cmd)
#   execute cmd. Redirect output depending on $mktexfmtMode.
###############################################################################
verbose()
{
  $mktexfmtMode && ${1+"$@"} >&2 || ${1+"$@"}
}


This function as it is written today is buggy in the sense that
if the command fails when $mktexfmtMode is "true", the failing command
is executed TWICE (!). The first execution is with the rediretion of
STDERR. The second execution is without such redirection.
I think this is quite unintentional.

Also, the exit code from this execution of
"verbose other command options ..." becomes,
when the command fails on the LEFT Hand Side (LHS) of "||",
it is the exit code of the executed command (without re-direction) on
the right-hand side (RHS)

Under very space-tight conditions of a local file system, the failure
may happen while the stderr is re-directed (LHS), but a failure may not
happen when the command is executed as is without re-direction (RHS).

If this rare but possible scenario happens, "set -e" near the top of
fmtutil.sh is useless to catch the error since "verbose ...." 'as a whole'
returns the successful exit code of non-redirected command on the RHS
of "||".

MORE DETAILS:

This slightly buggy function is called within run_initex
(yes, that is where the fmt files are created eventually!)

This "verbose" function should be rewritten to avoid the re-execution
of the command "${1+"$@"}" when it fails when $mktexfmtMode is true.

Correction-1:

verbose()
{
    if $mktexfmtMode
    then
         ${1+"$@"} >&2
    else
        ${1+"$@"}
    fi
}

Background/Detective Work

After bug 748271 was closed, I tried to test what could go wrong a little
further.
I downloaded TeXLive 2013 source file from TUG web site.
I untarred it and began checking fmtutil-sys and found that these end up
as links to fmtutil and that is /usr/bin/fmtutil and again is a symlink
to /usr/share/texlive/texmf-dist/scripts/texlive/fmtutil.sh in Debian.
I tried to check what could go wrong and yet remain undetected, thus
failing to stop apt-get/aptitude as unsuccessful.

First thing, first.
As Norbert noted that there is "set -e" near the installed
fmtutil.sh. Great!
Surprisingly, this "set -e" is missing from the stock TeXLive 2013
source from TUG?! So maybe this is Debian only now?  I hope the upstream
TeXLive 2014 is fixed to have this "set -e" in fmtutil.sh

But then what can possibly go wrong?  My bet was that some commands
that are called are not returning the exit error code correctly.  I
traced what would happen when "--all" is passed to fmtutil, and found
that the user-defined functions are called in this manner.

main
 ->
  recreate_all
  ->
   recreate_loop
   ->
    run_initex
    ->
     verbose

In run_initex, the following piece of code is where the meat of
processing takes place.

--- quote
  verboseMsg "$progname: running \`$engine -ini  $tcxflag $jobswitch
$prgswitch
$texargs' ..."

  # run in a subshell to get a local effect of TEXPOOL manipulation:
  (
    # If necessary, set TEXPOOL. Use absolute path, because of KPSE_DOT.
    $localpool && { TEXPOOL="`pwd`:$TEXPOOL"; export TEXPOOL; }
    verbose $engine -ini $tcxflag $jobswitch $prgswitch $texargs
  ) </dev/null
--- end quote

I realized that "verboseMsg" is to print out what is going on to the
log.
and somehow user-defined "verbose" command was used to invoke the
command, "$engine -ini $tcxflag $jobswitch $prgswitch $texargs".

Now, I checked what could go wrong with this small code snippet
by creating a very simplified script as follows.

I traced the function of verbose by running failing command (in this
case, "cp /dev/null /etc" that should fail under ordinary user) and
tested the operation in the following manner.

# /tmp/t-buggy.sh
---- begin quote of /tmp/t-buggy.sh
#!/bin/sh
#

# This script is to show that a failure of
# a command executed in verbose may be executed twice.

# false is not a good failure command sample

set -e

rc=0

verbose()  # original. This is buggy!
{
  $mktexfmtMode && ${1+"$@"} >&2 || ${1+"$@"}
}

#
run_initex()
{
  # run in a subshell to see if this changes anything.
  (
      echo "run_initex called"
      # DO SOMETHING THAT SHOULD FAIL
      cp /dev/null /etc
  ) </dev/null
}

# should succeed
verbose echo "hello, world"
echo "rc = $?"

echo before cp /dev/null /etc
# should fail
verbose cp /dev/null /etc
echo "rc = $?"

echo before subshell execution
# should fail
# run in a subshell ...
  (
    verbose cp /dev/null /etc
  ) </dev/null
echo "rc = $?"

echo before run_initex

mktexfmtMode=true
rc=1
# should fail since run_initex invokes false inside.
verbose run_initex $rc
echo "rc = $?"

echo before cp /dev/null /etc
# should fail under ordinary user account.
verbose cp /dev/null /etc
echo "rc = $?"

exit 0
--- end quote ---

The above is the content of the script.
Now let us run it.

$ /tmp/t-buggy.sh || echo failure
hello, world
rc = 0
before cp /dev/null /etc
cp: cannot create regular file '/etc/null': Permission denied
cp: cannot create regular file '/etc/null': Permission denied
failure
$

Please notice the repeated execution of the failed command by "verbose()".
That is because when the $mktexfmtMode is true the passed command is
executed with redirection.
In this case, the passed command "cp /dev/null /etc" should fail for
ordinary user.
But it is then  reexecited because of ||. Shell figured that Left-Hand
Side (LHS) failed and so tried to execute Right-Hand Side (RHS), too.

Now, when I rewrite verbose() in the suggested manner, what happens?
With fixed verbose (as in correction-1 above):

$ /tmp/t-OK.sh || echo fail
hello, world
rc = 0
before cp /dev/null /etc
cp: cannot create regular file '/etc/null': Permission denied
fail
$

Great. It is executed only once and I think this is the intended behavior.

BTW, suppose the intention of "verbose" was to capture whatever error
message (to file descriptor 2) to a file associated with STDOU and
later check error condition by searching for a certain string pattern, we
may encounter a subtle problem *IF and ONLY IF* the command fails during an
almost full file system condition when stderr is redirected to standard
output, AND IF it succeeds without such redirection.

(If the file system was almost full, then the LHS of the original "||"
in "verbose" may have failed due to redirection, but somehow RHS may
have succeeded since there was no re-direction of error message to
almost full system. This may cause a very perplexing situation, indeed.)

Near line 765, we have the following code snippet in fmtutil.sh:

--- quote
  if test -f "$fmtfile"; then
    grep '^! ' $format.log >/dev/null 2>&1 &&
      log_warning "\`$engine -ini $tcxflag $jobswitch $prgswitch $texargs'
possibly failed."
--- end quote

Like I said, format.log may have failed to capture the error message
under a very space-tight condition in "verbose()".
I would love to see the "log_warning" to be changed to a
failing "log_failure" instead. (correction-2)

There is another usage of "log_warning" near line 796.

--- quote
          if cp "$destfile" "$mplib_mem_file" </dev/null; then
            mktexupd "$fulldestdir" "$mplib_mem_name"
          else
            log_warning "cp $destfile $mplib_mem_file failed."
          fi
--- end quote

The warning is indeed produced when the file system is full and "cp"
can fail then!  So, for obvious reasons, I would like to see this
usage of "log_warning" changed into "log_failure", too. (cprrectopm-3)

To wrap up, I am attaching a patch.
Please find below the diff of modified fmtutil.sh

fmtutil is actually /usr/share/texlive/texmf-dist/scripts/texlive/fmtutil.sh
diff of modified fmtutil in /tmp/fmtutil.sh
and
/usr/bin/fmtutil which is a symlink to the script under
/usr/share/texlive/...
was produced and attached.

Thank you in advance for your attention.

PS: This bug report was created by Debian's reportbug under a PC (32-bit
Debian GNU/Linux)
and is not the same PC where the previous bug was reported, but
the shell script should be the same on 32-bit and 64-bit, and the script
certainly
has a subtle problem.



-- Package-specific info:
IMPORTANT INFORMATION: We will only consider bug reports concerning
the packaging of TeX Live as relevant. If you have problems with
combination of packages in a LaTeX document, please consult your
local TeX User Group, the comp.text.tex user group, the author of
the original .sty file, or any other help resource.

In particular, bugs that are related to up-upstream, i.e., neither
Debian nor TeX Live (upstream), but the original package authors,
will be closed immediately.

   *** The Debian TeX Team is *not* a LaTeX Help Desk ***

If you report an error when running one of the TeX-related binaries
(latex, pdftex, metafont,...), or if the bug is related to bad or wrong
output, please include a MINIMAL example input file that produces the
error in your report.

Please run your example with
    (pdf)latex -recorder ...
(or any other program that supports -recorder) and send us the generated
file with the extension .fls, it lists all the files loaded during
the run and can easily explain problems induced by outdated files in
your home directory.

Don't forget to also include minimal examples of other files that are
needed, e.g. bibtex databases. Often it also helps
to include the logfile. Please, never send included pictures!

If your example file isn't short or produces more than one page of
output (except when multiple pages are needed to show the problem),
you can probably minimize it further. Instructions on how to do that
can be found at

http://www.minimalbeispiel.de/mini-en.html (english)

or

http://www.minimalbeispiel.de/mini.html (german)

##################################
minimal input file


##################################
other files

######################################
 List of ls-R files

-rw-r--r-- 1 root root 1047 May 13 17:52 /var/lib/texmf/ls-R
-rw-rw-r-- 1 root staff 80 May 13 16:05 /usr/local/share/texmf/ls-R
lrwxrwxrwx 1 root root 29 Jun 15  2013 /usr/share/texmf/ls-R ->
/var/lib/texmf/ls-R-TEXMFMAIN
lrwxrwxrwx 1 root root 31 Apr  8 11:51 /usr/share/texlive/texmf-dist/ls-R
-> /var/lib/texmf/ls-R-TEXLIVEDIST
lrwxrwxrwx 1 root root 31 Apr  8 11:51 /usr/share/texlive/texmf-dist/ls-R
-> /var/lib/texmf/ls-R-TEXLIVEDIST
######################################
 Config files
-rw-r--r-- 1 root root 838 May 13 17:50 /etc/texmf/web2c/texmf.cnf
-rw-r--r-- 1 root root 4589 May 13 17:52 /var/lib/texmf/web2c/fmtutil.cnf
lrwxrwxrwx 1 root root 32 Apr  8 11:51 /usr/share/texmf/web2c/updmap.cfg ->
/var/lib/texmf/updmap.cfg-DEBIAN
-rw-r--r-- 1 root root 3497 May 13 17:52
/var/lib/texmf/tex/generic/config/language.dat
######################################
 Files in /etc/texmf/web2c/
total 2
-rw-r--r-- 1 root root 283 Nov 10  2008 mktex.cnf
-rw-r--r-- 1 root root 838 May 13 17:50 texmf.cnf
######################################
 md5sums of texmf.d
ca40c66f144b4bafc3e59a2dd32ecb9c  /etc/texmf/texmf.d/00debian.cnf
d41d8cd98f00b204e9800998ecf8427e  /etc/texmf/texmf.d/20ptexjtex.bak
1df66bc319cec731e202eaf39f5d85e1  /etc/texmf/texmf.d/96JadeTeX.cnf

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 3.2.0-4-686-pae (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to
C)
Shell: /bin/sh linked to /bin/dash

Versions of packages texlive-base depends on:
ii  debconf [debconf-2.0]  1.5.52
ii  dpkg                   1.17.6
ii  libpaper-utils         1.1.24+nmu3
ii  luatex                 0.76.0-3
ii  tex-common             4.04
ii  texlive-binaries       2013.20130729.30972-2+b3
ii  ucf                    3.0027+nmu1
ii  xdg-utils              1.1.0~rc1+git20111210-7

Versions of packages texlive-base recommends:
ii  lmodern  2.004.4-3

Versions of packages texlive-base suggests:
ii  evince [postscript-viewer]       3.4.0-3.1+b1
ii  ghostscript [postscript-viewer]  9.05~dfsg-8+b1
pn  perl-tk                          <none>
pn  xpdf-reader | pdf-viewer         <none>

Versions of packages tex-common depends on:
ii  debconf [debconf-2.0]  1.5.52
ii  dpkg                   1.17.6
ii  ucf                    3.0027+nmu1

Versions of packages tex-common suggests:
ii  debhelper  9.20140228

Versions of packages texlive-base is related to:
ii  tex-common        4.04
ii  texlive-binaries  2013.20130729.30972-2+b3

-- debconf information:
  texlive-base/texconfig_ignorant:
  tex-common/check_texmf_wrong:
  texlive-base/binary_chooser: pdftex, dvips, dvipdfmx, xdvi
  tex-common/check_texmf_missing:

[Message part 2 (text/html, inline)]

[report-diff-patch.diff (text/plain, attachment)]

Message #10 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <preining@logic.at>

To: Zephyrus C <zephyrus8080@gmail.com>, 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 23 May 2014 08:24:08 +0900

Hi

thanks for your bug report and analysis, but could you please
send this to
	tex-live@tug.org
Of course I can forward it, but then the discussion would probably
be held between me and upstream TeX Live, and there will be 
quite some discussion. You can drop the whole Debian specific
part (if you want).

Some remarks only on the patches:
>  verbose()
>  {
> -  $mktexfmtMode && ${1+"$@"} >&2 || ${1+"$@"}
> +  if $mktexfmtMode
> +  then
> +    ${1+"$@"} >&2;
> +  else
> +    ${1+"$@"};
> +  fi
>  }

I see your point here, but I am far from convinced. Thomas Esser wrote
this part *long*long* time ago, and he is known for excellent
programming work. He surely had an intention behind this code,
otherwise he would not write it like that.
               

>    if test -f "$fmtfile"; then
>      grep '^! ' $format.log >/dev/null 2>&1 &&
> -      log_warning "\`$engine -ini $tcxflag $jobswitch $prgswitch $texargs' possibly failed."
> +      log_failure "\`$engine -ini $tcxflag $jobswitch $prgswitch $texargs' possibly failed."

I am not sure - but probably this makes sense in Debian, but not
anywhere else. The problem is we don't want fmtutil to stop
but continue building other formats.

Imagine root calling
	fmtutil-sys --all
to rebuild all formats, and one is failing. Then we don't want to
stop there, but continue building, right?

Same with the other log_warning/_failure change


Norbert

------------------------------------------------------------------------
PREINING, Norbert                               http://www.preining.info
JAIST, Japan                                 TeX Live & Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Message #15 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Zephyrus C <zephyrus8080@gmail.com>

To: Norbert Preining <preining@logic.at>

Cc: 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 23 May 2014 14:24:45 +0900

[Message part 1 (text/plain, inline)]

Hi,

Thank you for your comment and all the help.
I will bring the discussion to TeX Live mailing list then.

A few comments below.

On Fri, May 23, 2014 at 8:24 AM, Norbert Preining <preining@logic.at> wrote:

> Hi
>
> thanks for your bug report and analysis, but could you please
> send this to
>         tex-live@tug.org
> Of course I can forward it, but then the discussion would probably
> be held between me and upstream TeX Live, and there will be
> quite some discussion. You can drop the whole Debian specific
> part (if you want).
>
> Some remarks only on the patches:
> >  verbose()
> >  {
> > -  $mktexfmtMode && ${1+"$@"} >&2 || ${1+"$@"}
> > +  if $mktexfmtMode
> > +  then
> > +    ${1+"$@"} >&2;
> > +  else
> > +    ${1+"$@"};
> > +  fi
> >  }
>
> I see your point here, but I am far from convinced. Thomas Esser wrote
> this part *long*long* time ago, and he is known for excellent
> programming work. He surely had an intention behind this code,
> otherwise he would not write it like that.
>
> It is possible then Thomas Esser wrote this way so that
the command is executed TWICE explicitly in the face of
failure. The command is idempotent (in the sense that it can be run twice
without
ill-effect) and so should be OK, and that he may have surmised that the
second
execution would print error message to console (unlike the re-directed
execution on LHS of "||").

But I would prefer that this be explicitly "documented" for maintenance
reason.
In that sense, my mundane rewrite may need a  suitable modification for the
extra second-time execution which Thomas Esser may have intended.

>    if test -f "$fmtfile"; then
> >      grep '^! ' $format.log >/dev/null 2>&1 &&
> > -      log_warning "\`$engine -ini $tcxflag $jobswitch $prgswitch
> $texargs' possibly failed."
> > +      log_failure "\`$engine -ini $tcxflag $jobswitch $prgswitch
> $texargs' possibly failed."
>
> I am not sure - but probably this makes sense in Debian, but not
> anywhere else. The problem is we don't want fmtutil to stop
> but continue building other formats.
>
> Imagine root calling
>         fmtutil-sys --all
> to rebuild all formats, and one is failing. Then we don't want to
> stop there, but continue building, right?
>
>
You are right. My point here was that the script seems to "forget" that the
error was there during the re-production of the whole set of .fmt files, and
did not return error code (my casual reading. Maybe I am wrong.) at the end.
I think it should.

I will propose a rewrite that log_warning would keep the error code and
return it at the end. (Maybe it did this way already, but it doesn't look
so after my casual reading.)


> Same with the other log_warning/_failure change
>
> Norbert
>
>
Thank you again. I noticed by reading tex live mailing list that
you and a few others are at the core of maintaining tex live.
Thank you for making the package so that a user like me who don't care much
about the intricate innards of these software packages can use on
Debian and elsewhere.

(Yeah, I even tried to install OLD ASCII ptex on my own after encountering
the strange error on my PC. It was an uphill and hilarious battle. At one
point, a certain date test in tex macro refused to proceed saying that
I was trying to install a file that was more than 5 years old and I should
get a
newer installation. Oh well. So, I lied by modifying the date in a style
file.
Then a patch file could not be applied since the time stamp did not match.
So I lied again by modifying the timestamp in the patch. Then it was
compiled, etc, and ran. And I still got a run-time warning about using an
outdated software. Good and bad.

I wonder what people would do when old TeX files that were once typeset
using the vertical writing feature supported by ASCII's ptex needed to be
modified afresh and typeset again.

We may need a time machine (of a sort) where the old typeset software is
kept intact so that in the future the old documents that need modification
can be typeset again on such a machine. Otherwise, Knuth's original idea of
being able to reproduce the same document even after a dozen year or so is
a pipe dream under the current situation. But I digress.

TIA and expect my post in tex-live mailing list soon.



> ------------------------------------------------------------------------
> PREINING, Norbert                               http://www.preining.info
> JAIST, Japan                                 TeX Live & Debian Developer
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
> ------------------------------------------------------------------------
>

[Message part 2 (text/html, inline)]

Message #20 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <preining@logic.at>

To: Zephyrus C <zephyrus8080@gmail.com>

Cc: 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 23 May 2014 15:01:06 +0900

Hi,

> second
> execution would print error message to console (unlike the re-directed
> execution on LHS of "||").

Yes, that was quite surely his intention ;-) But you can also try to
ask him ;-)

Anyway, rewriting fmtutil in perl is on my personal todo list.
I have already rewritten (based on previous work) updmap in perl,
and at the end I want to get rid of the separation c-code for Windpws
and shell code for the rest of fmtutil, by replacing it with all
perl code.

At that point, my aim is also to have a decent error handling ;-)

> You are right. My point here was that the script seems to "forget" that the
> error was there during the re-production of the whole set of .fmt files, and
> did not return error code (my casual reading. Maybe I am wrong.) at the end.
> I think it should.

Actually, that is a known problem. fmtutil does not do *any* proper
error handling. We are well aware of that, one other reason why I want
to rewrite it in perl ;-)

> Thank you again. I noticed by reading tex live mailing list that
> you and a few others are at the core of maintaining tex live.

Indeed, indeed. Although I also maintain the Debian packages, my
main part is in upstream (now, not originally).

> (Yeah, I even tried to install OLD ASCII ptex on my own after encountering

Oh, bad bad idea. There is Okumura-sensei's TeX Q-A and there was
ptetex and ptexlive, but I incorporated all the pTeX stuff and
support packages over the last years, with lots of support from
many of the Japanese TeX community, into TeX Live. Before it was
a drag to get pTeX (upTeX etc) running.

> I wonder what people would do when old TeX files that were once typeset
> using the vertical writing feature supported by ASCII's ptex needed to be
> modified afresh and typeset again.

That is not a problem. You are speaking about the *format*.
The format can be regenerated. There has been a problem 6 years
ago when the released version we had in Debian just fell out
of these 5 years, and we had to patch the latex.ltx, but that
has no influence. You can still run old code and get the same
output (unless you use many packages that have changed
considerably - if you *reall* want 100% reproducibility in the
future, you have to stay with *plain* ptex and not use latex
and add-on packages!).

> We may need a time machine (of a sort) where the old typeset software is
> kept intact so that in the future the old documents that need modification

We have mirrors of old TeX Live's, so you can go back at least with
TeX Live.


A, one more remark: TeX Live is in the hot phase of pre-release,
so don't expect toooo much activity and response *now* on fmtutil
requests, as everyone of (the few of us) is busy for the final
image burning etc etc.

Thanks for all your work on fmtutil, much appreciated.

Norbert

------------------------------------------------------------------------
PREINING, Norbert                               http://www.preining.info
JAIST, Japan                                 TeX Live & Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Message #25 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Zephyrus C <zephyrus8080@gmail.com>

To: Norbert Preining <preining@logic.at>

Cc: 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 30 May 2014 01:15:36 +0900

[Message part 1 (text/plain, inline)]

Hi,

This is going to be my last post here.

I posted a very lengthy post to tex-live mailing list to
follow up on my finding here.
http://tug.org/pipermail/tex-live/2014-May/thread.html

I posted a proper patch there. It is near the end since it was posted
a couple of days ago.

Good news is that I tested the modified fmtutil.sh under two Debian
linux systems by making "/var" partition full (totally full, and
partially full by creating a large file under /var/tmp), and I am
confident that the modified fmtutil.sh works much better:

 - it prints the warning and error messages much more reliably than the
   current version.
  (It succeeded under the near full/full conditions of my testing.)

 - it returns proper error code in each failure case.

I believe I have covered all the corner cases.

Well, Famous Last Words.
So, I don't claim it is perfect, but it is much better.

TWO MAJOR BUGS UNCOVERED

The two major bugs discovered after my last post to Debian bug
tracking are follows. I was not aware of the second problem below and
so was wondering in detail if the tricky syntax of "verbose" could be
the cause of the strange behavior in previous postings.  (It still
could be, but even after my modification to "verbose", the following
two bugs showed up..)

It turned out that a couple of commands may have been responsible for

 - the failure of flushing the warning and error messages at the end.
  (A command was not properly checked and so it triggered trap
   processing bypassing "byebye" processing that would have flushed
   the messages.)
   It would still return exit code 1. Hmm...

 - the failure to return the error code properly. An "mv" command was
   executed in the following manner.

   if mv .... < /dev/null
   then
     ...
     ...[lengthy processing]
     ...
   fi

   Note that there was NO "else" block to handle the error of "mv"
   (say, due to full file system) by calling "log_failure", etc.
   Possible error caused by full file system, etc. was used up by "if"
   checking itself, and no error exit code was ever produced.  In this
   case, error code is forgotten!  Bad, bad, bad.

(The second problem did not show up on a linux installation, but
it became apparent on a different linux installation. That is how I
noticed the bug.)

In the proposed patch to tex-live@tug.org, I reverted the test
condition of "if mv ... < /dev/null" to "if ! mv ... < /dev/null" so
that the error processing block comes first before the lengthy normal
processing. It is much easier this way to understand the structure of
the code.
Last time I checked, the very long normal processing block made me
lose the fact that there was no matching "else" block. I mistook
another "else" block for the missing one.

I will monitor tex-live mailing list although, as noted by Norbert,
people seem to be very busy preparing 2014 TeXLive image.

We have to wait for the stable upstream version.  But it will be great
if we can incorporate the gist of changes into Debian packages
eventually. ("set -e" was not in TeXlive 2013, it is a Debian
addition, for example.)

TIA

PS: In the end, rewriting fmtutil.sh in Perl, C, java, or whatever
reasonably powerful programming language might be
better because the loose standards regarding the
trap handling of the failure of command when "set -e" is issued.

I see "byebye called" in the posted log to tex-live
mailing list: the log was produced under almost full /var partition condition.
E.g.:
 [...]
'mv tex.fmt /var/lib/texmf/web2c/tex/tex.fmt' failed
'mv dviluatex.fmt /var/lib/texmf/web2c/luatex/dviluatex.fmt' failed
byebye  called
We had error(s).

If somehow subtle change of trap handling occurs due the differences of
shell versions, some installation failure may see

byebye 1 called

instead. (This is how "byebye" is called in my modified version from
trap processing.)  Such a line will give us the clue of the different
way of trap handling under the particular shell used.


Thank you again.



2014-05-23 15:01 GMT+09:00 Norbert Preining <preining@logic.at>:

> Hi,
>
> > second
> > execution would print error message to console (unlike the re-directed
> > execution on LHS of "||").
>
> Yes, that was quite surely his intention ;-) But you can also try to
> ask him ;-)
>
> Anyway, rewriting fmtutil in perl is on my personal todo list.
> I have already rewritten (based on previous work) updmap in perl,
> and at the end I want to get rid of the separation c-code for Windpws
> and shell code for the rest of fmtutil, by replacing it with all
> perl code.
>
> At that point, my aim is also to have a decent error handling ;-)
>
> > You are right. My point here was that the script seems to "forget" that
> the
> > error was there during the re-production of the whole set of .fmt files,
> and
> > did not return error code (my casual reading. Maybe I am wrong.) at the
> end.
> > I think it should.
>
> Actually, that is a known problem. fmtutil does not do *any* proper
> error handling. We are well aware of that, one other reason why I want
> to rewrite it in perl ;-)
>
> > Thank you again. I noticed by reading tex live mailing list that
> > you and a few others are at the core of maintaining tex live.
>
> Indeed, indeed. Although I also maintain the Debian packages, my
> main part is in upstream (now, not originally).
>
> > (Yeah, I even tried to install OLD ASCII ptex on my own after
> encountering
>
> Oh, bad bad idea. There is Okumura-sensei's TeX Q-A and there was
> ptetex and ptexlive, but I incorporated all the pTeX stuff and
> support packages over the last years, with lots of support from
> many of the Japanese TeX community, into TeX Live. Before it was
> a drag to get pTeX (upTeX etc) running.
>
> > I wonder what people would do when old TeX files that were once typeset
> > using the vertical writing feature supported by ASCII's ptex needed to be
> > modified afresh and typeset again.
>
> That is not a problem. You are speaking about the *format*.
> The format can be regenerated. There has been a problem 6 years
> ago when the released version we had in Debian just fell out
> of these 5 years, and we had to patch the latex.ltx, but that
> has no influence. You can still run old code and get the same
> output (unless you use many packages that have changed
> considerably - if you *reall* want 100% reproducibility in the
> future, you have to stay with *plain* ptex and not use latex
> and add-on packages!).
>
> > We may need a time machine (of a sort) where the old typeset software is
> > kept intact so that in the future the old documents that need
> modification
>
> We have mirrors of old TeX Live's, so you can go back at least with
> TeX Live.
>
>
> A, one more remark: TeX Live is in the hot phase of pre-release,
> so don't expect toooo much activity and response *now* on fmtutil
> requests, as everyone of (the few of us) is busy for the final
> image burning etc etc.
>
> Thanks for all your work on fmtutil, much appreciated.
>
> Norbert
>
> ------------------------------------------------------------------------
> PREINING, Norbert                               http://www.preining.info
> JAIST, Japan                                 TeX Live & Debian Developer
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
> ------------------------------------------------------------------------
>

[Message part 2 (text/html, inline)]

Message #30 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <preining@logic.at>

To: Zephyrus C <zephyrus8080@gmail.com>

Cc: 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 30 May 2014 12:23:31 +0900

Hi Zephyrus,

thanks for your long and useful email on the TeX Live mailing list.
I have already private communication with other members how we 
proceed, but for now (and the next 1-2 months) no changes are
possible ;-)

> PS: In the end, rewriting fmtutil.sh in Perl, C, java, or whatever

Actually, due to your input I finally kicked myself to start
this project in perl ;-) Of course perl, because we need it running
on all possible distributions including windows ;-)
(I am speaking now as TeX Live maintainer, not Debian!)
If you want to contribute to the perl code, I can arrange.

ALl the best

Norbert

------------------------------------------------------------------------
PREINING, Norbert                               http://www.preining.info
JAIST, Japan                                 TeX Live & Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Message #35 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <preining@logic.at>

To: Zephyrus C <zephyrus8080@gmail.com>

Cc: 748962@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 30 May 2014 12:28:26 +0900

> If you want to contribute to the perl code, I can arrange.

https://gitorious.org/tex-live-rewrites/fmtutil/

Norbert

------------------------------------------------------------------------
PREINING, Norbert                               http://www.preining.info
JAIST, Japan                                 TeX Live & Debian Developer
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Message #40 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Hilmar Preuße <hille42@web.de>

To: Norbert Preining <preining@logic.at>

Cc: 748962@bugs.debian.org, Zephyrus C <zephyrus8080@gmail.com>

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Thu, 4 Jul 2019 23:24:02 +0200

[Message part 1 (text/plain, inline)]

On 30.05.14 05:28, Norbert Preining wrote:

Hi Norbert,

>> If you want to contribute to the perl code, I can arrange.
> 

> Actually, due to your input I finally kicked myself to start
> this project in perl  Of course perl, because we need it running
> on all possible distributions including windows 
> (I am speaking now as TeX Live maintainer, not Debian!)
> If you want to contribute to the perl code, I can arrange.

> https://gitorious.org/tex-live-rewrites/fmtutil/
> 
Just curious: what is the status here? The repo does not exist any more.
Seems to be migrated to perl since 2017 = Debian 9. I also noticed some
code, which returns exit code > 0 in case of failure.

Can we close?

Hilmar
-- 
sigfault
#206401 http://counter.li.org

[signature.asc (application/pgp-signature, attachment)]

Message #45 received at 748962@bugs.debian.org (full text, mbox, reply):

From: Norbert Preining <preining@logic.at>

To: Hilmar Preuße <hille42@web.de>

Cc: 748962@bugs.debian.org, Zephyrus C <zephyrus8080@gmail.com>

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 5 Jul 2019 11:32:10 +0900

Hi Hilmar,

> Just curious: what is the status here? The repo does not exist any more.
> Seems to be migrated to perl since 2017 = Debian 9. I also noticed some
> code, which returns exit code > 0 in case of failure.

Yes, fmtutil the Perl variant is now the standard in TeX Live.

> Can we close?

No idea, sorry. I would say yes, but I haven't tested on a full disk.

Best

Norbert

--
PREINING Norbert                               http://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Message #50 received at 748962-done@bugs.debian.org (full text, mbox, reply):

From: Hilmar Preuße <hille42@web.de>

To: 748962-done@bugs.debian.org

Subject: Re: Bug#748962: texlive-bastexlive-base: fmtutil-sys --all may not return error exit code under low free space condition.

Date: Fri, 5 Jul 2019 09:38:53 +0200

[Message part 1 (text/plain, inline)]

Version: 2016.20170123-5

On 05.07.19 04:32, Norbert Preining wrote:

Hi,

>> Can we close?
> 
> No idea, sorry. I would say yes, but I haven't tested on a full disk.
> 
Test:

root@amd64-sid:~# fmtutil-sys --all
fmtutil: fmtutil is using the following fmtutil.cnf files (in precedence
order):
fmtutil:   /usr/share/texmf/web2c/fmtutil.cnf
fmtutil:   /usr/share/texlive/texmf-dist/web2c/fmtutil.cnf
fmtutil: fmtutil is using the following fmtutil.cnf file for writing
changes:
fmtutil:   /etc/texmf/web2c/fmtutil.cnf
Error in tempdir() using /tmp/XXXXXXXXXX: Could not create directory
/tmp/6o_dtx0l2C: No space left on device at /usr/bin/fmtutil line 367.
root@amd64-sid:~# echo $?
28

When there is a little more space and some formats can be built we have
exit code 7:

! Could not write 3297 4-byte item(s) to mptopdf.fmt.
 (preloaded format=mptopdf 2019.7.5)fmtutil [ERROR]: running `pdftex
-ini   -jobname=cont-en -progname=context -8bit *cont-en.mkii
</dev/null' return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [ERROR]: Cannot copy cont-en.fmt to
/var/lib/texmf/web2c/xetex/cont-en.fmt.
fmtutil [ERROR]: running `xetex -ini   -jobname=xelatex
-progname=xelatex -etex xelatex.ini </dev/null' return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [ERROR]: running `luatex -ini   -jobname=dviluatex
-progname=dviluatex dviluatex.ini </dev/null' return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [ERROR]: running `luatex -ini   -jobname=dvilualatex
-progname=dvilualatex dvilualatex.ini </dev/null' return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [ERROR]: running `pdftex -ini   -jobname=etex -progname=etex
-translate-file=cp227.tcx *etex.ini </dev/null' return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [ERROR]: running `pdftex -ini   -jobname=mptopdf
-progname=context -translate-file=cp227.tcx mptopdf.tex </dev/null'
return status 1
fmtutil [ERROR]: return error due to options --strict
fmtutil [INFO]: Disabled formats: 3
fmtutil [INFO]: Successfully rebuilt formats: 9
fmtutil [INFO]: Failed to build: 7 (pdftex/cont-en xetex/cont-en
xetex/xelatex luatex/dviluatex luatex/dvilualatex pdftex/etex
pdftex/mptopdf)
fmtutil [INFO]: Total formats: 19
fmtutil [INFO]: exiting with status 7
root@amd64-sid:~# echo $?
7

So I'd guess the bug is solved. I tested just unstable, but the code
looks basically the same in stable, hence I close w/ the version from
stable (soon oldstable).

Hilmar
-- 
sigfault
#206401 http://counter.li.org

[signature.asc (application/pgp-signature, attachment)]

Send a report that this bug log contains spam.