Date:     14 March 91                           Message No:     038

To:       TeX implementors and distributors

From:     Barbara Beeton

Subject:  Messages from DEK, part 3


************************************************************************

>>> DRF's "remaining comments"

Date: Fri, 3 Jan 92 17:23:42 PST
From: drf@frame.com (David Fuchs)
Subject: Re:  flash!!! notice that tex update is about to happen

My remaining comments:

============================================================================

Plain.tex never sets emergencystretch.

 [ dek:
        True; it doesn't set \tolerance real high either.  Changing that
        would affect huge no of existing documents, not to mention
        STORY.TEX in Chapter 6
 ]

============================================================================

Change "twice" to "thrice" here:

 [ dek:                                                 \check $2.56 ]

 @ The `\ignorespaces|loop|\unskip' in the following code is performed at most
 twice per call of |line_break|, since it is actually a pass over the
 entire paragraph.

 @<Find optimal breakpoints@>=

============================================================================

The last line makes a lie out of "two"; and there's a little misinformation
about the absoluteness of the second pass, which may not happen at all(?):

 Two passes might be made through the paragraph in an attempt to find at
[^^^ Several         dek:                                       $2.56 ]
 least one set of feasible breakpoints. On the first pass, we have
 |threshold=pretolerance| and |second_pass=final_pass=false|.
 If this pass fails to find a
 feasible solution, |threshold| is set to |tolerance|, |second_pass| is set
 |true|, and an attempt is made to hyphenate as many words as possible.
 If that fails too, we add |emergency_stretch| to the background
 stretchability and set |final_pass=true|.

<<< end DRF's "remaining comments"

************************************************************************

>>> suggestion -- additional tracing facilities for \if*

 [ dek:
        Barbara -- such suggestions are out of order and need not be
        sent to me in future -- just a waste of your time and mine
 ]

Date: Thu, 21 Nov 91 11:47:12 +0100
From: Bernd Raichle <raichle@azu.informatik.uni-stuttgart.de>
Subject: Proposal for an enhancement of TeX's \tracingcommands

Dear Ms. Beeton,

I have the following proposal for a TeX enhancement; the idea is
initiated by an article in TUGboat.  Perhaps it's worth (and small
enough) to get incorporated in the next version of TeX?

 [ dek:
        NEVER -- WILL PEOPLE PLEASE LEAVE ME ALONE.
        YIPES, I HAVE SAID IT ENOUGH, NOBODY LISTENS.
        I'M THROUGH!
        ALL SYSTEMS CAN BE IMPROVED _ENDLESSLY_ BUT IMPROVEMENTS
        EVENTUALLY HAVE A NEGATIVE EFFECT
 ]

In TUGboat vol.12, no.2, p.238ff, David Salomon wrote a tutorial about
"The \if, \ifx and \ifcat Comparisons".  At the end of the article he
suggested an enhancement of TeX's tracing facilities to show the
result (current feature) and the two comparands (new feature) of these
comparisons.  (He suggested a new control sequence \tracingcomparands
to toggle the additional log output, but I think that it's better to
use \tracingcommands with a value >= 3.)

I have made the following changes to TeX 3.14 to print additional
lines containing the comparands:

For \if and \ifcat change \S 506:

 begin
  ...
  get_x_token_or_active_char;
  if (cur_cmd > active_char) or (cur_chr > 256) then
    begin cur_cmd := relax; cur_chr := 256;
    end;
(* begin of new part *)
  if tracing_commands > 2 then
    begin begin_diagnostic; print_char("{");
    if this_if = if_char_code then print("if") else print("ifcat");
    print_char(" "); print_cmd_chr(m, n); print(" (");
    if this_if = if_char_code then print_int(n) else print_int(m);
    print(") = "); print_cmd_chr(cur_cmd, cur_chr); print(" (");
    if this_if = if_char_code then print_int(cur_chr) else print_int(cur_cmd);
    print(")}"); end_diagnostic(false);
  end;
(* end of new part *)
  if this_if = if_char_code then b := (n = cur_chr) else b := (m = cur_cmd);
 end

 [ dek:
        I appreciate a student's enthusiasm but wish it were channeled
        into new directions ...
        How about a really integrated system for Literate Programming,
        for example ...
        or how about a well documented replacement for UNIX?
 ]

For \ifx change \S 507:

 begin
  ...
  q := cur_chr; get_next;
(* begin of new part *)
  if tracing_commands > 2 then
    begin begin_diagnostic; print_char("{");
    print("ifx"); print_char(" "); print_cmd_chr(p, q);
    if p >= call then
      begin if n <> 0 then
        begin print_char(" "); sprint_cs(n);
        end;
      if q <> 0 then
        begin print_char(":");
          show_token_list(link(q), 0, 32);
          print_char(".");
        end;
      end;
    print_ln; print("   = "); print_cmd_chr(cur_cmd, cur_chr);
    if cur_cmd >= call then
      begin if cur_cs <> 0 then
        begin print_char(" "); sprint_cs(cur_cs);
        end;
      if cur_chr <> 0 then
        begin print_char(":");
          show_token_list(link(cur_chr), 0, 32);
          print_char(".");
        end;
      end;
    print_char("}"); end_diagnostic(false);
  end;
(* end of new part *)
  if cur_cmd <> p then b := false
  else if cur_cmd < call then b := (cur_chr = q )
    ...
 end


The changed "TeX" passes the TRIP test because TRIP sets
\tracingcommands to 0, 1 or 2 only.  There is no test for other
values.
 [ dek:
        So what? Anybody can enhance their private version but they
        can't put it on a public system and call it TeX if it has any
        extensions.
 ]
Is this a forgotten topic in TRIP?  Is the "TeX" with the changes
above still a TeX?  Is it allowed to make further tracing enhancement
 [ dek:
        I do believe however, that some kind of `system call' is a
        permissible implementation option if it is handled entirely
        via \read and \write to specific special files ...
        That is as far as I believe extensions are tolerable.
 ]
which affects the log output only and to call the result TeX?

 [ dek:
        Obviously TRIP cannot possibly check for all possible extensions,
        all of which are impermissible
 ]

Yours,
 -bernd
 __________________________________________________________________________
 Bernd Raichle, Student der Universit"at Stuttgart | "Le langage est source
 home:  Stettener Str. 73, D-W-7300 Esslingen, FRG |  de malentendus"
 email: raichle@azu.informatik.uni-stuttgart.de    |  (A. de Saint-Exupery)

<<< end suggestion -- additional tracing facilities for \if*

************************************************************************

>>> DEK's "local" change files, not at labrea

extract from dek's bug responses dated 26 Sep 91:
        My favored conventions on implementation questions in general
        are expressed by the change files I have contributed to the
        distribution [under `local' directory] ... 
it seems that your change files haven't been updated at labrea recently;
i'm hoping that the local mod referred to in this interchange really is
something you've done for yourself, and an update to labrea will solve
the problem.
 -------
Date: Thu, 17 Oct 91 01:55 PDT
From: barry@reed.edu (Barry Smith)

Barbara, perhaps you can help me easily?  I'm looking for an optional
change/patch to TeX by Knuth, around version 2.94; it changed the
handling of end-of-file to normal termination, for UNIX[-like] systems.
Doug and I poked around labrea, getting sticky with tar; I'm no
FTP expert, and couldn't locate the appropriate file or changes.

Any ideas?

Barry Smith, Blue Sky Research
barry@reed.edu
 -------
(reply, 17 Oct 91)
hi, barry.
i have to admit, i don't remember that one; of course, here at ams,
we weren't really interested in the unix mods until earlier this year.
but i'd guess that karl berry (karl@cs.umb.edu, the web2c guru), might
 [ dek:                                             ^^^^^
        Note: I do _not_ use web2c, I use the pascal compiler on Sun
 ]
remember it.  worth a try, anyhow.
if this fails, i can dig around in my own archives to see what might
be there, but i don't promise much.  what i've kept is the canonical
changes (tex82.bug, etc.) and items from knuth in answer to questions
i forwarded.  since he still had a net connection at the time of 2.94
(at least i think he still did), if a question came from another source
i might well not have any record.  i figure it'll take at least half a
day to do the necessary exploring, so you understand why i'm suggesting
that you check first with karl.  even if he doesn't have knuth's code,
it's quite likely that he will be able to come up with something on his
own pretty quickly.
let me know what happens.  and good luck.
                                                -- bb
 -------
Date: Sat, 30 Nov 91 15:36 PST
From: barry@reed.edu (Barry Smith)
 
Barbara, can I ask you to take a longer look (for that Knuth patch to
TeX to make end-of-file a normal exit\end)?  Neither Karl Berry nor
Pierre MacKay can find it, and the .local directories on labrea seem to
be confused... (that's where I'd expect to really find what I'm looking
for, in a Knuth change file for UNIX; but I find no trace of Knuth change
files on labrea even though he explicitly referred to such in his most
recent missive via yourself.)
 
Anyway, if it's at all easy to "grep" or equivalent on everything you've
received from Don for, say, 'UNIX' or ('end' and 'file'), I'd appreciate
the work.  I'm beginning to doubt that I ever saw it...
 
barry@reed.edu
Barry Smith
-------
(reply, 10 Dec 91)
barry,
sorry to take so long to reply, but i've been out of town (again!).
although knuth did say he intended that the .local directories on
labrea would hold his examples, i can't find any evidence of them
having been updated with this material.  i'll try to get in touch
with the person who was last doing the updating, to see what he
might know.

as for having the patch for end-of-file here in my collection of
stuff, i can't find it, so i suspect it wasn't among anything he's
sent me.  (i'm a pretty thorough pack rat, and if i ever got it,
i'm sure i didn't get rid of it.  on the other hand, since we've
only recently started to use a unix implementation of tex, that's
not something i would have sought out.)  it's vaguely possible
that it might have been posted on texhax, and i didn't check there.

i'll forward your inquiry to don, but am not sure when he'll be
answering mail again; i'd been led to believe that he wouldn't
be back in gear until january.
sigh.                                           -- bb

 [ dek:
        I no longer remember exactly what Barry asked that I was
        answering on 26 Sep; but I did check the sources at labrea
        and found everything on the directory  ~ftp/pub/tex/local
        (and its subdirectories)
 ]

<<< end DEK's "local" change files, not at labrea

************************************************************************
-------
Date: Sun 5 Jan 92 21:41:55-EST
From: bbeeton <BNB@MATH.AMS.COM>
To: winkler@cs.stanford.edu
Subject: bug reports, part 2 of 2

>>> glue_ratio overflow

[ this is a discussion from the tex-implementors discussion list ]
 -------
Date: Mon, 23 Dec 91 13:23:32 +0100
From: Eberhard Mattes <mattes@azu.informatik.uni-stuttgart.de>
To: tex-implementors@math.ams.com
Subject: overflow when converting real -> integer (scaled)

% This TeX input causes an overflow in the following line of |vlist_out|:
%
%   rule_ht:=rule_ht+round(float(glue_set(this_box))*stretch(g));
%
% The |round| Pascal function usually prints an error message and aborts
% if the |real| value cannot be converted to an |integer| (as in this case).
%
% What should a decent TeX implementation do?
%
% Does TeX need a special |round| implementation which uses some $x$ or
% $-x$ (depending on the sign of the argument) on overflow, where both $x$
% and $-x$ are |integer| values (|maxint|, for instance)? But that `solution'
% would break other combinations of values. Is there any real solution?
%
% DEK writes in glue.web:
%
%    surely we need not bother trying to accommodate such anomalous
%    combinations of values.
%
% but there are real-life texts, where this happens, see below. LaTeX uses
%
%   \vskip 0pt plus .0001fil
%
% in \raggedbottom! (Maybe that can be changed.)
%
% The behavior of TeX is machine/implementation-dependent for this file,
% maybe @^Overflow in arithmetic@> should be added to tex.web for that line
% and similar lines.
%

\shipout
  \vbox to 100pt{%
    \vskip0pt plus 1fil
    \vskip0pt plus -1fil
    \vskip0pt plus 0.00001fil
    }
\end


%
% Here's a short LaTeX/REVTEX input file which shows the problem (the
% problem is not due to the almost empty tabular environment!):
%

\documentstyle[revtex]{aps}
\begin{document}
\begin{table}
\begin{tabular}{rr}
\\
\end{tabular}
\end{table}
\pagebreak
\end{document}

%
% Yours,
%   Eberhard Mattes (mattes@azu.informatik.uni-stuttgart.de)
%
 -------
Date: Mon, 23 Dec 91 10:35:26 EST
 [ dek:       Merry Christmas ]
From: jjs@arbortext.com
To: mattes@azu.informatik.uni-stuttgart.de,
Subject: Re:  overflow when converting real -> integer (scaled)

Greeting,

  We ran into this quite a while back as well at ArborText.  I reported the
problem to Knuth, whose response basically was: "Why would you ever want
to do that?  That's really not a bug in tex.web, it's in implementation 
restriction that you're more than welcome to fix up in your change files."  

  Turns out we had quite a few legimate reasons to want to do that as well...

  The only "correct" way to solve this would be to use 8-byte long arithmetic
for these computations.
 [ dek:
        Well then you would exceed dvi limitations ... and if you patch
        that you will break the drivers.
 ]
                         Short of that, we did the following.  Note that the
problem shows up in both |hlist_out| and |vlist_out|.

@x Module 625
@ @<Move right or output leaders@>=
begin g:=glue_ptr(p); rule_wd:=width(g);
if g_sign<>normal then
  begin if g_sign=stretching then
    begin if stretch_order(g)=g_order then
      rule_wd:=rule_wd+round(float(glue_set(this_box))*stretch(g));
@^real multiplication@>
    end
  else  begin if shrink_order(g)=g_order then
      rule_wd:=rule_wd-round(float(glue_set(this_box))*shrink(g));
    end;
  end;
if subtype(p)>=a_leaders then
  @<Output leaders in an hlist, |goto fin_rule| if a rule
    or to |next_p| if done@>;
goto move_past;
end
@y4
@ For Pub\TeX, we add some logic to the step where the glue set is multiplied
by the stretch/shrink.  The product can be greater than |max_int| in
certain situations.  To prevent an integer overflow on the subsequent
|round| operation, truncate large products to |max_allowed_glue|.

The rationale for picking the value of |max_allowed_glue| was simply to
use a very large number approaching |maxdimen|.  There is the potential
that this truncation may result in incorrect results, but our experience
has shown that with glue values this large the truncation makes little
difference and is much preferable to simply letting an integer
overflow exception happen.
 [ dek:
        ( I agree )
 ]

@d max_allowed_glue==float_constant(100000000)
 [ dek:
        about 21 inches ... fine,
        but 200 000 000 should be OK too [and slightly better]
 ]

@<Move right or output leaders@>=
begin g:=glue_ptr(p); rule_wd:=width(g);
if g_sign<>normal then
  begin if g_sign=stretching then
    begin if stretch_order(g)=g_order then
      begin
        g_temp:=float(glue_set(this_box))*stretch(g);
@^real multiplication@>
        if abs(g_temp)>max_allowed_glue then 
          begin
            if g_temp > float_constant(0) then g_temp:=max_allowed_glue
            else g_temp:=-max_allowed_glue;
          end;
        rule_wd:=rule_wd+round(g_temp);
      end;
    end
  else  begin if shrink_order(g)=g_order then
      begin
        g_temp:=float(glue_set(this_box))*shrink(g);
@^real multiplication@>
        if abs(g_temp)>max_allowed_glue then 
          begin
            if g_temp > float_constant(0) then g_temp:=max_allowed_glue
            else g_temp:=-max_allowed_glue;
          end;
        rule_wd:=rule_wd-round(g_temp);
      end;
    end;
  end;
if subtype(p)>=a_leaders then
  @<Output leaders in an hlist, |goto fin_rule| if a rule
    or to |next_p| if done@>;
goto move_past;
end
@z

@x Module 629
@!g_order: glue_ord; {applicable order of infinity for glue}
@!g_sign: normal..shrinking; {selects type of glue}
@y4
@!g_order: glue_ord; {applicable order of infinity for glue}
@!g_sign: normal..shrinking; {selects type of glue}
@!g_temp: real; {used while stretching/shrinking glue}
@z

@x Module 634
@ @<Move down or output leaders@>=
begin g:=glue_ptr(p); rule_ht:=width(g);
if g_sign<>normal then
  begin if g_sign=stretching then
    begin if stretch_order(g)=g_order then
      rule_ht:=rule_ht+round(float(glue_set(this_box))*stretch(g));
@^real multiplication@>
    end
  else  begin if shrink_order(g)=g_order then
      rule_ht:=rule_ht-round(float(glue_set(this_box))*shrink(g));
    end;
  end;
if subtype(p)>=a_leaders then
  @<Output leaders in a vlist, |goto fin_rule| if a rule
    or to |next_p| if done@>;
goto move_past;
end
@y4
@ See the comments in |hlist_out| for an explanation of the 
Pub\TeX\ related changes made to the way the floating point multiplication
is done in this module.

@<Move down or output leaders@>=
begin g:=glue_ptr(p); rule_ht:=width(g);
if g_sign<>normal then
  begin if g_sign=stretching then
    begin if stretch_order(g)=g_order then
      begin
        g_temp:=float(glue_set(this_box))*stretch(g);
@^real multiplication@>
        if abs(g_temp)>max_allowed_glue then 
          begin
            if g_temp > float_constant(0) then g_temp:=max_allowed_glue
            else g_temp:=-max_allowed_glue;
          end;
        rule_ht:=rule_ht+round(g_temp);
      end;
    end
  else  begin if shrink_order(g)=g_order then
      begin
        g_temp:=float(glue_set(this_box))*shrink(g);
@^real multiplication@>
        if abs(g_temp)>max_allowed_glue then 
          begin
            if g_temp > float_constant(0) then g_temp:=max_allowed_glue
            else g_temp:=-max_allowed_glue;
          end;
        rule_ht:=rule_ht-round(g_temp);
      end;
    end;
  end;
if subtype(p)>=a_leaders then
  @<Output leaders in a vlist, |goto fin_rule| if a rule
    or to |next_p| if done@>;
goto move_past;
end
@z


  Jim Sterken
  ArborText, Inc.

============================================================

        [ repeat of message from Eberhard Mattes ]
 -------
Date: Sun, 29 DEC 91 23:22:49 GMT
From: CA_ROWLEY@vax.acs.open.ac.uk
Subject: RE: overflow when converting real -> integer (scaled)

A few observations on this problem.

1.  Despite the use of:

\vskip 0pt plus 0.0001fil
 [ dek:
        by the way I believe .001fil or even .01fil will be
        sufficiently undetectable to be acceptable in LaTeX
 ]

*pure* LaTeX will not produce this problem (unless the actual 
page-size and, hence, the size of some glue really gets too big).

However, the example below shows that the run-time error can be
produced by the use of \filbreak, a command that is not documented in
LaTeX but is nevertheless available (and useful).

It (\filbreak) has been in frequent use within our LaTeX-based system 
for about 5 years now, and has only produced this run-time error when 
running test documents: ie it is not, in practice, a problem.

Here is the type of LaTeX+\filbreak document which does produce it.

\documentstyle{article}    
\raggedbottom
\begin{document}
Small amount of text.
\filbreak
Small amount of text.
\par
\pagebreak
Next page
\end{document}

[I have not yet looked at the REVTEX styles to see why they produce 
it, as documented by Eberhardt.]

2. Although the Arbortext idea will work for Eberhardt's example:

> \shipout
>   \vbox to 100pt{%
>     \vskip 0pt plus  1fil
>     \vskip 0pt plus -1fil
>     \vskip 0pt plus 0.00001fil
>                  }
> \end

it will produce completely incorrect layout for this, similar, example:

\shipout 
  \vbox to 100pt{%
    \vskip 0pt plus 0.00001fil
      \hbox{Text.}
    \vskip 0pt plus 1.00001fil
    \vskip 0pt plus -1fil}
\end
 [ dek:
        actually I think this will come out fine but other (similar)
        examples will break
 ]

This is not intended to imply that their solution is not good enough: 
it will certainly be OK for all the code in use which I know of (the 
above example was constructed, in the true spirit of numerical 
analysis, explicitly to break their solution!).

What it does illustrate is that certain pieces of TeX code lead to 
such inherently ill-conditioned calculations; thus nothing short of 
 [ dek:                         ^^^^^^^^^^^^
        but only in places where the user (=style designer) will
        understand her error
 ]
higher precision arithmetic will solve the problem in general. 

3.  What is needed in the TeX code is something such as Arbortext's 
solution PLUS something which writes in the LOG file to record where the 
value was changed and the fact that the spacing may therefore be 
incorrect.  This would be better than implementation-dependent code 
 [ dek:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        sure
 ]
but could be open to the criticism that it does not allow an
 [ dek:                                    ^^^^^^^^^^^^^^
        ? why doesn't it allow higher precision?
 ]
implementation to use higher precision arithmetic as its solution.

4.  What is needed elsewhere is education of TeX programmers which:

A.  Describes why certain types of TeX code are dangerous in that they
can lead to such calculations.

B.  Explains what the TeX program does about the problem and what 
affects this can have (eg how it can give incorrect layout).

C.  For known dangerous uses of such code, shows how to change the 
TeX coding so that it still does what is needed whilst avoiding the 
problems.

4.  To implement 3C, we need to know what sort of code is in use which
either has, or may one day, produce this error (I wonder how many 
times it has been seen and not reported?).  For example, what are 
Arbortext's uses?
 [ dek:
        yes
 ]

chris
 -------
Date: Fri, 03 Jan 92 09:46:25 GMT
From: "Wayne G. Sullivan"
      <WSULIVAN%IRLEARN.UCD.IE@ICNUCEVM.CNUCE.CNR.IT>
Subject: glue ratio overflow

Glue ratio overflow can never occur with nonnegative glue entries provided
that individual terms and their sums do not exceed TeX max dimen value
(the `minus' in glue specifications refers to shrink, not negation).

Negative glue values are `tricks', but there are important applications.
When negative values are used, overflow is possible at any fill level,
including `normal'. In practical cases overflow will not occur unless
the total is nearly, but not equal, zero, and relatively large glue
entries occur. No range or accuracy of floating point arithmetic can
overcome glue ratio overflow without changing the basics of TeX
arithmetic.

A glue value of the form \vskip 10pt plus 0.0001 fil will cause problems
only when there the additional trick of the form
\vskip 0pt plus 1 fil \penalty -500 \vskip 0pt plus -1 fil
is used. Since TeX does have 3 levels of infinity for fil, is it wise to
use 0.0001 fil? The basic LaTeX files never use filll, if a skip less than
1 fil is necessary, perhaps it would have been better to have used fill and
filll instead of fil and fill in the basic files.

To avoid glue ratio overflow one need only insure that the use of teeny
fil's does not occur with plus 1 fil and plus -1 fil, or better, make
these tricks at a different fil level.

<<< end glue_ratio overflow


########################################################################

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%  Character code reference
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                       Upper case letters: ABCDEFGHIJKLMNOPQRSTUVWXYZ
%                       Lower case letters: abcdefghijklmnopqrstuvwxyz
%                                   Digits: 0123456789
% Square, curly, angle braces, parentheses: [] {} <> ()
%           Backslash, slash, vertical bar: \ / |
%                              Punctuation: . ? ! , : ;
%          Underscore, hyphen, equals sign: _ - =
%                Quotes--right left double: ' ` "
%"at", "number" "dollar", "percent", "and": @ # $ % &
%           "hat", "star", "plus", "tilde": ^ * + ~
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[ end of message 038 ]