This compatibility behaviour implements the buggy behaviour of FOR /F
token parsing that can be observed in Windows' CMD, and that is tested
by the cmd_winetests.
It can be disabled at compile time via the MSCMD_FOR_QUIRKS define.
It fixes additional cmd_winetests, in concert with commit cb2a9c31.
Explanation of the implemented buggy behaviour
==============================================
In principle, the "tokens=x,y,m-n[*]" option describes a list of token
numbers (must be between 1 and 31) that will be assigned into variables.
Theoretically this option does not cumulate: only the latest 'tokens='
specification should be taken into account.
However things are not that simple in practice. First, not all of the
"tokens=" option state is reset when more than one specification is
provided. Second, when specifying a token range, e.g. "1-5", Windows'
CMD just ignores without error ranges that are not specified in
increasing order. Thus for example, a range "5-1" is ignored without
error. Then, token numbers strictly greater than 31 are just ignored,
and if they appear in a range, the whole range is ignored.
Another bug is the following one: suppose that the 'tokens'
specification reads:
"tokens=1-5,1-30" , or: "tokens=1-5,3" ,
i.e. more than one range, that overlap partially. Then the actual total
number of variables will not be of the larger range size, but will be
the sum, instead.
Thus, in the first example, a total of 5 + 30 == 35 variables (> 31) is
allocated, while in the second example, a total of 5 + 1 == 6 variables
is allocated, even if they won't all store data !!
In the first example, only the first 30 FOR variables will be used, and
the 5 others will contain an empty string. In the second example, only
the first 5 FOR variables will be used, and the other one will be empty.
We also see that due to that, the "Variables" buffer of fixed size
cannot always be used (since it can contain at most 32 variables).
Last but not least, when more than one "tokens=" specification is
provided, for example:
"tokens=1-31 tokens=1-20"
a total number of 31 FOR variables (because 31 is the max of 31 and 20)
is allocated, **but** only 20 are actually used, and the 11 others
return an empty string.
And in the specification: "tokens=1-31,* tokens=1-20", a total of
31 + 1 + 20 = 52 variables is initialized, but only the first 20 will
be used, and no "remaining-line" token (the '*' one) is used.
Suppose the following FOR-loop command, to be run from the command-line
(if using a batch file, double each percent '%' sign):
FOR %l IN ("a,b,c,d,e" "f,g,h,i,j") DO (
FOR /F "delims=, tokens=1-3*" %a IN (%l) DO @echo %a-%b-%c-%d
)
The outermost FOR-loop enumerates the two strings "a,b,c,d,e" and
"f,g,h,i,j" (placed in %l), and parse each of these in turn, splitting
them at each specified delimiter (here only one character) ',' and storing
the results in consecutive tokens %a, %b, %c, %d, with the last token %d
containing all the remaining string (non-split).
The expected result is:
a-b-c-d,e
f-g-h-i,j
However, due to the way the delimiters string specified by the "delims="
option is stored (no stack/heap duplication of the FOR-option substring,
but reading from it directly), during the first run of the innermost
FOR-loop, the option string "delims=, tokens=1-3*" was truncated to just
after the ',' due to the erroneous "delims=" parsing, so that when this
FOR-loop ran for a second time (to deal with the second string), the option
string was already erroneously truncated, without the "tokens=..." part,
so that the parsing results were not stored in the tokens and resulting in:
a-b-c-d,e
f-%b-%c-%d
instead. The solution is to save where the "delims=" string needs to be
cut, but wait until running the actual FOR-loop to terminate it (and
saving the original character too), run the FOR-loop body, and then
restore the original character where termination took place. This allows
having the FOR-loop option string valid for the next execution of the
FOR-loop.
- To this purpose use the ParseErrorEx() that correctly sets the
bParseError flag, and return the partially-parsed command so that
it gets echoed as well for diagnostics purposes (Windows-compatible).
- Any other parameters specified after (or before) the '/?' switch for
the FOR and IF commands, are considered fatal syntax errors as well,
thus we employ the ParseErrorEx() as well.
CORE-13713 CORE-13736
- In case execution of all batch contexts is stopped (by selecting "All"
at the Ctrl-C/Ctrl-Break prompt), notify as well the CheckCtrlBreak()
signal handler once there are no more batch contexts (this in effect
resets the internal 'bLeaveAll' static flag in CheckCtrlBreak).
This is an adaptation of the fix present in FreeCOM 1.5, first
described in https://gcfl.net/FreeDOS/command.com/bugs074g.html .
- Introduce a ParseErrorEx() helper that sets the 'bParseError' flag and
displays a customized syntax-error message, only for the first syntax
error encountered. Implement ParseError() around the *Ex function.
- In batch mode, echo the original pre-parsed batch file line if a parse
error has been encountered.
- When running a compound command - including IF, FOR, command blocks -,
and that control flow is modified by any CALL/GOTO/EXIT command,
detect this while running the compound command so as to stop it and go
back to the main batch execution loop, that will then set up the actual
new command to run.
- In GOTO, do not process any more parts of a compound command only when
we have found a valid label.
