(C2) prefix("foo");
(D2) "FOO"
(C3) foobar;
(D3) FOO bar
(C4)
is hardly a sensible behaviour. For example, I think it is the only
serious reason why vect.mac has still some glitches (try the example
of vect.dem which computes the DIVergence of some expression).
The part of the tokenizer which is supposed to detect such operators
(for a suitable meaning of `such', which does not include, e.g., the
NOT operator) is the function READ-COMMAND-TOKEN-AUX in nparse.lisp.
I rewrote it in a way which both seems to be more structured (and thus
allows for a less haphazard insertion of the bug fix) and to preserve
the various optimisations in the present code (which is due to
Schelter, I think). I also added an indecent amount of commenting
stuff.
Some simple-minded testing shows no difference at all in the amount of
consing or the number of calls between the present version and my
rewrite (sans bug fix). On the other hand, I was not able to get
reliable timings. (I used Kantrowitz's metering.lisp on Clisp for all
that.)
So here are the questions.
1) Does the proposed bug fix really give the desired behaviour?
2) Should we use something like the rewrite below or just hack a bug
fix into the present code?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ cut ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(in-package "MAXIMA")
(defun constituentp (char)
(or (alphabetp char)
(ascii-numberp char)))
(defun read-command-token-aux (tree)
"Essentially a transition tree parser. TREE is something like
(NIL (ANS NIL) (#\\+ (ANS $+)) (#\\< (ANS $<) (#\\= (ANS $<=))) ...
(#\\F #\\O #\\O (ANS $FOO) (#\\B #\\A #\\R (ANS $FOOBAR))
(#\\Q #\\U #\\U #\\X (ANS $FOOQUUX))) ... ).
Return the longest possible token or NIL."
(let ((ch (parse-tyipeek)))
(when (and (characterp ch)
(setf tree (parser-assoc (char-upcase ch) tree)))
(parse-tyi)
(pop tree)
(or
(cond ((atom (first tree))
;; An ordinary edge: try to match it with the next input char.
(read-command-token-aux (list tree)))
;; Terminal or branch node. We begin by checking the rest of
;; the tree since we have to do this anyway and we might
;; already win at this point. Kudos to Schelter for this.
((when (rest tree) (read-command-token-aux (rest tree))))
((eq (caar tree) 'ans)
;; A terminal node. Since the last clause didn't win we know
;; that this is already the longest match.
(unless (and (constituentp ch)
(constituentp (parse-tyipeek)))
(cadar tree)))
(t
;; A branch node: the first branch remains to be checked.
(read-command-token-aux (list (first tree)))))
(progn (unparse-tyi ch) nil)))))
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ cut ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Wolfgang