Fixing ?foo case folding



On encountering `?foo' (where `?' is not followed by whitespace, colon
or double-quote) Maxima collects characters in the following way
(cf. SCAN-LISP-TOKEN in nparse.lisp):

1) If the character immediately following `?' is not a backslash
   collect it, otherwise collect the character following the
   backslash.  Then collect all following characters as long as they
   are either alphanumeric (in the sense of Maxima) or escaped (that
   is, preceded by a backslash, which is not collected).

2) IMPLODE1 the resulting list of characters.  No further escape
   processing is done at this stage: All characters are literal.
   However, if the list of characters does not contain the vertical
   bar character `|' all characters are folded to uppercase.

   Conversely, if `|' is contained anywhere in the list of characters
   all characters are retained as they are, including `|' itself.
 
I think the behaviour described in the last sentence is broken.  It
means that there is no way to get a symbol whose name contains lower
case letters except if it happens to contain `|' as well (you could
use ?"foo" for this, though)

(C1) ?foo\|ba\|r\|;
(D1)                               foo|ba|r|

Perhaps, who wrote SCAN-LISP-TOKEN was thinking that the stream of
characters in the list would be passed to the lisp reader instead of
being IMPLODE1ed.  However, doing this would mean that the list
produced by step (1) must be acceptable to READLIST (defined in
nparse.lisp) or some similar function.  It would also introduce a
second round of escape processing.

A less drastic approach, which would change the behaviour only
where it is currently broken, is to let `|' quote itself, that is, `|'
works as a switch for preserving/folding case but `||', contrary to
what it means in lisp, is a literal `|'.

(C3) ?foo\|ba\|\|\|r\|quux;
(D3)                              FOOba|Rquux
(C4) 

An implementation of this can be found in

http://members.inode.at/wjenkner/maxima/scan-token-fix.lisp

Other suggestions? Corrections? Preferences?

Wolfgang