I was looking over some of the other functions in stringproc and a bit
confused by the term "token" being used here. In parsetoken, it means a
substring representing a number, in split, it means a substring delimited by
a given character, etc. Then I saw the "token" function, which refers to
Paul Graham's definition of tokens, namely "a substring whose characters
satisfy a certain test function".
Given that Maxima already has a definition for the term "token", namely the
atomic units dealt with by the parser, I think it would be better if we used
that sense consistently, especially since it remains (as far as I know) the
standard sense in computer science.
-s