Barton explains a possible syntax, but
his advice depends on what you want to do with this notion. write it
out, manipulate it or evaluate it.
if f and g have one argument, then f o g could be
compose(f,g):= lambda([x], apply(f, [ apply(g, [x]) ]) )
compose(f,g)(3) --> f(g(3))
My assumption is that this kind of explicit evaluation is NOT what you
want, but that you want (for example)
simplification of expressions involving operators. But you have to
define what you want, and do that very carefully,
because the notation in your area of interest tends to be ambiguous.
There are papers on Macsyma and Lie brackets though,
and
here is one, that eventually gets to it, and has some references
http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19890017254_1989017254.pdf
see
http://www.cim.mcgill.ca/~migueltt/ltp/ltp.pdf
for a large report of tools for Lie Groups and operators. (same Lie,
different issue, I expect).