As Barton suggested, try
ratmx : true$
nullspace(rat(m))$
rat() isn't really a simplification function -- it is a change of
representation from general representation (a general tree of operators and
operands) to Canonical Rational Expression (CRE) representation (nested
polynomials where the "variables" (kernels) are in general representation).
CRE representation is contagious, so if the input is in rat form, the
calculations will be in rat form. Thus it should suffice to convert your
input to CRE form using rat.
Note though that there are certain simplifications that aren't "seen" in CRE
representation. For example, rat(sqrt(x))^2 remains sqrt(x)^2. If you have
roots in your input, you should probably set algebraic:true.
As it happens, many matrix operations, including triangularize, *always*
operate internally in CRE form and then convert back to general
represenation. If you set ratmx:true, they don't convert back.
In summary, if you want all matrix operations performed in CRE form, convert
the inputs to CRE form and set ratmx:true.
If your version of triangularize faster than the built-in one, perhaps you
should contribute it to Maxima so that others can benefit from it.... Is it
tuned specially for the sparse case?
-s