Maxima stats huge numbers of files at startup?

Hi,

This email is a followup about  "1. maxima opens the root directory /
and stats each file found there. Then it does the same thing for the
/u (home) directory..."

Thanks to everybody that responded to my query below.  The
"MAXIMA-SHAREDIR" variable looks fine according to the output of
"maxima --directories".
Off list, Andrej Vodopivec remarked "It could be share-subdirs-list in
init-cl.lisp. If that is true, then it should be easy to remove the
call to share-subdirs-list.".    I tried doing what Andrej suggested,
and it worked perfectly.  Before doing that, Maxima would state about
4000 files (including all users' home directories) on startup, and
afterwards it stat'd only about 70 files.   The difference in
performance on some NSF filesystems is huge -- a second versus
potentially *minutes*.    Even the difference on
sage.math.washington.edu is quite noticeable (a very fast machine with
a fast network).  Looking at the output of makes this very clear:

    strace maxima --directories  > out 2>&1; grep stat out|wc -l


For now we'll be patching the Maxima in Sage so that
share-subdirs-list (in init-cl.lisp) falls back to the old "default
behavior" instead of the new behavior that was introduced in the
recent rewrite of init-cl.lisp.    I really hope whoever rewrote
init-cl.lisp can think about the significant performance regression
that was caused, and find a better solution.

Thanks again for all the incredibly helpful feedback!

 -- William


On Thu, Dec 10, 2009 at 10:58 PM, Robert Dodier <robert.dodier at gmail.com> wrote:
> On 12/10/09, William Stein <wstein at gmail.com> wrote:
>
>> 1. "maxima opens the root directory / and stats each file found there.
>> ?Then it does the same thing for the /u (home) directory. ?The sys
>> admin believes the bottleneck is the slow response of doing a stat on
>> each NFS home directory, especially slowed down by the home
>> directories that are remotely NFS mounted from across the country."
>
> My first guess is that the Maxima build was misconfigured
> and Maxima doesn't have the right value of *MAXIMA-SHAREDIR*.
> What does "maxima --directories" report?
>
> Maxima tries to construct the list of share directories at run time.
> I'm guessing it's looking at / in the mistaken belief that it's the
> top of the share directory.
>
>> 2. The user reports that "maxima reads the file
>> local/lib/ecl-9.10.2/sysfun.lsp one character at a time doing 72K
>> read() calls of one character each. ?The sys admin doubts that is anywhere near the
>> problem of the NFS stat calls, but it does seem inefficient."
>
> Well, when I try it, I see sysfun.lsp being read in blocks of 8 K.
> Dunno why it would try to read it one byte at a time.
> Maybe ECL tried to determine a suitable block size and failed?
> Just a guess.
>
> FWIW
>
> Robert Dodier
>



-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org