Is integration and differentiation code sepparate?

On 11/12/10 03:46 PM, Steve Haflich wrote:
> Dr. David Kirkby<david.kirkby at onetel.net>  wrote:
>
>     1) Generate a "random" polynomial using a pseudo random number generator.
>     2) Simplify the polynomail, so the output is in a form that's reproducible.
>     3) Integrate the polynomial.
>     4) Differentiate the result from (3)
>     5) Simplify the result from (4)
>     6) Compare the results from (2) and (5).
>
> There may be some who disagree, but our experience with test suites for
> large, complex, and long-lived software packages is this: It is a very
> bad idea to generate test cases by any random process in the test suite.
>
> The reason is that when something fails (or fails to fail) one cannot
> simply rerun the entire test suite to duplicate the failure, whether for
> confirmation of the failure, or debugging, or testing that a fix
> actually fixes the problem.

Well in my case, if any results failed, they would be recorded. I could now 
however run the entire test suite again. However, that would be trivially easy 
to modify to add that ability, by saving seed for the pseudo random number 
generator. If I wanted to get the same data file, I'd use the same seed again.

> This harm can be mitigated somewhat by providing a way to reset the
> random generator (which certainly can be done in Common Lisp) but this
> means the random-state must be isolated from anything else in the system
> (i.e., it cannot be the same random-state used by other computations)
> and even if this is done successfully, the CL random generator is
> guaranteed only to reproduce the same sequence when executed in the same
> implementation (including version and platform).  So when a test failure
> suddenly shows up when a new version of your favorite Lisp
> implementation is released, you cannot tell whether the failure was a
> real new bug, or merely revealed by a change in the generated random
> sequence.

I'd actually written my best code in C, not Lisp. I'd used lrand48(), though I 
am aware there are better random number generators. That has a period of 2^48. 
though I suspect the numbers would not pass any decent test for randomness. I 
don't think that should be an issue however.

I have in the past used some random number generators from Numerical Recipes in 
C, where I knew the random number generator had to be high quality.

> It is seductive to think that by executine randome tests with every
> execution of the test suite, more exhaustive coverage is attained over
> time.  Don't get seduced by this notion!

Can you explain the reason for this.

> Of course it can be effective to generate a large set of random tests
> just once, and then capture those particular generated tests
> indefinitely as source in the test suite.  That eliminates the
> variability between executions.

Thank you for that idea. That's not something I had considered.

Dave