Richard Hennessy wrote:
> "Floating-point operations are not exact, but they can be modeled as:
> fl( x op y ) = (1+D) (x op y)
> where |D| <= u, u is the unit round off or machine precision"
>
> I found this on the Internet and it is about running error analysis for numerical algorithms. I just want to know the value of u for big floats as a function of fpprec (as opposed to machine precision) since Maxima big floats are running in a virtual machine? Does anyone know how round off is handled with big floats.
u is 10^(-fpprec), approximately. In reality, fpprec is the number of
base-10 digits, but the implementation is in base-2, so fpprec is
converted to the number of bits.
Not sure what you mean by running in a virtual machine.
I think roundoff is pretty much handled in the same way as IEEE floats
so rounding is round to nearest even.
Ray