Appendix:
Accuracy
of
Numerical
Calculations
179
Level
°°:
Overflow/Underflow
Results which
would
lie
closer
to
zero
than
10""
underflow quietly
to
zero.
Any
result
that
would
lie
beyond
the
overflow
thresholds
±9.999999999
X 10" is
replaced
by the
nearest
threshold,
and
then
flag
9 is set and the
display blinks. (Pressing
[ON|
[ON|
or
[CF]
9 or
H
will
clear flag
9 and
stop
the
blinking.) Most functions
that
result
in
more
than
one
component
can
tolerate
overflow/
underflow
in
one
component without contaminating
the
other; examples
are
r+R
}
,
r+P"|,
complex arithmetic,
and
most
matrix
operations.
The
exceptions
are
matrix inversion
(1
1/x|
and
|T)),
I
MATRIX]
9
(determinant),
and|
LR."|.
Level
1
:
Correctly
Rounded,
or
Nearly
So
Operations
that
deliver "correctly rounded" results whose error
cannot exceed
Vz
unit
in
their
last
(10th) significant digit include
the
real algebraic operations
H
,
H
>
B
>
B
>
H
>
QEJ
.
1
1/*l
>
and
f%1
,
the
complex
and
matrix operations
[+]
and
[-1,
matrix
by
scalar
operations
[x]
and
{±\g division
by a
matrix),
and |
-M-I.MSl
.
These results
are the
best
that
10
significant digits
can
represent,
as are
familiar constants
QT],
1
[e*1,
2
[TNl,
1P
|LN|,
1
|-»RAD|,
and
many more. Operations
that
can
suffer
a
slightly larger error,
but
still
significantly smaller
than
one
unit
in the
10th significant digit
of
the
result, include
|
A%
|
,
|-»H|
,
|-»RAD|
,
|-»DEG|
,
|
Py.x
\
and
|Cy,x|
;
[LNl
,
JLOG|
,
1
10*
|
,
and
|TANH|
for
real arguments;
|-»P|
,
|
SIN'1
1
,
|
COS"1
1
,
|
TAN
1
1
,
|
SIMM'1
1,
|
COSH"1
1,
and
JTANH"1!
for
real
and
complex
arguments;
I
ABS|,
\lx~\,
and
1
1/x|
for
complex arguments; matrix
norms
|
MATRIX
| 7 and
[MATRIX]
8; and
finally
|
SIN
|,
|COS|,
and
|TAN|
for
real arguments
in
Degrees
and
Grads
modes (but
not in
Radians
mode
—
refer
to
Level
2,
page 184).
A
function
that
grows
to
°°
or
decays
to 0
exponentially fast
as its
argument approaches
±°°
may
suffer
an
error larger
than
one
unit
in
its
10th significant digit,
but
only
if its
magnitude
is
smaller
than
10~20
or
larger
than
1020;
and
though
the
relative error
gets
worse
as the
result gets more extreme
(small
or
large),
the
error
stays
below
three units
in the
last
(10th) significant
digit.
The
reason
for
this
error
is
explained later. Functions
so
affected
are
[e*~\,
[y*1.
fxTI
(for noninteger
x),
|SINH|,
and
|COSH|
for
real
arguments.
The
worst case known
is
3201,
which
is
calculated
as
7.968419664
X
1095.
The
last digit
4
should
be 6
instead,
as is the
case
for
7.2933'5,
calculated
as
7.968419666
X
1028.
To Next Page
Loading...
Need help?
General
