Jeff Epler's blog

7 March 2012, 15:42 UTC

Cancellation error

I was recently reminded of the importance of choosing numeric algorithms that don't behave catastrophically for certain inputs. One example is the calculation of 1-cos(θ) for small θ. In this case, cos(θ) is very close to 1, leading to a large cancellation error in the subtraction step.

read more…

All older entries
Website Copyright © 2004-2018 Jeff Epler