From: John Charlton (johncharlton_at_mac.com)
Date: Tue Dec 09 2003 - 20:26:23 PST
Close enough, isn't it? :-)
This is a common 'feature' of many smart and not-so-smart calculators.
When a decimal 'float' (a number in base 10 with a decimal part) is
stored in the binary notation used by the computer the conversion isn't
always exact. After some operations that any eight year old can do (um,
maybe not /all/ of them) the little bits at the bottom get out of whack
and some rounding occurs.
It's an annoying problem, especially if you need to test for equality.
Work-around: try to use integers, they're less likely to drive you mad.
Time for my favourite sig:
"There are 10 kinds of people in the
world: Those who understand binary
and those who don't."
On Dec 09, 2003, at 21:57, Gordon Stuart wrote:
> Today when I did a simple subtraction 22.35 - 22.8 the answer returned
> was 0.449999999999999
>
> Has anyone else had this problem?
-- This is the NewtonTalk list - http://www.newtontalk.net/ for all inquiries List FAQ/Etiquette/Terms: http://www.newtontalk.net/faq.html Official Newton FAQ: http://www.chuma.org/newton/faq/
This archive was generated by hypermail 2.1.5 : Wed Dec 10 2003 - 03:00:01 PST