topic-17785

Posts: 1096

Joined: Fri Oct 28, 2005 02:27 AM

Some confusion in math result
Posted: Thu Jan 07, 2010 03:25 AM

Using windows calculator, 2957.45 / 9 = 328.60555555555555555555555555556. In my program I got 328.61 not the expected result of 328.605555...

What am I missing here? Using xHarbour+BCC55

TIA

p.s. Just tested in xbscript.exe by simple means of "? 2957.45/9". Got 328.61 here too

FWH 11.08/FWH 19.12

BCC5.82/BCC7.3

xHarbour/Harbour

Posts: 10733

Joined: Sun Nov 19, 2006 05:22 AM

Re: Some confusion in math result
Posted: Thu Jan 07, 2010 03:41 AM

Internal value has full accuracy. What is displayed may be rounded off to 2 decimal places.

msginfo( 2957.45 / 9 ) shows 328.605556

We can use the result for further processing without losing accuracy.

Regards

G. N. Rao.

Hyderabad, India

Posts: 1096

Joined: Fri Oct 28, 2005 02:27 AM

Re: Some confusion in math result
Posted: Thu Jan 07, 2010 03:53 AM

Thanks for the reply Rao.

I think age is catching up to me :oops:

Resolved the issue easily by just temporarily increasing SET DECIMALS value

FWH 11.08/FWH 19.12

BCC5.82/BCC7.3

xHarbour/Harbour

FiveTech Support Forums