# Trying to use RX precision = double for 64-bit math . . .

I am attempting to get my RX621 configured to perform 64-bit math.

I have included <math.h> and <mathf.h>, changed C89 to C99 in both "C/C++" and "Standard Library" tabs, and on the "CPU"tab inside the "Details" section, I have selected "Double precision" for the Precision of double field.

My processor is performing the following :

5803636 x (5803636 / (55669978 / 10000)) = 404249823

The answer is actually :  6050333057

My code snippet follows :

dTemp = ((double)Numerator * ((double)Numerator / ((double)Denominator / 10000)));

dTemp is a double

Numerator and Denominator are both unsigned long

Am I missing something on the compiler settings?

Lane

• A quick check using Windows calculator says 6050333057 is the correct result. A rough estimate also says the processor is right:

(6*10^6)*(6*10^6)=36*10^12

(36*10^12)*10^4=36*10^16

(36*10^16)/(6*10^7)=6*10^9

• Wait, Windows Calculator and an estimate yield >>>>>    6*10^9, while the processor yields >>>>>  4*10^8 ???? That is not even close.

6050333057 = 0x168A0C181 which is a 33-bit answer.

404249823 = 0x18185CDF which is a 29-bit answer

• Sorry, my mistake.

The calculation on my RX62T gives the correct 6050333057.52267

What you give as "result" is a double float interpreted as long. You use only the first 4 byte of the result and use this as simple hexadecimal value.

The result is 0xDF 0x5C 0x18 0x18 0x0C 0x8A 0xF6 0x41

Please assign this result to a longlong variable and you will see the result you expect.

• That seems to do it.  Thank you for getting me through that layer of the onion.