Tapatalk

Why has the output sum of values in a data column 11 decimal places while each value has only one decimal place?

Why has the output sum of values in a data column 11 decimal places while each value has only one decimal place?

2

    Jul 07, 2022#1

    I am using the Sum selection command to sum up the values in 10,000 lines in column editing mode.

    This particular data column is basically this sum : 0.1+0.5+0.3+0.2+0.2+(9995*0.1) ... Can somebody please explain how / why UltraEdit sums up these values to 1000.80000000016?

    Where on earth does it find extra 10 decimal places?
    sum_selected_text.png (10.58KiB)
    Output sum of selected data column

    6,685587
    Grand MasterGrand Master
    6,685587

      Jul 07, 2022#2

      Floating point values as in your case with 0.1 plus 0.5 plus 0.3 plus 0.2 plus 0.2 plus 9995 times 0.1 are summed up using variables of type double as defined by the standard IEEE 754-2008. A double precision floating point number consists in memory of just 64 bits although the value range is far beyond −2^63 (−9223372036854775808) to +2^63−1 (+9223372036854775807) as it is for 64-bit signed integers. How is this possible? How can just 8 bytes store a value which needs as integer much more bytes in memory of a computer. Well, that is explained by the referenced Wikipedia article (and of course by the standard) – with loss of precision.

      UltraEdit does not know how many decimal places should be output for an accurate sum result. It would be necessary to find out how many decimal places each value summed up has to determine the correct number of decimal places on output of the double precision floating point number, i.e. if no value is in exponential notation and each value has only one decimal place, then the sum can be output also with just one decimal place. But UltraEdit does not contain code to find out what is the optimal number of decimal places on summing up all values of a selected column or a selection in normal or column editing mode. So it outputs the value as is which results in this case in the output of the floating point number with eleven decimal places. That is C/C++ standard. UltraEdit is coded in C++. So there is nothing unexpected for me writing C and C++ code since more than 25 years. See also on programmers website Stack Overflow: Is floating point math broken?

      It is really difficult to write C/C++ code which has to handle floating point numbers without knowing the value range which must be really handled and how many decimal places are really needed. I know that very well because I had to handle that also in one of my last programs running in a device used in power plants for various applications (excitation, protection, synchronization, turbine governor) which have completely different requirements regarding to the value range and the number of decimal places. I needed a while to find a solution how to output optimal as string a floating point value which is satisfactory for all users in all their use cases and nevertheless works also for output of floating point values being by exception out of the typical value ranges or need more decimal places than usually required by exception.

      BTW: UltraEdit does not have an arithmetic expression evaluation command. The command Sum columns/selection is for calculating the sum of all values in a selection.

      32-bit UltraEdit v2022.0.0.102 interprets a selection of 0.1+0.5+0.3+0.2+0.2+(9995*0.1) as:

      Code: Select all

         0.1
        +0.5
        +0.3
        +0.2
        +0.2
      9995
         0.1
      So it calculates 0.1 + +0.5 + +0.3 + +0.2 + +0.2 + 9995 + 0.1 and produces as result correct 9996.4. I just wanted to add this as most users of UltraEdit do not know that the command Sum columns/selection can be used also on any selection made in normal text editing mode on a text containing values with some (not all) other characters also inside the selected text.

      PS: Calculators - real physical calculators (pocket calculators) as well as programs written just for calculations like Calculator of Windows - use much more bits to increase the precision of integer and floating point calculations.
      Best regards from an UC/UE/UES for Windows user from Austria

      2

        Jul 08, 2022#3

        Thank you, Mofi, for a very comprehensive reply !!