How does Prism round decimal values on data tables?

Numbers with many digits

When you enter data with more than seven digits, Prism will store only seven (actually it is binary, so it is 'about' seven decimal digits) so will not store the exact value you entered. For example, if you enter the value 99.2492427 into a data table, Prism rounds it to 99.249250. Prism stores numbers with single precision, so it can only store about seven significant figures. If you enter more, Prism rounds off. The conversion from decimal to binary and back means that the rounding is not always what you'd predict.

How Prism rounds

By default, Prism tries to show all the digits you entered, and increases the number of digits after the decimal place to accommodate the extra digits. But you can use the Format Numbers dialog to show fewer digits. And the results of analyses can be rounded as well. How Prism rounds values can seem erratic. 

Prism rounds values that end with 5 somewhat unpredictably, as this table demonstrates. The left column shows the digits you enter into Prism, and the right column shows those that Prism displays on the data table when you use the Number Format dialog to show one fewer digit than you entered. These are all correct, since rounding a 5 up or down is an arbitrary decision. 

The problem of rounding

Rounding values is straightforward unless the next digit is 5. Should 2.5 be rounded up to 3 or rounded down to 2? Both are equally accurate, and many strategies have been proposed to deal with this issue. The Wikipedia article on rounding explains the many possibilities. 

Entering numbers into computers adds another layer of complexity. Computers store the number as binary digits, so need to convert to and from decimal. For many values, this requires rounding as well. For example, let's say you type in the number 0.055, and ask Prism to display the value to only two decimal places. The next, third, digit is 5. One common rule to always round up when the next digit is 5, so you'd expect Prism to display 0.06. But the computer can't store the value 0.055. Instead, all values are converted to binary, so this value is stored as 0.05499996. When rounding this value to two decimal places, the next (third) digit is a 4. So Prism rounds down and displays 0.05. 

Prism Windows uses the C function sprintf() to display real numbers.  This Microsoft knowledgebase article explains how this function rounds.  The Mac Xcode compiler seems to use an identical function. 

Note that Microsoft Excel is different. It uses two methods,  one when the calculation is done within a worksheet and another when done by a macro. 

It doesn't really matter in most cases

What you see on the Prism data table matters only when you look at that table. Note:

• Choosing to show fewer digits on the data table won't affect how Prism analyzes or graphs your data. Analyses are always based on all the digits you entered (up to the seven Prism can store).
• When you save the Prism file (as .PZF or .PZFX), Prism saves all the digits you entered (up to seven), so the values are not truncated when you save.
• When you copy data from a Prism data table to the clipboard, Prism copies the values you entered, not only the digits you see on the table.
• When you export a data table as a .TXT or .CSV file, Prism exports the digits you entered, and not only those that appear on the table.