Questions tagged [precision]
For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].
precision
5,379
questions
3967
votes
36
answers
587k
views
Is floating-point math broken?
Consider the following code:
0.1 + 0.2 == 0.3 -> false
0.1 + 0.2 -> 0.30000000000000004
Why do these inaccuracies happen?
- 45.2k
2409
votes
36
answers
5.7m
views
Limiting floats to two decimal points
I want a to be rounded to 13.95. I tried using round, but I get:
>>> a
13.949999999999999
>>> round(a, 2)
13.949999999999999
For the analogous issue with the standard library ...
kevin
1024
votes
7
answers
606k
views
decimal vs double! - Which one should I use and when? [duplicate]
I keep seeing people using doubles in C#. I know I read somewhere that doubles sometimes lose precision.
My question is when should a use a double and when should I use a decimal type?
Which type is ...
- 18.8k
539
votes
14
answers
1.2m
views
What is the difference between float and double?
I've read about the difference between double precision and single precision. However, in most cases, float and double seem to be interchangeable, i.e. using one or the other does not seem to affect ...
- 7,863
505
votes
14
answers
619k
views
JavaScript displaying a float to 2 decimal places
I wanted to display a number to 2 decimal places.
I thought I could use toPrecision(2) in JavaScript .
However, if the number is 0.05, I get 0.0500. I'd rather it stay the same.
See it on JSbin.
...
- 487k
452
votes
16
answers
709k
views
How do I print a double value with full precision using cout?
In my earlier question I was printing a double using cout that got rounded when I wasn't expecting it. How can I make cout print a double using full precision?
314
votes
4
answers
406k
views
How do I interpret precision and scale of a number in a database?
I have the following column specified in a database: decimal(5,2)
How does one interpret this?
According to the properties on the column as viewed in SQL Server Management studio I can see that it ...
- 28.5k
298
votes
29
answers
377k
views
How do you round a double in Dart to a given degree of precision AFTER the decimal point?
Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP's round() function.
The closest thing I can find in the Dart docs is double....
- 3,059
293
votes
12
answers
587k
views
Get DateTime.Now with milliseconds precision
How can I exactly construct a time stamp of actual time with milliseconds precision?
I need something like 16.4.2013 9:48:00:123. Is this possible? I have an application, where I sample values 10 ...
- 2,941
261
votes
5
answers
87k
views
Why are floating point numbers inaccurate?
Why do some numbers lose accuracy when stored as floating point numbers?
For example, the decimal number 9.2 can be expressed exactly as a ratio of two decimal integers (92/10), both of which can be ...
225
votes
14
answers
840k
views
Printing the correct number of decimal places with cout
I have a list of float values and I want to print them with cout with 2 decimal places.
For example:
10.900 should be printed as 10.90
1.000 should be printed as 1.00
122.345 should be printed as ...
- 9,395
208
votes
11
answers
357k
views
What's the difference between a single precision and double precision floating point operation?
What is the difference between a single precision floating point operation and double precision floating operation?
I'm especially interested in practical terms in relation to video game consoles. ...
- 22.6k
196
votes
3
answers
11k
views
Why is a round-trip conversion via a string not safe for a double?
Recently I have had to serialize a double into text, and then get it back. The value seems to not be equivalent:
double d1 = 0.84551240822557006;
string s = d1.ToString("R");
double d2 = double.Parse(...
- 1,591
194
votes
36
answers
29k
views
Unexpected results when working with very big integers on interpreted languages
I am trying to get the sum of 1 + 2 + ... + 1000000000, but I'm getting funny results in PHP and Node.js.
PHP
$sum = 0;
for($i = 0; $i <= 1000000000 ; $i++) {
$sum += $i;
}
printf("%s", ...
187
votes
24
answers
266k
views
Retain precision with double in Java
public class doublePrecision {
public static void main(String[] args) {
double total = 0;
total += 5.6;
total += 5.8;
System.out.println(total);
}
}
The above ...
- 4,009
169
votes
9
answers
80k
views
Difference between toFixed() and toPrecision()?
I'm new to JavaScript and just discovered toFixed() and toPrecision() to round numbers. However, I can't figure out what the difference between the two is.
What is the difference between number....
- 1,701
164
votes
9
answers
398k
views
Printf width specifier to maintain precision of floating-point value
Is there a printf width specifier which can be applied to a floating point specifier that would automatically format the output to the necessary number of significant digits such that when scanning ...
- 11.8k
162
votes
10
answers
18k
views
Is floating-point math consistent in C#? Can it be?
No, this is not another "Why is (1/3.0)*3 != 1" question.
I've been reading about floating-points a lot lately; specifically, how the same calculation might give different results on different ...
154
votes
3
answers
19k
views
Why does adding 0.1 multiple times remain lossless?
I know the 0.1 decimal number cannot be represented exactly with a finite binary number (explanation), so double n = 0.1 will lose some precision and will not be exactly 0.1. On the other hand 0.5 can ...
- 409k
153
votes
4
answers
328k
views
Controlling number of decimal digits in print output in R
There is an option in R to get control over digit display. For example:
options(digits=10)
is supposed to give the calculation results in 10 digits till the end of R session. In the help file of R, ...
- 10.3k
149
votes
13
answers
55k
views
PHP7.1 json_encode() Float Issue
This isn't a question as it is more of a be aware. I updated an application that uses json_encode() to PHP7.1.1 and I was seeing an issue with floats being changed to sometimes extend out 17 digits. ...
- 1,742
138
votes
5
answers
524k
views
Double precision floating values in Python?
Are there data types with better precision than float?
- 11k
134
votes
24
answers
129k
views
How to properly round-up half float numbers?
I am facing a strange behavior of the round() function:
for i in range(1, 15, 2):
n = i / 2
print(n, "=>", round(n))
This code prints:
0.5 => 0
1.5 => 2
2.5 => 2
3.5 => 4
4.5 ...
- 19.3k
117
votes
6
answers
161k
views
BigDecimal, precision and scale
I'm using BigDecimal for my numbers in my application, for example, with JPA. I did a bit of researching about the terms 'precision' and 'scale' but I don't understand what are they exactly.
Can ...
- 1,866
113
votes
11
answers
21k
views
In which order should floats be added to get the most precise result?
This was a question I was asked at my recent interview and I want to know (I don't actually remember the theory of the numerical analysis, so please help me :)
If we have some function, which ...
- 22.5k
113
votes
10
answers
90k
views
Is it safe to check floating point values for equality to 0?
I know you can't rely on equality between double or decimal type values normally, but I'm wondering if 0 is a special case.
While I can understand imprecisions between 0.00000000000001 and 0....
- 2,222
110
votes
7
answers
67k
views
C# DateTime.Now precision
I just ran into some unexpected behavior with DateTime.UtcNow while doing some unit tests. It appears that when you call DateTime.Now/UtcNow in rapid succession, it seems to give you back the same ...
- 87.6k
105
votes
11
answers
116k
views
Should I use double or float?
What are the advantages and disadvantages of using one instead of the other in C++?
- 96.1k
104
votes
6
answers
13k
views
How many double numbers are there between 0.0 and 1.0?
This is something that's been on my mind for years, but I never took the time to ask before.
Many (pseudo) random number generators generate a random number between 0.0 and 1.0. Mathematically there ...
- 381k
99
votes
1
answer
282k
views
Dividing two integers to produce a float result [duplicate]
Possible Duplicate:
Why can't I return a double from two ints being divided
My C++ program is truncating the output of my integer devision even when I try and place the output into a float. ...
- 10.8k
96
votes
8
answers
62k
views
PHP - Floating Number Precision [duplicate]
$a = '35';
$b = '-34.99';
echo ($a + $b);
Results in 0.009999999999998
What is up with that? I wondered why my program kept reporting odd results.
Why doesn't PHP return the expected 0.01?
- 1,295
89
votes
12
answers
14k
views
Endless sine generation in C
I am working on a project which incorporates computing a sine wave as input for a control loop.
The sine wave has a frequency of 280 Hz, and the control loop runs every 30 µs and everything is written ...
- 897
88
votes
4
answers
82k
views
What's the C++ suffix for long double literals?
In C++ (and C), a floating point literal without suffix defaults to double, while the suffix f implies a float. But what is the suffix to get a long double?
Without knowing, I would define, say,
...
- 45k
87
votes
7
answers
109k
views
What range of numbers can be represented in a 16-, 32- and 64-bit IEEE-754 systems?
I know a little bit about how floating-point numbers are represented, but not enough, I'm afraid.
The general question is:
For a given precision (for my purposes, the number of accurate decimal ...
- 14.7k
86
votes
8
answers
92k
views
Is there any way to get current time in nanoseconds using JavaScript?
So, I know I can get current time in milliseconds using JavaScript. But, is it possible to get the current time in nanoseconds instead?
- 3,112
83
votes
2
answers
6k
views
Why is 199.96 - 0 = 200 in SQL?
I have some clients getting weird bills. I was able to isolate the core problem:
SELECT 199.96 - (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)))) -- 200 what the?
SELECT ...
- 1,483
73
votes
2
answers
28k
views
In MATLAB, are variables REALLY double-precision by default?
This question arose out of something strange that I noticed after investigating this question further...
I always understood MATLAB variables to be double-precision by default. So, if I were to do ...
- 126k
72
votes
5
answers
26k
views
Why is 24.0000 not equal to 24.0000 in MATLAB?
I am writing a program where I need to delete duplicate points stored in a matrix. The problem is that when it comes to check whether those points are in the matrix, MATLAB can't recognize them in the ...
- 8,226
72
votes
5
answers
24k
views
Are all integer values perfectly represented as doubles? [duplicate]
My question is whether all integer values are guaranteed to have a perfect double representation.
Consider the following code sample that prints "Same":
// Example program
#include <iostream>
...
- 4,994
71
votes
8
answers
70k
views
Adjusting decimal precision, .net
These lines in C#
decimal a = 2m;
decimal b = 2.0m;
decimal c = 2.00000000m;
decimal d = 2.000000000000000000000000000m;
Console.WriteLine(a);
Console.WriteLine(b);
Console.WriteLine(c);
Console....
- 109k
70
votes
8
answers
17k
views
Emulate "double" using 2 "float"s
I am writing a program for an embedded hardware that only supports 32-bit single-precision floating-point arithmetic. The algorithm I am implementing, however, requires a 64-bit double-precision ...
user210870
67
votes
4
answers
128k
views
Python: Find index of minimum item in list of floats [duplicate]
How can I find the index of the minimum item in a Python list of floats? If they were integers, I would simply do:
minIndex = myList.index(min(myList))
However, with a list of floats I get the ...
- 5,042
67
votes
4
answers
118k
views
Convert Java Number to BigDecimal : best way
I am looking for the best way to convert a Number to a BigDecimal.
Is this good enough?
Number number;
BigDecimal big = new BigDecimal(number.toString());
Can we lose precision with the toString() ...
user1721413
63
votes
5
answers
88k
views
Set the display precision of a float in Ruby
Is it possible to set the display precision of a float in Ruby?
Something like:
z = 1/3
z.to_s #=> 0.33333333333333
z.to_s(3) #=> 0.333
z.to_s(5) #=> 0.33333
Or do I have to override ...
- 22.1k
63
votes
2
answers
6k
views
Why does double in C print fewer decimal digits than C++?
I have this code in C where I've declared 0.1 as double.
#include <stdio.h>
int main() {
double a = 0.1;
printf("a is %0.56f\n", a);
return 0;
}
This is what it prints, a is 0....
59
votes
6
answers
140k
views
Set specific precision of a BigDecimal
I have an XSD that requires me to use a BigDecimal for a lat/lon. I currently have the lat/lon as doubles, and convert them to BigDecimal, but I am only required to use about 12 places of precision. ...
- 2,167
59
votes
7
answers
148k
views
Double precision - decimal places
From what I have read, a value of data type double has an approximate precision of 15 decimal places. However, when I use a number whose decimal representation repeats, such as 1.0/7.0, I find that ...
- 4,179
56
votes
10
answers
18k
views
Are doubles faster than floats in C#?
I'm writing an application which reads large arrays of floats and performs some simple operations with them. I'm using floats, because I thought it'd be faster than doubles, but after doing some ...
- 12.3k
54
votes
5
answers
75k
views
How to calculate precision and recall in Keras
I am building a multi-class classifier with Keras 2.02 (with Tensorflow backend)，and I do not know how to calculate precision and recall in Keras. Please help me.
- 555