Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] hardware, software, and counting




On 9/2/2014 12:54 AM, John Denker wrote: [in part]
/snip/ Taking the negative of a number does /not/ commute with truncating insignificant trailing digits. This creates a conceptual problem if you wanted to have a countdown timer with fractional-second resolution. /snip/ Post-launch, the display "00:07" indicates 7 /or more/ seconds elapsed. This is the result you would naturally get by truncating a more-detailed representation such as 7.777. Pre-launch, the same display ("00:07") indicates 7 /or less/ seconds remaining. Alas, this is *not* the result you would get by truncating a more-detailed representation such as 6.543. The post-launch case is easy, but we need to look closely at the pre-launch case. Pre-launch, during most of the interval in question, the time is "minus 6 point something". That is, "7" is displayed during the interval {-6.5 ± .5}. That gets you into trouble if you try to display any resolution beyond the decimal point, e.g. -6.543 or something like that. You cannot format the time as a decimal fraction and then look only at the part in front of the decimal point. Instead you have to start by taking the floor() of the number, so that -6.543 becomes -7, and then display that. Any sub-second resolution is lost. /snip/


Probably missed some crucial point in merely skimming this post - but it came to mind that people may retain the desired precision by multiplying a value by the desired number of decimal places, taking the floor(), dividing by the same factor, and displaying that - so that -6.54321 becomes -6543.21 which floors to -6544 and is displayed as -6.544

Brian Whatcott