Skip to main content

Posts

Showing posts from March, 2022

The Fun of Floating Point Numbers in one Image

Programming with floating point is always fun. Here's a nice little screen capture summarizing the insanity that sometimes arises: .NET keeps 9 digits of precision internally, but typically only displays 7 digits of precision, so I had a hell of a time figuring out why a value from what's effectively a no-op was exceeding the 0.33F threshold I was looking for. Losing equational reasoning is always fun, but this is even more bizarre than usual. Yay floating point!