Skip to content
This repository has been archived by the owner on Sep 2, 2023. It is now read-only.

Float to int conversion: Clamp the result to the integer range #79

Open
mbitsnbites opened this issue Feb 20, 2019 · 1 comment
Open

Comments

@mbitsnbites
Copy link
Member

mbitsnbites commented Feb 20, 2019

Instead of always returning 0xffffffff for every out-of-range / overflow situation, instead do this:

< min > max NaN
Signed −2^31 2^31 - 1 0
Unsigned 0 2^32 - 1 0

Note: Several data processing systems (OpenCL, CUDA, SIMD, ...) prefer that NaN translates to zero.

@mbitsnbites
Copy link
Member Author

According to: https://github.com/mbitsnbites/leanfloat

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant