Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discussion: use order_float instead of f32,f64 #116

Open
ZENOTME opened this issue Jul 24, 2023 · 7 comments
Open

discussion: use order_float instead of f32,f64 #116

ZENOTME opened this issue Jul 24, 2023 · 7 comments

Comments

@ZENOTME
Copy link
Contributor

ZENOTME commented Jul 24, 2023

To support partition write, we need something like HashMap<StructValue,_>. So this means that we need to impl Eq for AnyValue. To do this, we need to replace f32, f64 to order_float. (e.g. https://crates.io/crates/ordered-float)

As https://internals.rust-lang.org/t/f32-f64-should-implement-hash/5436/33#:~:text=If%20you%20want%20to%20have%20a%20content%20hash%20of%20a%20struct%20that%20includes%20floats says, to support Hash for the struct, we also can use serde. But I guess there may be other scenes we need Eq.

@ZENOTME
Copy link
Contributor Author

ZENOTME commented Jul 24, 2023

cc @liurenjie1024 @Xuanwo

@Xuanwo
Copy link
Contributor

Xuanwo commented Jul 24, 2023

I'm open with using order_float. But here are some edge cases to take care:

  • The order: spec said we need to use -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN.
  • The hash: iceberg has hash specs for float value that: hashLong(doubleToLongBits(double(v))

Maybe we should do something extra here instead of store StructValue in hashmap directly.

@liurenjie1024
Copy link
Contributor

I'm open with using order_float. But here are some edge cases to take care:

  • The order: spec said we need to use -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN.
  • The hash: iceberg has hash specs for float value that: hashLong(doubleToLongBits(double(v))

Maybe we should do something extra here instead of store StructValue in hashmap directly.

We may need to implement customized ordering trait, hash function.

@ZENOTME
Copy link
Contributor Author

ZENOTME commented Jul 25, 2023

There are two problem in https://crates.io/crates/ordered-float

  1. -0 == +0
  2. it don't have concept of -nan

I guess the only solve way is to maintain our own ordered_float.

@Xuanwo
Copy link
Contributor

Xuanwo commented Jul 25, 2023

I guess the only solve way is to maintain our own ordered_float.

Maybe it's easier to implement Hash and Eq by our own.

@ZENOTME
Copy link
Contributor Author

ZENOTME commented Jul 25, 2023

Maybe it's easier to implement Hash and Eq by our own.

Do you means that implement the custom behaviour in PrimitiveValue🤔, like

impl<T: Float> Ord for PrimitiveValue<T> {
    fn cmp(&self, other: &Self) -> Ordering {
        match (self,other) => {
            (Float(l),Float(r)) => {
                // custom behaviour
            }
        } 
    }
}

@liurenjie1024
Copy link
Contributor

Do you means that implement the custom behaviour in PrimitiveValue🤔, like

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants