Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternative fallback fields implementation #441

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

LoopyAshy
Copy link

@LoopyAshy LoopyAshy commented Oct 21, 2022

This is an alternative implementation to #440.
This implementation is notably different in that it is more controllable by the implementor and feels more like a first class feature.
The main changes are the addition of types deriving Any to state the priority of dynamic fields vs actual fields. The options are as follows:

  • First - Will try to find a dynamic field first before checking static fields.
  • Last - Will try to find a static field first before checking dynamic fields.
  • Never - Will completely disregard any dynamic fields and exclusively use static fields. (Same behavior as prior to this implementation and the default)
  • Only - Will completely disregard static fields and exclusively use dynamic fields.

All of these can be applied to any type using the Derive(Any) macro by applying the attribute
#[rune(dynamic_fields="never|only|first|last")] to the struct itself.

Dynamic fields functionality is basically the same as the #440 however due to the fact First and Only exist means there needed to be a change to their implementation.
DYNAMIC_FIELD_GET now handles Options slightly differently: if you return None it will presume there was no field, Some(None) is still a option if desired. The value will be unwrapped from the Option on the callers side so doesn't need to be worried about by the script writer.
DYNAMIC_FIELD_SET is basically the same as DYNAMIC_FIELD_GET however instead of a Option you can choose to return false which will be treat as a failure to set the field, allowing it to try and set the static field next if the type is First or simply return a error otherwise as expected with static fields.
For both of the above changes to DYNAMIC_FIELD_SET and DYNAMIC_FIELD_GET: you can return any other type (including () of course) and it will not treat it any differently to before.

Examples

A struct with dynamic fields which are prioritised over static fields:

#[derive(Any, Clone)]
#[rune(dynamic_fields="first")]
struct TestStruct {
    values: HashMap<String, u32>,
    b: u32, 
    c: u32
}

implementing fallible DYNAMIC_FIELD_GET and DYNAMIC_FIELD_SET

let mut module = Module::new();

module.ty::<TestStruct>()?;
module.inst_fn(
    Protocol::DYNAMIC_FIELD_GET,
    |this: &TestStruct, key: String| {
        if let Some(v) = this.values.get(&key) {
            Some(*v)
        } else {
	    None
	}
    },
)?;
module.inst_fn(
    Protocol::DYNAMIC_FIELD_SET,
    |this: &mut TestStruct, key: String, value: u32| {
        if key == "nerd" {
            false
        } else {
            this.values.insert(key, value);
            true
	}
    },
)?;

Reasoning

Sometimes it is desired for user readability to pretend a specific custom type has fields and instead accessing the data from elsewhere such as from a HashMap or another collection.
possible use cases is having types with dynamic layouts while having rules to their usage such as locking the layout once created.
it also allows custom types to be dynamically extended by the scripter while keeping the benefits of being a static Shared<AnyObj> but not seeming like a collection which [] accesses give the impression of.

Benchmarks

Note
Due to the tiny footprints of the functions, the differences are very hard to see if any and can be improved within my benchmarks with a faster hashmap regardless but honestly values these small are easily within margin of error.

Before

test fields::actual_field_get                    ... bench:         418 ns/iter (+/- 88)
test fields::actual_field_set                    ... bench:         434 ns/iter (+/- 161)

test fields::string_key_classic_index_get        ... bench:         523 ns/iter (+/- 52)
test fields::static_string_key_classic_index_get ... bench:         486 ns/iter (+/- 80)

After

test meta_fields::actual_field_get_first             ... bench:         614 ns/iter (+/- 18)
test meta_fields::actual_field_get_last              ... bench:         420 ns/iter (+/- 46)
test meta_fields::actual_field_get_never             ... bench:         414 ns/iter (+/- 27)
test meta_fields::actual_field_set_first             ... bench:         514 ns/iter (+/- 63)
test meta_fields::actual_field_set_last              ... bench:         427 ns/iter (+/- 14)
test meta_fields::actual_field_set_never             ... bench:         408 ns/iter (+/- 28)
test meta_fields::actual_field_set_only              ... bench:         515 ns/iter (+/- 16)
test meta_fields::static_string_index_get_first      ... bench:         467 ns/iter (+/- 14)
test meta_fields::static_string_index_get_last       ... bench:         471 ns/iter (+/- 35)
test meta_fields::static_string_index_get_never      ... bench:         465 ns/iter (+/- 43)
test meta_fields::static_string_index_get_only       ... bench:         459 ns/iter (+/- 17)
test meta_fields::static_string_meta_field_get_first ... bench:         531 ns/iter (+/- 64)
test meta_fields::static_string_meta_field_get_last  ... bench:         546 ns/iter (+/- 31)
test meta_fields::static_string_meta_field_get_only  ... bench:         518 ns/iter (+/- 35)
test meta_fields::static_string_meta_field_set_first ... bench:         525 ns/iter (+/- 231)
test meta_fields::static_string_meta_field_set_last  ... bench:         545 ns/iter (+/- 42)
test meta_fields::static_string_meta_field_set_only  ... bench:         515 ns/iter (+/- 34)
test meta_fields::string_index_get_first             ... bench:         504 ns/iter (+/- 24)
test meta_fields::string_index_get_last              ... bench:         505 ns/iter (+/- 42)
test meta_fields::string_index_get_never             ... bench:         504 ns/iter (+/- 17)
test meta_fields::string_index_get_only              ... bench:         501 ns/iter (+/- 20)
test meta_fields::string_meta_field_get_first        ... bench:         552 ns/iter (+/- 21)
test meta_fields::string_meta_field_get_last         ... bench:         595 ns/iter (+/- 26)
test meta_fields::string_meta_field_get_only         ... bench:         558 ns/iter (+/- 26)
test meta_fields::string_meta_field_set_first        ... bench:         508 ns/iter (+/- 23)
test meta_fields::string_meta_field_set_last         ... bench:         553 ns/iter (+/- 39)
test meta_fields::string_meta_field_set_only         ... bench:         510 ns/iter (+/- 23)

Closes

#381 #263

@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

I'm gonna have to think this over. It's not a style choice that I would choose, so if I include it I'd like to see the following changes done:

  • Introduce your feature as extra Vm instructions which are feature gated. So instead of rewriting the ObjectSlotIndexGet instruction, add a new one which uses the new field loading convention.
  • Feature gate the extra plumbing in AnyObj which allows for dynamic index get conventions.
  • Add a compiler option which changes how index gets are performed so that they make use of the new feature. This would allow you to use the new feature, but it wouldn't be enabled by default.

That way the feature can be tested and refined in parallel.

Concerns

  1. Dynamically deciding on the field loading convention in use is a form of metaprogramming which doesn't obviously mesh well with a future gradual type system. 2. Having dynamic field loading conventions might make future compile time optimizations harder or not possible.

Both of these are future potentials and not current blockers, which is why I wouldn't mind including the feature on an experimental basis to get a feel for it. Maybe improve it and mainline include in the future.

Thank you!

@udoprog udoprog added enhancement New feature or request lang Issues related to language design. labels Oct 23, 2022
@LoopyAshy
Copy link
Author

I'm gonna have to think this over. It's not a style choice that I would choose, so if I include it I'd like to see the following changes done:

  • Introduce your feature as extra Vm instructions which are feature gated. So instead of rewriting the ObjectSlotIndexGet instruction, add a new one which uses the new field loading convention.
  • Feature gate the extra plumbing in AnyObj which allows for dynamic index get conventions.
  • Add a compiler option which changes how index gets are performed so that they make use of the new feature. This would allow you to use the new feature, but it wouldn't be enabled by default.

That way the feature can be tested and refined in parallel.

Concerns

  1. Dynamically deciding on the field loading convention in use is a form of metaprogramming which doesn't obviously mesh well with a future gradual type system. 2) Having dynamic field loading conventions might make future compile time optimizations harder or not possible.

Both of these are future potentials and not current blockers, which is why I wouldn't mind including the feature on an experimental basis to get a feel for it. Maybe improve it and mainline include in the future.

Thank you!

Thank you for your reply. I am happy to make those changes and will do so when I am free.

Regarding you concerns: I do believe it shouldn't make it completely impossible but I can see the possibility of increasing the complexity of the compilation, if it does come to that I am however more than happy to try and assist in the ease of implementation as this is a QoL feature that I know will be very useful in quite a few of my projects for example in one of my projects the types are generated at load time and stored as templates and generated dynamically at runtime when requested and are optimised around these facts.
Reason for this is so the project doesn't need recompiled as it relies on third party data types which are very often change but the layouts are available, however there layouts are statically typed so if I gave a float instead of a int it would be incorrect, so I have to check the typing and field name validity etc.

When I was using Lua for this project I simply overwrote the meta function for getting and setting fields and it worked like any other type and looked no different, it was seamless and effective.
When moving to rune due to it having many many preferred design decisions over lua, it was only possible by having dedicated instance functions or using INDEX_* which means using ["field_name"] which makes it look strange to the script writers who think of these types are true types not HashMaps or collections.

@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 23, 2022

I have added the requested changes along with making the naming more consistent (renaming the feature to dynamic fields from meta fields)
Let me know what you think and if anything requires changes or clean up to meet the desired structure.
I have run the previous benchmarks and no changes have been found with the previous implementation.

I want to note it was actually fun to apply these changes as it allowed me to further understand the source and its flow.

Copy link
Collaborator

@udoprog udoprog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very neat, and thanks for the quick turnaround!

So most of the feature flagging can be avoided. All that really needs to be feature flagged is:

  • The option that enables the use of the different field get/set instructions in Options (see comment).
  • The enum variants in DynamicFieldMode which are not Never (see comment).
  • Use of those other variants when deriving Any in rune-macros (and the appropriate error if the feature is not enabled).
  • The AnyObj runtime support (just nice to avoid the vtable increase if it's not used but not super important). see comment.

It is perfectly all right for the instructions to exist alongside other instructions in the Vm. As long as their use is behind an option.

@@ -18,7 +22,32 @@ pub use rune_macros::Any;
/// name: String,
/// }
/// ```
pub trait Any: Named + MetaFieldMode + std::any::Any {
#[cfg(not(feature = "dynamic_fields"))]
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, avoid feature gating two incompatible traits of Any. What can happen is that you have coherence issues under a scenario such as:

Crate a defines a type that implements Any without activating the dynamic_fields feature, so they leave FIELD_MODE undefined.

Crate b defines a type that implements Any while activating the dynamic_fields feature, so they define FIELD_MODE.

Crate c depends on a and b, so the dynamic_fields feature is activated, causing a to fail to build.

Instead:

  • Unconditionally include FIELD_MODE (and possibly rename to DYNAMIC_FIELD_MODE :)).
  • Feature gate the other variants of DynamicFieldMode which are expected to cause alternate behaviors (First, Last, and Only).
  • Mark DynamicFieldMode as #[non_exhaustive]. This prevents dependents which have not enabled the feature from using it in a manner which is incompatible with the feature being enabled.

@@ -61,6 +65,10 @@ impl Options {
Some("v2") => {
self.v2 = it.next() != Some("false");
}
#[cfg(feature = "dynamic_fields")]
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be confusing if it fails to parse despite being in the documentation.

Instead:

  • Return a special parse error indicating that the option exists, but it's not supported without enabling a feature; or,
  • Emit a warning through Diagnostics during a build if the dynamic_fields option is used but the feature is disabled.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call.

#[cfg(feature = "dynamic_fields")]
let meta_fields = Some(quote! {Never});
#[cfg(not(feature = "dynamic_fields"))]
let meta_fields = None;
Copy link
Collaborator

@udoprog udoprog Oct 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some additional simplifications in rune-macros would be possible when we go for the other feature gating strategy I'm proposing. So I'm not gonna comment on them now ;).

@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

The AnyObj runtime support (just nice to avoid the vtable increase if it's not used but not super important).

Actually, no this can't be done. AnyObjVtable is public API! So it would suffer the same issues as the Any trait. The field needs to be unconditionally included.

@LoopyAshy
Copy link
Author

I have applied the requested changes, did take me a while due to how long I been coding today lol so sorry if I made any silly mistakes but all the tests passed and the benches are stable.

@udoprog
Copy link
Collaborator

udoprog commented Oct 23, 2022

Don't sweat it! And don't feel like you have to make requested changes immediately. I don't have any plans to change things in Rune for at least a week or two ;).

@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 23, 2022

Don't sweat it! And don't feel like you have to make requested changes immediately. I don't have any plans to change things in Rune for at least a week or two ;).

Oh it is not for that reason lol, I just like getting stuff sorted, and when I get stuck in I usually stay focused on the task for good while. I am just glad it is all going smoothly; this is the first public repo I ever made a PR for due to confidence issues honestly, private is another thing all together though :D.

Copy link
Collaborator

@udoprog udoprog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good, a few nits!

@@ -50,7 +50,7 @@
//! [rune]: https://github.com/rune-rs/rune

use anyhow::{anyhow, Result};
use rune::compile::ParseOptionError;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to rename the error type, you've only turned it into an enum which is still one error (one of its variants). "Errors" gives the impression that it's a collection.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem I can change that easily.

self.errors.push(syn::Error::new_spanned(
s,
"attempted to set dynamic_fields to `only` while the feature `dynamic_fields` is disabled."
));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation here seems a bit off?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Funnily enough it looked perfect until cargo fmt decided to make it look like that lol, I'll appease it shortly.

attrs.meta_fields = Some(match s.value().as_str() {
"never" => format_ident!("Never"),
"only" => {
if cfg!(feature = "dynamic_fields") {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this kind of test is repeated three times it might be worth breaking out into a function.


/// Allow use of META_FIELD_GET and META_FIELD_SET
#[cfg(feature = "dynamic_fields")]
pub dynamic_fields: bool,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pub(crate) since it has a mutator method (else we can set it directly).

The mutator method can be feature gated. This field does not need to be.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking that but I wasn't 100%.

required_feature: "dynamic_fields",
});
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if cfg!(feature = "dynamic_fields") { /* */ } else { /* */ } is probably cleaner here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if cfg!(feature = "dynamic_fields") { /* */ } else { /* */ } is probably cleaner here.

Yeah easily sorted; it is just like this due to ObjectIndexDynamicGet originally being feature gated so it wouldn't compile with that if statement lol (kind of wish rust had a macro which mimicked if else cfg macro but actually ignored the invalid branch since its more sane to read.

},
span,
);
#[cfg(not(feature = "dynamic_fields"))]
c.asm.push(Inst::ObjectIndexSet { slot }, span);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving into a function which is feature gated with two different implementations instead?

Block-level feature gates are a bit awkward to deal with in my experience.

Something like:

push_field_lookup(c, slot);

},
span,
);
#[cfg(not(feature = "dynamic_fields"))]
c.asm.push(Inst::ObjectIndexGet { slot }, span);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

/// Search for a field by `String`
pub const DYNAMIC_FIELD_GET: Protocol = Protocol {
name: "dynamic_field_get",
hash: Hash::new(0x6dda58b140dfeaf9),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how did you generate these?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how did you generate these?

I used the Inst hasher and used a name which wouldn't be allowed as a field name.

CallResult::Unsupported(target) => CallResult::Unsupported(target),
}
}
#[cfg(feature = "dynamic_fields")]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a huge fan of these big featured off blocks but not much to be done about them either. Maybe consider if these sections can be cleaned up a bit (somehow) in the future. But I'm not sure how right now.

}
} else {
CallResult::Ok(result)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My preferred style in instances like thse would be this to save some of the indentation:

let result = match result {
    Value::Option(result) => result,
    _ => return Ok(CallResult::Ok(result)),
};

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally I agree, was just bit tired during the coding session lol.

@LoopyAshy
Copy link
Author

I'll add the requested changes when I get back later as I am busy today, thank you for the review <3

@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 28, 2022

I been busy past few days but I added the requested changes and forgot to mention it lol, I also added a tiny idea for the clean up but admittedly I ain't the most fond of it and I am happy to revert that change if you feel it is too out of place, honestly though it is quite fiddly to clean up admittedly.

I have been using these changes on one of my projects to test the usability and ease of implementation etc, and honestly it has been a very smooth experience and I haven't ran into any problems yet, which is a wonderful sentence to type lol.

@LoopyAshy
Copy link
Author

LoopyAshy commented Oct 28, 2022

My bad, I genuinely completely forgot to push my changes to tests. I am surprised I didn't notice sooner as I had them ready but I ran the tests myself and completely blanked out lol.

Fixed now <3

Honestly though I should really clean the tests up later as they are horrible to read and refactor.

@udoprog
Copy link
Collaborator

udoprog commented Oct 30, 2022

I'll be back after this weekend and then I'll take a look!

Cheers!

@LoopyAshy
Copy link
Author

I have updated the PR to the latest version of rune however I have questions before we decide to add this. mostly in regards to naming; what name for this feature would be most appropriate for it? Fallback_Set/Get, Meta_Set/Get, Dynamic_Set/Get, Dynamic_Field_Set/Get, Metafield_Set/Get, etc, etc. there is many options and I feel it should be consistent throughout but I was unable to decide a name I was completely satisfied with.

@LoopyAshy LoopyAshy reopened this Oct 31, 2023
@udoprog
Copy link
Collaborator

udoprog commented Nov 1, 2023

Naming can always be punted on, initially this will be released as a --cfg flag, because enabling it as a feature could end up changing the behavior of the project in unrelated places. Features either have to be adopted wholesale and define the semantics of the language or optionally opted into like that.

That being said, after digging into this now I'm prone to cutting down this feature by only supporting DYNAMIC_FIELD_GET / DYNAMIC_FIELD_SET. If any of these protocols are registered it indicates that the implementer takes over full control of field getting and setting.

I find the semantics as it currently stands with the different modes to be more confusing to describe than such a method, and both would ultimately support the same set of use cases.
E.g.
If first is set and DYNAMIC_FIELD_GET returns None then it falls back to GET.

There's also a lack of reflexivity in the features. Only DYNAMIC_FIELD_GET supports falling back to default GET, DYNAMIC_FIELD_SET does not because that would involve some kind of sentinel value indicating whether the field was set or not.

So this is an instance where I think it's better to define the behavior in code, rather than metadata since it both leads to clearer semantics, and a smaller change. To this end I've pushed a suggestion I'd like you to take for a spin.

@LoopyAshy
Copy link
Author

LoopyAshy commented Nov 1, 2023

I am fine with this. I must ask though: Do you basically think that instead of modes existing just simply have a flag and treat all DYNAMIC_FIELD_GET's and DYNAMIC_FIELD_SET's as First? or do you think Only would be more appropriate, the reasoning being that I had Last, First etc was purely for times when you feel the type will be more likely used with DYNAMIC_FIELD_GET vs GET.
I could if First becomes only option also potentially make DYNAMIC_FIELD_SET handle returning Option (same as DYNAMIC_FIELD_GET) to fallback to SET which to be honest I thought I had implemented regardless. Just checked my DYNAMIC_FIELD_SET implementation and it does allow fallback but it relies on a bool (false means fallback) but I can change that to be clearer if you feel it would be appropriate.

@udoprog
Copy link
Collaborator

udoprog commented Nov 1, 2023

What I intended was that if DYNAMIC_FIELD_GET is defined for a type, it's responsible for returning the field. It will not internally fall back to GET.

Just checked my DYNAMIC_FIELD_SET implementation and it does allow fallback but it relies on a bool (false means fallback) but I can change that to be clearer if you feel it would be appropriate.

I missed this! But the broader point would still be that the semantics are bit messy. If an implementor wants to fall back to e.g. calling GET themselves I think it's cleaner to give them access to the equivalent of Python's getattr rather than complicating the language implementation.

@LoopyAshy
Copy link
Author

LoopyAshy commented Nov 1, 2023

That is all good I will just have to think of how to implement the function so it's easily known to the users and works properly. Only reason I had it implemented the way I did was it didn't require any extra thought by the implementer but I can see the desire for it to be more explicit and customizable.

My main concern which was the rationale behind my implementation choices was to avoid paying for what you don't use, aka if a type has no dynamic fields then checking if it does by going through a lookup etc would obviously come with a cost which would be pointless and potentially cache dirtying, especially when custom field logic is arguably rare in when it is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request lang Issues related to language design.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants