Add more documentation

[hube12]

Hi,
Since I tinkered a lot with how to do my specific stuff, I have to write this issue, I will try to describe what I wished to achieve and what I managed to do, this would be a great starting point for people that like nice dual way serialization/deserialization. At the end I will also mentions current improvement to the pretty config (please READ them), I would love a page dedicated to those specific in the wiki or discussion page (you can take this whole article if you want, I give you all the right on it).

My use case is pretty simple, I have a config file I made manually and I need to load it in a specific structure, then add fields and populate it as it goes then write it back. But since it is half/half manual/automatic work, I need to work with a standard format on both side. However my structure is quite big and doesn't have all its fields everytime, but having such in my config would be tiresome, so I had to jump into serde! Let's take a real use case:

use serde::{Serialize, Serializer, Deserialize, Deserializer};
use std::collections::HashMap;

#[derive(Debug, Deserialize, Serialize)]
pub struct Game {
    macros: HashMap<String, Macros>, // those macros are applied to the actions
    actions: HashMap<String, String>, // in this version the actions are not described for conciseness
}

#[derive(Debug, Deserialize, Serialize)]
pub struct Macros {
    superseded: Option<String>, // this is the macros that is extended
    subject: SubjectMapping, // we want to force a subject even if the subject has no element
    action: Option<ActionMapping>, // we don't really care if there is an action
    how: Vec<String>, // this is either empty or filled
}

#[derive(Debug, Deserialize, Serialize)]
pub struct SubjectMapping {
    primary: Option<String>,
    secondary: Option<String>,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct ActionMapping {
    primary: Option<String>,
}

game.ron :

(
    macros:{
       "defaults" : (
           action:()
       ),
       "hero" : (
           superseded: "defaults",
           subject: (
             primary: "hero",
             secondary: "paladin",
           ),
           action:(
             primary: "attack"
           ),
           how: ["sword","magic"],
       ),
    },
    actions:{}
)

As we can see our structure is semi filled, so if we try to deserialize it we will face some issues (I will use here include_str but as this is a dynamic asset you want to use a bufreader and from_reader.

#[test]
fn test_deserialize() {
    const GAME: &str = include_str!("../constants/game.ron");
    use ron::from_str;
    let r: Result<Game, _> = from_str(GAME);
    dbg!(&r);
    assert!(r.is_ok());
}

We get a big error

[/src/util/test.rs:75] &r = Err(
    Error {
        code: ExpectedOption,
        position: Position {
            line: 4,
            col: 19,
        },
    },
)

That's actually an easy one, you just need to add those two enable extensions (see docs/Extensions.md) at the top of your file see #281 for why this is not yet in current release. #![enable(implicit_some,unwrap_newtypes)] >> game.ron
Now we recompile our example and we face our first challenge, we are missing a field (like I said our config data is only filled with the necessary informations.

[/src/util/test.rs:75] &r = Err(
    Error {
        code: Message(
            "missing field `subject`",
        ),
        position: Position {
            line: 0,
            col: 0,
        },
    },
)

Ok that's an easy one, you see serde needs to know what to do with missing fields, so we need to add #[serde(default)] to the subject: SubjectMapping, field (see https://serde.rs/attr-default.html for more informations) but we notice immediatly something we are missing a Default trait for SubjectMapping.

#[derive(Debug, Deserialize, Serialize)]
   |                 ^^^^^^^^^^^ the trait `Default` is not implemented for `util::test::SubjectMapping`

Thus we add it:

#[derive(Debug, Deserialize, Serialize,Default)]
pub struct SubjectMapping {
    primary: Option<String>,
    secondary: Option<String>,
}

Ok now that we have that done we move to the next error

[src/util/test.rs:76] &r = Err(
    Error {
        code: Message(
            "missing field `how`",
        ),
        position: Position {
            line: 0,
            col: 0,
        },
    },
)

Since how has no default option and is not an Option (thus filled with the extension with None) then we need to tinker with serde again and force it to default with #[serde(default)] again.

Ok we compile that and we have now:

#[derive(Debug, Deserialize, Serialize)]
pub struct Game {
    macros: HashMap<String, Macros>, // those macros are applied to the actions
    actions: HashMap<String, String>, // in this version the actions are not described for conciseness
}

#[derive(Debug, Deserialize, Serialize)]
pub struct Macros {
    superseded: Option<String>, // this is the macros that is extended
    #[serde(default)]
    subject: SubjectMapping, // we want to force a subject even if the subject has no element
    action: Option<ActionMapping>, // we don't really care if there is an action
    #[serde(default)]
    how: Vec<String>, // this is either empty or filled
}

#[derive(Debug, Deserialize, Serialize,Default)]
pub struct SubjectMapping {
    primary: Option<String>,
    secondary: Option<String>,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct ActionMapping {
    primary: Option<String>,
}

[src/util/test.rs:77] &r = Ok(
    Game {
        macros: {
            "hero": Macros {
                superseded: Some(
                "defaults",
                ),
                subject: SubjectMapping {
                    primary: Some(
                        "hero",
                    ),
                    secondary: Some(
                        "paladin",
                    ),
                },
                action: Some(
                    ActionMapping {
                        primary: Some(
                            "attack",
                        ),
                    },
                ),
                how: [
                    "sword",
                    "magic",
                ],
            },
            "defaults": Macros {
                superseded: None,
                subject: SubjectMapping {
                    primary: None,
                    secondary: None,
                },
                action: Some(
                    ActionMapping {
                        primary: None,
                    },
                ),
                how: [],
            },
        },
        actions: {},
    },
)

That's a perfect structure, we got it, we can now use it in rust, however I said earlier we actually want to serialize it back to the original form, right? Yeah if we serialize it back we will have a real surprise since we will have litterally all the None in it.

Let's use this function to deserialize then serialize, you will notice we use prettyConfig extensions and some options, all of which are documented, since I don't link numbering on my array I deactivated it and also I made sure to have IMPLICIT_SOME and UNWRAP_NEWTYPES even tho in this process they are not that useful (the structure already has all the option in it and the type are wrapped in their correct structure.

#[test]
fn test_reserialize() {
    const GAME: &str = include_str!("../constants/game.ron");
    use ron::extensions::Extensions;
    use ron::from_str;
    let r: Game = from_str(GAME).expect("no issue on deserialization");
    dbg!(&r);
    use ron::ser::{PrettyConfig, to_string_pretty};
    let pretty = PrettyConfig::new()
        .with_separate_tuple_members(true)
        .with_enumerate_arrays(false)
        .with_extensions(Extensions::IMPLICIT_SOME)
        .with_extensions(Extensions::UNWRAP_NEWTYPES);
    let r = to_string_pretty(&r, pretty);
    assert!(r.is_ok());
    assert!(r.is_ok());
    let r=r.unwrap();
    print!("{}",r);
    assert_eq!(GAME, r);
}

And yeah as expected we don't get the same input at all, it's pretty messy ;).

(
    macros: {
        "hero": (
            superseded: Some("defaults"),
            subject: (
                primary: Some("hero"),
                secondary: Some("paladin"),
            ),
            action: Some((
                primary: Some("attack"),
            )),
            how: [
                "sword",
                "magic",
            ],
        ),
        "defaults": (
            superseded: None,
            subject: (
                primary: None,
                secondary: None,
            ),
            action: Some((
                primary: None,
            )),
            how: [],
        ),
    },
    actions: {},
)

Ok we need to remove the None element so for instance superseded doesn't need to be in defaults at all. Again we are going to use serde and the skip_serializing_if macro, here with if it is None then we don't show it. ( https://serde.rs/field-attrs.html#skip_serializing_if )

#[serde(skip_serializing_if = "Option::is_none")]
superseded: Option<String>, // this is the macros that is extended

Now if we look at the "default" key serialized we get:

"defaults": (
            subject: (
                primary: None,
                secondary: None,
            ),
            action: Some((
                primary: None,
            )),
            how: [],
        ),

which is already much better, ok let's do that for all the option fields:

#[derive(Debug, Deserialize, Serialize)]
pub struct Game {
    macros: HashMap<String, Macros>, // those macros are applied to the actions
    actions: HashMap<String, String>, // in this version the actions are not described for conciseness
}

#[derive(Debug, Deserialize, Serialize)]
pub struct Macros {
    #[serde(skip_serializing_if = "Option::is_none")]
    superseded: Option<String>, // this is the macros that is extended
    #[serde(default)]
    subject: SubjectMapping, // we want to force a subject even if the subject has no element
    #[serde(skip_serializing_if = "Option::is_none")]
    action: Option<ActionMapping>, // we don't really care if there is an action
    #[serde(default)]
    how: Vec<String>, // this is either empty or filled
}

#[derive(Debug, Deserialize, Serialize,Default)]
pub struct SubjectMapping {
    #[serde(skip_serializing_if = "Option::is_none")]
    primary: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    secondary: Option<String>,
}

#[derive(Debug, Deserialize, Serialize)]
pub struct ActionMapping {
    #[serde(skip_serializing_if = "Option::is_none")]
    primary: Option<String>,
}

(
    macros: {
        "hero": (
            superseded: Some("defaults"),
            subject: (
                primary: Some("hero"),
                secondary: Some("paladin"),
            ),
            action: Some((
                primary: Some("attack"),
            )),
            how: [
                "sword",
                "magic",
            ],
        ),
        "defaults": (
            subject: (),
            action: Some(()),
            how: [],
        ),
    },
    actions: {},
)

You will notice a few things which are now weird, there is still the Some(key) which are annoying (we will fix those) but also we have how:[] which is empty so we could remove it and we have subject:(), action:Some(()) and actions:{}, let's clean that up.

The first one is to remove the Some, this one is tricky since you need to get the value inside, but serde allow to create our own serializer function and since all the None are remove our function can simply be key.unwrap.

fn option_remove_serialize<S,T:Serialize>(x: &Option<T>, s: S) -> Result<S::Ok, S::Error> where S: Serializer {
    x.as_ref().unwrap().serialize(s)
}

We just need to apply it after the skip_serializing (very important since we unwrap without checking).
We replace #[serde(skip_serializing_if = "Option::is_none")] with #[serde(skip_serializing_if = "Option::is_none",serialize_with="option_remove_serialize")]
And we get

(
    macros: {
        "defaults": (
            subject: (),
            action: (),
            how: [],
        ),
        "hero": (
            superseded: "defaults",
            subject: (
                primary: "hero",
                secondary: "paladin",
            ),
            action: (
                primary: "attack",
            ),
            how: [
                "sword",
                "magic",
            ],
        ),
    },
    actions: {},
)

Ok this is really nice but the some fields are useless, so let's remove them, we use skip_serializing_if = "Vec::is_empty" for the how:[] and for the actions: {} we can remove if with #[serde(skip_serializing_if = "HashMap::is_empty")].
For the subject:() in default we can define

fn is_empty_subject(x: &SubjectMapping) -> bool {
    x.primary.is_none() && x.secondary.is_none()
}

and call on SubjectMapping with serializing_if

#[serde(default,skip_serializing_if = "is_empty_subject")]
    subject: SubjectMapping, // we want to force a subject even if the subject has no element

Our final deserialize then re-serialize file looks like that:

(
    macros: {
        "hero": (
            superseded: "defaults",
            subject: (
                primary: "hero",
                secondary: "paladin",
            ),
            action: (
                primary: "attack",
            ),
            how: [
                "sword",
                "magic",
            ],
        ),
        "defaults": (
            action: (),
        ),
    },
)

Which is almost the same as the original one:

#![enable(implicit_some,unwrap_newtypes)]
(
    macros:{
       "defaults" : (
           action:()
       ),
       "hero" : (
           superseded: "defaults",
           subject: (
             primary: "hero",
             secondary: "paladin",
           ),
           action:(
             primary: "attack"
           ),
           how: ["sword","magic"],
       ),
    },
    actions:{}
)

The only difference are now in the fact that the array is not packed the same way, the order is not respected (that's because we use a Hashmap and not a LinkedHashmap use use linked_hash_map::LinkedHashMap; from linked-hash-map = {version="0.5.3", features=["serde_impl"]} ) and the top line is not present (this can be added before outputting to the file.

We can now use transcoding efficiently to manually edit the file as well as automatically and all of that with the same "formatting" and direct usage of rust structure, welcome to the future !

Here the full end source code:
https://gist.github.com/hube12/c87ccd91c519560b4b1cdcefa2e534af

Please read the following lines for my comment on what to add next to ron-rs:

As I said at the beginning, I would love to see some pretty config options such as with_array_breakdown(false) which would allow same line array, or with_extra_coma(true) to add an extra comma at the end of an enum/struct/tuple. I think that since it was mentionned in earlier issue: #189 , this is not yet possible to recover the name of the structure but It would be a nice feature. Also I know this will be a big no-no but I would love to see super/extends fields like in YAML where << : x can reference &x and so on, same goes for variable like x: 2field.y but since this would cause issues with a constant parser, I don't mind a runtime processing.

[kvark]

Thank you for describing your process!
First of all, I think you can simplify the attributes by replacing #[serde(skip_serializing_if = ...)] with just #[serde(skip)].

with_array_breakdown sounds fine to me.
with_extra_coma - I don't think we want this. Instead, we should always have a comma whenever the fields are multi-line, and have no extra comma if fields are on the same line. This appears to be objectively best for everybody.

Also I know this will be a big no-no but I would love to see super/extends fields like in YAML where

Yeah, really no appetite to complicate it in the same way. It's good to keep this purely data, not code :)

[hube12]

First of all, I think you can simplify the attributes by replacing #[serde(skip_serializing_if = ...)] with just #[serde(skip)].

This is not true https://serde.rs/field-attrs.html#skip since this will stop serializing of the value but I actually want to serialize it if it is Some(x), so using the serializing_if is the right one in my use case ;).
Anyway, thanks for the kind words, can't wait for more option for the prettyconfig !

[github-actions]

Issue has had no activity in the last 180 days and is going to be closed in 7 days if no further activity occurs

[kvark]

comment