Generate code using JSON schema

[astrojuanlu]

https://github.com/AnalyticalGraphicsInc/czml-writer/tree/master/Schema

[mhaberler]

nice idea, unfortunately that Schema collects dust just as the CZML Writer spec - last change 11 months ago

do you see a way of extracting that from the source code some way, similar to the documentation?

[astrojuanlu]

Well - maybe CZML did not change in the past 11 months? :) (One can only hope)

I am not sure how to do that. Also, it's not clear how to use the schema either, apart from a last step validation, or integration test, or something.

[astrojuanlu]

Idea: instead of validating against the JSON Schema, let's generate czml3 source code from the schema!

In fact, instead of generating Python code directly, we could programmatically generate attrs classes:

https://www.attrs.org/en/stable/api.html#attr.make_class

Just checked that, by using attr.ib instead of the provisional attr.field, we can even attach types.

For verification purposes, the fields of the attrs class can be retrieved with https://www.attrs.org/en/stable/api.html#attr.fields.

And finally, writing the original code was extremely tedious... and #76 was being also tedious.

I will try to make a proof of concept.

[astrojuanlu]

In fact, I think the wiki is generated from the JSON schema: https://github.com/AnalyticalGraphicsInc/czml-writer/blob/master/DotNet/GenerateFromSchema/MarkdownGenerator.cs

[astrojuanlu]

from typing import Optional
from functools import lru_cache

import attr
import requests

from czml3.base import BaseCZMLObject

BASE_CZML_URL = "https://analyticalgraphicsinc.github.io/czml-writer/Schema/"


@lru_cache
def get_schema(schema_name):
    return requests.get(f"{BASE_CZML_URL}/{schema_name}").json()


def get_czml_type(type_str):
    return {"boolean": bool, "string": str}[type_str]


def get_property_metadata(property_metadata):
    if ref := property_metadata.get("$ref"):
        return get_schema(ref)
    else:
        return property_metadata


def make_field(property_metadata):
    property_metadata = get_property_metadata(property_metadata)
    field_type = get_czml_type(property_metadata["type"])

    if property_metadata.get("czmlRequiredForDisplay", False):
        return attr.ib(type=field_type)
    else:
        return attr.ib(type=Optional[field_type], default=None)


def make_spec(schema_name):
    schema = get_schema(schema_name)
    return {
        name: make_field(metadata) for name, metadata in schema["properties"].items()
    }


Deletable = attr.make_class(
    "Deletable",
    make_spec("DeletableProperty.json"),
    (BaseCZMLObject,),
    repr=False,
    frozen=True,
    kw_only=True,
)
Uri = attr.make_class(
    "Uri",
    make_spec("Uri.json"),
    (Deletable, BaseCZMLObject),
    repr=False,
    frozen=True,
    kw_only=True,
)


if __name__ == "__main__":
    print(Uri)
    print(Uri())
    print(Uri(uri="image.png"))

Result:

$ python build_uri.py 
<class '__main__.Uri'>
{}
{
    "uri": "image.png"
}

[astrojuanlu]

I think this approach is very promising, it just needs some work :)

[rushabh-v]

Hi @astrojuanlu, I am Rushabh Vasani, a 3rd-year student from India and I am interested in working on this project under GSoC.

If I understood it correctly, the project is about automating the process of creating the property classes using JSON Schema and adding tests for the tool, type checks, and convenience functions.

You are welcome to add more things to the list and correct me if anything is wrong.

---

I have experience working on similar things in the past and I love automating and generalizing stuff. Please let me know if there is anything I can do right now to get a better idea of the project and showcase my abilities.

Thanks!

[astrojuanlu]

Hi @rushabh-v ! I outlined my idea for this in #81 (comment) . If we could do it in an incremental way, instead of having to wait for 1 massive pull request that changes everything, that would be great. I'm all eyes :)

[rushabh-v]

Hi @astrojuanlu, Thanks for your response.

If we could do it in an incremental way, instead of having to wait for 1 massive pull request that changes everything, that would be great.

Sure, here is how I am planning to break it down (one PR per point):

1. The class generator module with the type checks using mypy
2. Tests for the generator and generated classes
3. Code quality - GitHub Actions (integrate black, isort, etc.)
4. Extras (could be discovered while implementing the former functionalities)

Your suggestions would be welcomed. Also, I see that you started working on this in #86. Should I refer to that?

Thanks.

[astrojuanlu]

I don't like this breakdown :) Code quality should be ingrained in every pull request, we can't merge code that doesn't pass our checks. Also, every pull request must have tests. I suggest that we discuss this in our chat.

And yes, #86 is an unfinished proof of concept. In fact, I will extract some parts of it.

[rushabh-v]

Sure, I thought the class generator module would be highly coupled and I couldn't see any way of a vertical breakdown (the way you mentioned). I might be missing something.

Yes, let's discuss this. By chat do you mean the polaris's matrix room or is there a separate channel for czml3?

And yes, #86 is an unfinished proof of concept. In fact, I will extract some parts of it.

Okay.

Thanks.

[astrojuanlu]

By chat do you mean the polaris's matrix room or is there a separate channel for czml3?

https://github.com/poliastro/czml3#support

[Stoops-ML]

Some issues I have with this direction:

1. Downloading the schema on every installation of CZML3 will take a long time.
2. There are discrepancies between the schema and Cesium (see here for an example). This is due to the schema not being up to date and bugs in Cesium. Fixing the discrepancies would require writing the class explicitly, which is the current implementation.
3. Versioning of CZML3 becomes meaningless as it depends on the schema at the time of installation. The same distribution of CZML3 will produce different results (i.e. fail tests) before and after any schema update. This probably won't be acceptable in production code.
4. I don't see a way to add typing to properties using this method.

[astrojuanlu]

Yep, good points @Stoops-ML... feel free to close this issue as not planned.