Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nullable items in a list #96

Open
mjhaller opened this issue Nov 28, 2023 · 0 comments
Open

nullable items in a list #96

mjhaller opened this issue Nov 28, 2023 · 0 comments

Comments

@mjhaller
Copy link

Consider one would like a list that can alternatively have nulls or strings in it:

{"bar": [null, "hi", null, null, "howareu?"]}

The working avro schema for this is as follows:

foo_schema = '''
{
  "type": "record",
  "namespace": "Foo",
  "name": "Foo",
  "fields": [
    {
      "type": {
        "type": "array",
        "items": ["null", "string"]
      },
      "name": "bar"
    }
  ]
}
'''

This schema is valid and will validate using fastavro

validate({ "bar" : [None, "hi", None, None, "howareu?"]}, schema.parse_schema(json.loads(foo_schema)))
True

How would I express this using AvroBase? I've tried this

class Foo(AvroBase):
    bar: List[Optional[str]]

Here's a full example of what i tried:

from pydantic_avro.base import AvroBase
from typing import Optional, List
import json
from fastavro import schema
class Foo(AvroBase):
    bar: List[Optional[str]]
print(json.dumps(Foo.avro_schema(), indent=2))
schema.parse_schema(Foo.avro_schema())

{
  "type": "record",
  "namespace": "Foo",
  "name": "Foo",
  "fields": [
    {
      "type": {
        "type": "array",
        "items": {
          "type": [
            "string",
            "null"
          ]
        }
      },
      "name": "bar"
    }
  ]
}
Traceback (most recent call last):
  File "/Users/mhaller/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/232.8660.197/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 10, in <module>
  File "fastavro/_schema.pyx", line 173, in fastavro._schema.parse_schema
  File "fastavro/_schema.pyx", line 408, in fastavro._schema._parse_schema
  File "fastavro/_schema.pyx", line 476, in fastavro._schema.parse_field
  File "fastavro/_schema.pyx", line 326, in fastavro._schema._parse_schema
  File "fastavro/_schema.pyx", line 433, in fastavro._schema._parse_schema
TypeError: unhashable type: 'list''

The issue it appears is that

"items": {
          "type": [
            "string",
            "null"
          ]
        }

Shoudl really be

"items": [
            "string",
            "null"
          ]

Also see this SO post which is trying to do the same thing and uses the same approach to express null elements. Avro specification does not get into the details of how this works.

@mjhaller mjhaller changed the title Optional items in a list nullable items in a list Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant