Writing np.int64 or np.int32 scalars raises an error

[fmaussion]

Hi, this is on python3 with fiona 1.7.0.post2. Attempting to write properties as numpy scalars can fail for certain types. The following example reproduces the problem:

import fiona
import numpy as np

feat = {'geometry': {'coordinates': (((21.0, 22.0), (22.0, 23.0), 
                                      (24.0, 20.0), (21.0, 22.0)),),
        'type': 'Polygon'},
        'id': '0',
        'properties': {'PROP': np.int64(12)},  # this line is the problem
        'type': 'Feature'}
schema = {'geometry': 'Polygon', 'properties':{'PROP':'str'}}

with fiona.open('test.shp', 'w', driver="ESRI Shapefile", schema=schema) as c:
    c.write(feat)

will raise:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-84-e58056597894> in <module>()
     11 
     12 with fiona.open('test2.shp', 'w', driver="ESRI Shapefile", schema=schema) as c:
---> 13     c.write(feat)

/home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/fiona/collection.py in write(self, record)
    331     def write(self, record):
    332         """Stages a record for writing to disk."""
--> 333         self.writerecords([record])
    334 
    335     def validate_record(self, record):

/home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/fiona/collection.py in writerecords(self, records)
    325         if self.mode not in ('a', 'w'):
    326             raise IOError("collection not open for writing")
--> 327         self.session.writerecs(records, self)
    328         self._len = self.session.get_length()
    329         self._bounds = self.session.get_extent()

fiona/ogrext.pyx in fiona.ogrext.WritingSession.writerecs (fiona/ogrext1.c:17022)()

fiona/ogrext.pyx in fiona.ogrext.OGRFeatureBuilder.build (fiona/ogrext1.c:6615)()

ValueError: Invalid field type <class 'numpy.int64'>

Note that np.int or np.float would work. (see geopandas/geopandas#348 and pandas-dev/pandas#13468 for more background).

[jorisvandenbossche]

Note that np.int or np.float would work.

@fmaussion That's because these are aliases for the builtin python types.

Also note that specifying 'str' in the schema does not matter, you get the same error when specifying 'int'. It is the numpy scalar that cannot be handled.

GeoDataFrame.to_file() (which is an interface to fiona for writing files) uses GeoDataFrame.iterfeatures() (which itself uses DataFrame.iterrows() under the hood) to create the features that are passed to fiona, which normally converts the contents of the rows into python types, but in some corner cases numpy scalars can be returned (e.g. geopandas/geopandas#348). Hence the example of @fmaussion above.

[sgillies]

@fmaussion thanks for the report. Currently, Fiona obliges developers to cast values to Python types that map to the OGR field types. If your feature property was str(np.int64(12)) the write() call would succeed. I'm open to the idea of casting values within write() if that helps.

[fmaussion]

@sgillies thanks! Intuitively, I would say that as a scientific tool such as Fiona should be able to handle numpy dtypes (somehow, np.float64 doesn't seem to be a problem?).

At the same time, my use case is a bit of a corner case (slight misuse of GeoSeries and GeoDataFrames with Geopandas), and if nobody complained until now this is probably not very relevant for most of your users.

So that's up to you I would say!

[jasonwheritage]

I would certainly like Geopandas to support int64 writes, I am working with spatial data the has long int64 integer keys, I would like to be able to retain the datatype throughout instead of converting everything to string. In my case though I am limited to how many digits are supported by the Shapefile standard:
https://en.wikipedia.org/wiki/Shapefile#:~:text=Maximum%20number%20of%20fields%20is,text%20(maximum%20254%20character%20storage)

I'm pretty sure that Fiona will return an exception if the data being written does not conform to the standard. a 12-digit integer will need to be converted to string before Fiona will write it.

[fmaussion]

@jasonwheritage no need to convert to strings : python's int works as well: https://github.com/OGGM/oggm/blob/de963a56359339274cc61763a2cab237ac3c33c9/oggm/utils/_workflow.py#L1823-L1826

[sheecegardezi]

Just use float:

   'properties': {'PROP': float},  

before writing convert the np.float32 value to float.