Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

masked_area_interpolate errors #136

Open
adamConnerSax opened this issue Feb 25, 2021 · 13 comments
Open

masked_area_interpolate errors #136

adamConnerSax opened this issue Feb 25, 2021 · 13 comments

Comments

@adamConnerSax
Copy link

Hi! Tobler is awesome! Thank you for all the work!

I am aggregating census count data from the census-tract level to the congressional district level. Using "area_interpolate" this works fine. Thanks!!

Now I'm trying to improve this using the NLCD data and "masked_area_interpolate". When I do that I get lots of errors of the form:
"CBR: result (after common-bits addition) is INVALID: Self-intersection at or near point..."
or
"CBR: result (after common-bits addition) is INVALID: Too few points in geometry component at or near point..."

each followed by many many numbers. I've tried projecting both source and target geometries to EPSG:3857 and EPSG:4326 and it happens either way.

It continues to run, despite the errors. So I have 2 questions:

  1. Is there something I should be doing with the data before calling "masked_area_interpolate" that might fix this?
  2. Will the interpolation still be useful? That is, is masked_area_interpolate going to be an improvement on area_interpolate regardless of all these errors, or should I just stick with the results of area_interpolate until/unless I figure out the errors?

Thanks!

@sjsrey
Copy link
Member

sjsrey commented Feb 25, 2021

Thanks for reporting this.

Can you provide a bit more info regarding your os and version of geopandas?

@adamConnerSax
Copy link
Author

Running on mac OSX (10.15.7m Catalina)
Python 3.8.5
geopandas 0.8.2
Tobler 0.7

@knaaptime
Copy link
Member

tough to diagnose, but my first guess would be you have some topology errors in your source/target data (both of those errors come from shapely).

try running

source = source.buffer(0)
target = target.buffer(0)

before using area_interpolate_masked (though if this were the issue, im not sure why area_interpolate would work, but masked wouldnt...)

Another thing to try would be to see if you can follow this notebook using your data (though you can omit the OSM step if you want) which performs essentially the same task as masked_area_interpolate but with a different approach

@martinfleis
Copy link
Member

You'll need

source.geometry = source.buffer(0)
target.geometry = target.buffer(0)

Buffer returns GeoSeries.

@knaaptime
Copy link
Member

sorry, yes thank you @martinfleis

@adamConnerSax
Copy link
Author

Tried

source.geometry = source.buffer(0)
target.geometry = target.buffer(0)

and I still get the same/similar errors. Will try the notebook thing when I get some time...

@jtamerius
Copy link

I'm getting this same error doing a similar calculation. Specifically, I am trying to estimate population from census tracts using developed land rasterized by the geocube api. I also tried the NLCD data and received same error.

@martinfleis
Copy link
Member

@jtamerius Can you share the full traceback and a specification of your environment?

@knaaptime
Copy link
Member

knaaptime commented Jul 22, 2021

can you share the full traceback so I can try and see where the root error is coming from? or possibly the source/target data you're using? That's a low-level error coming from geos and usually has to do with invalid geometries afaik, though i have no idea why it would occur only in masked_area_interpolate.

As a workaround, you may be able to use the method described in this notebook instead

@knaaptime
Copy link
Member

knaaptime commented Jul 22, 2021

@martinfleis lol beat me to the punch while i was pasting in the url and i didnt even notice

@jtamerius
Copy link

As a workaround, you may be able to use the method described in this notebook instead

@knaaptime : this looks like a great method for estimating population. Unfortunately, this link is not working for me: https://s3-us-west-2.amazonaws.com/mrlc/NLCD_2016_Impervious_L48_20190405.zip

p.s. I'll get you all the traceback soon.

@knaaptime
Copy link
Member

hm, they may have updated the links but you should be able to pick an appropriate year from here and use that

https://www.mrlc.gov/data?f%5B0%5D=category%3AUrban%20Imperviousness&f%5B1%5D=category%3Aland%20cover&f%5B2%5D=region%3Aconus

@jtamerius
Copy link

jtamerius commented Jul 22, 2021

So, there is not much of a traceback:

CBR: result (after common-bits addition) is INVALID: Too few points in geometry component at or near point 144907.12541067583 -286284.33574648993 (144907.12541067582788 -286284.3357464899309) <A> GEOMETRYCOLLECTION (LINESTRING (157566.5237387096858583 -293342.2898403313010931, 157817.6488116196996998 -293103.4683011914021336), LINESTRING (155667.2435246870154515 -291631.9672550756367855, 155665.4470387034816667 -291624.1666764495312236), LINESTRING (155742.3051454601809382 -291823.6784229896729812, 155748.8153089357656427 -291825.4875020556501113),...

I truncated list of coordinates because it goes on forever. I added teh target and source geometries to zip file attached. Here are packages I have installed:

Name Version Build Channel
affine 2.3.0 py_0
amqp 5.0.6 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
atomicwrites 1.4.0 py_0
attrs 21.2.0 pyhd3eb1b0_0
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 3.2.0 py39hb82d6ee_1 conda-forge
beautifulsoup4 4.9.3 pyha847dfd_0
billiard 3.6.4.0 py39hb82d6ee_0 conda-forge
blas 1.0 mkl
blosc 1.21.0 h0e60522_0 conda-forge
bokeh 2.3.3 py39hcbf5309_0 conda-forge
branca 0.4.2 pyhd8ed1ab_0 conda-forge
brotli-python 1.0.9 py39h415ef7b_4 conda-forge
brotlipy 0.7.0 py39h2bbff1b_1003
bs4 4.9.3 hd3eb1b0_0
bzip2 1.0.8 he774522_0
ca-certificates 2021.5.30 h5b45459_0 conda-forge
cachetools 4.2.2 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 hb19e0ff_1008 conda-forge
celery 5.1.2 pyhd8ed1ab_0 conda-forge
censusdata 1.13 pypi_0 pypi
certifi 2021.5.30 py39hcbf5309_0 conda-forge
cffi 1.14.6 py39h2bbff1b_0
cfitsio 3.470 he774522_6
cftime 1.5.0 py39h5d4886f_0 conda-forge
chardet 4.0.0 py39haa95532_1003
click 7.1.2 pyhd3eb1b0_0
click-didyoumean 0.0.3 pyh8c360ce_0 conda-forge
click-plugins 1.1.1 py_0
click-repl 0.2.0 pyhd8ed1ab_0 conda-forge
cligj 0.7.2 py39haa95532_0
cloudpickle 1.6.0 py_0 conda-forge
colorama 0.4.4 pyhd3eb1b0_0
cryptography 3.4.7 py39h71e12ea_0
curl 7.77.0 h789b8ee_0 conda-forge
cycler 0.10.0 py39haa95532_0
cytoolz 0.11.0 py39hb82d6ee_3 conda-forge
dash 1.20.0 pyhd8ed1ab_0 conda-forge
dash-core-components 1.16.0 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.3 pyhd8ed1ab_0 conda-forge
dash-renderer 1.9.1 pyhd3eb1b0_0
dash-table 4.11.3 pyhd8ed1ab_0 conda-forge
dask 2021.7.0 pyhd8ed1ab_0 conda-forge
dask-core 2021.7.0 pyhd8ed1ab_0 conda-forge
datacube 1.8.3 py_1 conda-forge
decorator 5.0.9 pyhd3eb1b0_0
distributed 2021.7.0 py39hcbf5309_0 conda-forge
expat 2.4.1 h6c2663c_2
fiona 1.8.19 py39h9f1b043_0 conda-forge
flask 1.1.2 pyhd3eb1b0_0
flask-compress 1.10.1 pyhd3eb1b0_0
folium 0.12.0 pyhd8ed1ab_1 conda-forge
fontconfig 2.13.1 h1989441_1005 conda-forge
freetype 2.10.4 hd328e21_0
freexl 1.0.6 h2bbff1b_0
fsspec 2021.7.0 pyhd8ed1ab_0 conda-forge
future 0.18.2 py39haa95532_1
gdal 3.2.2 py39h6795fcd_7 conda-forge
geocube 0.0.17 pyhd8ed1ab_0 conda-forge
geopandas 0.9.0 pyhd8ed1ab_0 conda-forge
geos 3.9.1 h6c2663c_0
geotiff 1.6.0 ha8a8a2d_6 conda-forge
gettext 0.19.8.1 h1a89ca6_1005 conda-forge
greenlet 1.1.0 py39h415ef7b_0 conda-forge
h3-py 3.7.3 py39h415ef7b_1 conda-forge
hdf4 4.2.15 h0e5069d_3 conda-forge
hdf5 1.10.6 nompi_h5268f04_1114 conda-forge
heapdict 1.0.1 py_0 conda-forge
icc_rt 2019.0.0 h0cc432a_1
icu 68.1 h6c2663c_0
idna 2.10 pyhd3eb1b0_0
importlib-metadata 4.6.1 py39hcbf5309_0 conda-forge
iniconfig 1.1.1 pyhd3eb1b0_0
intel-openmp 2021.3.0 haa95532_3372
itsdangerous 2.0.1 pyhd3eb1b0_0
jbig 2.1 h8d14728_2003 conda-forge
jinja2 3.0.1 pyhd3eb1b0_0
joblib 1.0.1 pyhd3eb1b0_0
jpeg 9d h8ffe710_0 conda-forge
jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge
kealib 1.4.14 h96bfa42_2 conda-forge
kiwisolver 1.3.1 py39hd77b12b_0
kombu 5.1.0 py39hcbf5309_0 conda-forge
krb5 1.19.1 h5b6d351_0
lark-parser 0.11.3 pyhd8ed1ab_0 conda-forge
lerc 2.2.1 hd77b12b_0
libblas 3.9.0 9_mkl conda-forge
libboost 1.67.0 hd9e427e_4
libcblas 3.9.0 9_mkl conda-forge
libclang 11.1.0 default_h5c34c98_1 conda-forge
libcurl 7.77.0 h789b8ee_0 conda-forge
libdeflate 1.7 h2bbff1b_5
libffi 3.3 h0e60522_2 conda-forge
libgdal 3.2.2 ha6cada3_7 conda-forge
libglib 2.68.1 h1e62bf3_0 conda-forge
libiconv 1.16 he774522_0 conda-forge
libkml 1.3.0 he5f2a48_4
liblapack 3.9.0 9_mkl conda-forge
libnetcdf 4.8.0 nompi_hee75d68_101 conda-forge
libpng 1.6.37 h2a8f88b_0
libpq 13.3 hfcc5ef8_0 conda-forge
libpysal 4.5.1 pyhd3eb1b0_0
librttopo 1.1.0 hb340de5_6 conda-forge
libspatialindex 1.9.3 h6c2663c_0
libspatialite 5.0.1 h762a7f4_5 conda-forge
libssh2 1.9.0 h7a1dbc1_1
libtiff 4.3.0 h0c97f57_1 conda-forge
libwebp-base 1.2.0 h2bbff1b_0
libxml2 2.9.12 h0ad7f3c_0
libxslt 1.1.34 he774522_0
libzip 1.7.3 hfed4ece_0 conda-forge
locket 0.2.0 py_2 conda-forge
lxml 4.6.3 py39h9b66d53_0
lz4-c 1.9.3 h2bbff1b_0
markupsafe 2.0.1 py39h2bbff1b_0
matplotlib 3.4.2 py39hcbf5309_0 conda-forge
matplotlib-base 3.4.2 py39h581301d_0 conda-forge
mkl 2021.2.0 hb70f87d_389 conda-forge
mkl-service 2.4.0 py39h2bbff1b_0
mock 4.0.3 pyhd3eb1b0_0
more-itertools 8.8.0 pyhd3eb1b0_0
msgpack-python 1.0.2 py39h2e07f2f_1 conda-forge
munch 2.5.0 py_0
netcdf4 1.5.7 nompi_py39hc20e565_100 conda-forge
networkx 2.5 py_0 conda-forge
numexpr 2.7.3 py39hb80d3ca_1
numpy 1.20.2 py39h6635163_0 conda-forge
olefile 0.46 py_0
openjpeg 2.4.0 h48faf41_0 conda-forge
openssl 1.1.1k h8ffe710_0 conda-forge
osmnx 1.1.1 pyhd8ed1ab_0 conda-forge
packaging 21.0 pyhd3eb1b0_0
pandas 1.2.4 py39h2e25243_0 conda-forge
paramiko 2.7.2 pyh9f0ad1d_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
patsy 0.5.1 py39haa95532_0
pcre 8.45 hd77b12b_0
pillow 8.3.1 py39h4fa10fc_0
pip 21.1.3 py39haa95532_0
pixman 0.40.0 he774522_0
plotly 4.14.3 pyh44b312d_0 conda-forge
pluggy 0.13.1 py39haa95532_0
poppler 21.03.0 h9ff6ed8_0 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 13.3 h1c22c4f_0 conda-forge
progressbar2 3.37.1 py39haa95532_0
proj 8.0.1 h1cfcee9_0 conda-forge
prompt-toolkit 3.0.19 pyha770c72_0 conda-forge
prompt_toolkit 3.0.19 hd8ed1ab_0 conda-forge
psutil 5.8.0 py39hb82d6ee_1 conda-forge
psycopg2 2.9.1 py39h0878f49_0 conda-forge
py 1.10.0 pyhd3eb1b0_0
pycparser 2.20 py_2
pygam 0.8.0 py_0 conda-forge
pygeos 0.10 py39haadaec5_0 conda-forge
pynacl 1.4.0 py39hb3671d1_2 conda-forge
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyhd3eb1b0_0
pyproj 3.1.0 py39ha996c60_3 conda-forge
pyqt 5.12.3 py39hcbf5309_7 conda-forge
pyqt-impl 5.12.3 py39h415ef7b_7 conda-forge
pyqt5-sip 4.19.18 py39h415ef7b_7 conda-forge
pyqtchart 5.12 py39h415ef7b_7 conda-forge
pyqtwebengine 5.12.1 py39h415ef7b_7 conda-forge
pyrsistent 0.17.3 py39hb82d6ee_2 conda-forge
pysocks 1.7.1 py39haa95532_0
pytables 3.6.1 py39h42e6cd8_3 conda-forge
pytest 6.2.4 py39haa95532_2
pytest-runner 5.3.1 pyhd3eb1b0_0
python 3.9.5 h6244533_3
python-dateutil 2.8.2 pyhd3eb1b0_0
python-utils 2.5.6 py39haa95532_0
python_abi 3.9 1_cp39 conda-forge
pytz 2021.1 pyhd3eb1b0_0
pyyaml 5.4.1 py39h2bbff1b_1
qt 5.12.9 h5909a2a_4 conda-forge
rasterio 1.2.6 py39hb4f844e_1 conda-forge
rasterstats 0.14.0 py_0
redis-py 3.5.3 pyh9f0ad1d_0 conda-forge
requests 2.25.1 pyhd3eb1b0_0
retrying 1.3.3 py_2
rioxarray 0.5.0 pyhd8ed1ab_0 conda-forge
rtree 0.9.7 py39h2eaa2aa_1
scikit-learn 0.24.2 py39he931e04_0 conda-forge
scipy 1.6.2 py39h66253e8_1
setuptools 52.0.0 py39haa95532_0
shapely 1.7.1 py39h90c6b7e_4 conda-forge
simplejson 3.17.3 py39h2bbff1b_2
six 1.16.0 pyhd3eb1b0_0
snuggs 1.4.7 py_0
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.2.1 pyhd3eb1b0_0
sqlalchemy 1.4.22 py39hb82d6ee_0 conda-forge
sqlite 3.36.0 h2bbff1b_0
sshtunnel 0.3.1 pyhd8ed1ab_0 conda-forge
statsmodels 0.12.2 py39h2bbff1b_0
tbb 2021.2.0 h59b6b97_0
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 2.2.0 pyhb85f177_0
tiledb 2.3.2 h78dabda_0 conda-forge
tk 8.6.10 he774522_0
tobler 0.8.2 pyhd8ed1ab_0 conda-forge
toml 0.10.2 pyhd3eb1b0_0
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py39h2bbff1b_0
tqdm 4.61.2 pyhd3eb1b0_1
typing_extensions 3.10.0.0 pyha770c72_0 conda-forge
tzdata 2021a h52ac0ba_0
urllib3 1.26.4 pypi_0 pypi
vc 14.2 h21ff451_1
vine 5.0.0 pyhd8ed1ab_1 conda-forge
vs2015_runtime 14.27.29016 h5e58377_2
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
werkzeug 1.0.1 pyhd3eb1b0_0
wheel 0.36.2 pyhd3eb1b0_0
win_inet_pton 1.1.0 py39haa95532_0
wincertstore 0.2 py39h2bbff1b_0
xarray 0.18.2 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.3 ha925a31_0
xz 5.2.5 h62dcd97_0
yaml 0.2.5 he774522_0
zict 2.0.0 py_0 conda-forge
zipp 3.5.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h62dcd97_4
zstd 1.5.0 h6255e5f_0 conda-forge

source_target_vars.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants