feature request: pls host parquet and spatial extensions for linux_arm64_gcc4

[nikhil-scholarlify]

I want to run a python program using quackosm, on an Azure server that is having arm64 CPU architecture (as opposed to amd64).
Quackosm uses duckdb. The program is getting stuck on this error:

Traceback (most recent call last):
  File "/home/azureuser/pbf_to_geoparquet/country_extract.py", line 156, in <module>
    gpq_path = quackosm.convert_pbf_to_gpq(pbf,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/azureuser/.pyenv/versions/3.11.8/lib/python3.11/site-packages/quackosm/functions.py", line 229, in convert_pbf_to_gpq
    ).convert_pbf_to_gpq(
      ^^^^^^^^^^^^^^^^^^^
  File "/home/azureuser/.pyenv/versions/3.11.8/lib/python3.11/site-packages/quackosm/pbf_file_reader.py", line 224, in convert_pbf_to_gpq
    self.connection = _set_up_duckdb_connection(tmp_dir_path=self.tmp_dir_path)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/azureuser/.pyenv/versions/3.11.8/lib/python3.11/site-packages/quackosm/pbf_file_reader.py", line 2418, in _set_up_duckdb_connection
    connection.install_extension(extension_name)
duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "parquet" at URL "http://extensions.duckdb.org/v0.9.2/linux_arm64_gcc4/parquet.duckdb_extension.gz"
Extension "parquet" is an existing extension.

If I run the same program in an amd64 (or referred to as x86) CPU server, then in the working folder specified, these files under "extensions" folder are created:

working/extensions/v0.9.2/linux_amd64_gcc4$ l
total 110M
-rw-r--r-- 1 ubuntu ubuntu 39M Apr  9 14:06 parquet.duckdb_extension
-rw-r--r-- 1 ubuntu ubuntu 71M Apr  9 14:06 spatial.duckdb_extension

So for amd64 there the parquet.duckdb_extension is properly fetched; this url is valid : http://extensions.duckdb.org/v0.9.2/linux_amd64_gcc4/parquet.duckdb_extension.gz

But the url "http://extensions.duckdb.org/v0.9.2/linux_arm64_gcc4/parquet.duckdb_extension.gz" is invalid. I'm guessing these files aren't built / hosted. Either that, or the path needs to be something else.

So, please build and host the arm64 extension files on extensions.duckdb.org, or please show how I can build them myself and make the quackosm > duckdb pick those up instead of trying an invalid url.

[Mause]

The parquet extension should already be built into all DuckDB distributions, so you won't need to install it. The spatial extension should probably be requested in DuckDB/spatial

[nikhil-scholarlify]

@Mause even with pip install duckdb done in my arm64 cpu architecture server, the program with quackosm function call still fails:

duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "parquet" at URL "http://extensions.duckdb.org/v0.9.2/linux_arm64_gcc4/parquet.duckdb_extension.gz"

The fact is that nothing exists at the url mentioned : http://extensions.duckdb.org/v0.9.2/linux_arm64_gcc4/parquet.duckdb_extension.gz

Anyways, removing quackosm from the picture, even these simple duckdb lines put into python prompt trigger the same error, this time for the spatial extension

$ python
Python 3.11.8 (main, Apr 10 2024, 04:25:44) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import duckdb
>>> cursor = duckdb.connect()
>>> print(cursor.execute('INSTALL spatial; LOAD spatial;').fetchall())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
duckdb.duckdb.HTTPException: HTTP Error: Failed to download extension "spatial" at URL "http://extensions.duckdb.org/v0.9.2/linux_arm64_gcc4/spatial.duckdb_extension.gz"
Extension "spatial" is an existing extension.

this is following the instructions given at https://duckdb.org/docs/extensions/spatial

So, i'm not even able to activate the spatial extension in duckdb on a arm64 machine. it's trying to download a file from extenstions.duckdb.org which isn't there.

[Mause]

As I said, the parquet extension is built in, so attempts to install it will fail.

And spatial is likely just not available yet for that arch combo, so you'll need to request it in the spatial repository

[RaczeQ]

I wasn't aware that parquet extension is included by default in DuckDB. I'll remove the installation step from QuackOSM
kraina-ai/quackosm#103