simple select/sum giving vague error in monetdbe

[dancromartie]

Describe the bug
I have uploaded this covid dataset to a table in monetdbe. I gave the new_deaths column an INT type, and new_cases_per_million a DOUBLE type.

These 3 queries work fine:

SELECT avg(new_deaths) FROM covid;
SELECT sum(cast(new_deaths AS BIGINT)) FROM covid;
SELECT sum(new_cases_per_million) FROM covid;

But this one fails:

SELECT sum(new_deaths) FROM covid;

The stacktrace is:

ERROR: Traceback (most recent call last):
  File "/home/dcromartie/dev/try-monet/app.py", line 66, in page_db
    result = fetch_initial_rows_df(cursor, query_part)
  File "/home/dcromartie/dev/try-monet/app.py", line 30, in fetch_initial_rows_df
    results = cursor.execute(query)
  File "/home/dcromartie/virtualenvs/trymonet/lib/python3.9/site-packages/monetdbe/cursors.py", line 155, in execute
    return self._execute_monetdbe(operation, parameters)
  File "/home/dcromartie/virtualenvs/trymonet/lib/python3.9/site-packages/monetdbe/cursors.py", line 143, in _execute_monetdbe
    self._set_description()
  File "/home/dcromartie/virtualenvs/trymonet/lib/python3.9/site-packages/monetdbe/cursors.py", line 51, in _set_description
    self.description = self.connection.get_description()
  File "/home/dcromartie/virtualenvs/trymonet/lib/python3.9/site-packages/monetdbe/connection.py", line 143, in get_description
    descriptions = list(zip(name, type_code, display_size, internal_size, precision, scale, null_ok))
  File "/home/dcromartie/virtualenvs/trymonet/lib/python3.9/site-packages/monetdbe/connection.py", line 137, in <genexpr>
    type_code = (monet_c_type_map[rcol.type].sql_type for rcol in columns)
KeyError: 5

Then I found this triggers the same KeyError: 5:

SELECT columnsize + heapsize FROM sys.storage;

To Reproduce

import monetdbe

conn = monetdbe.connect("bug.db")
conn.cursor().execute("SELECT columnsize + heapsize FROM sys.storage")

Expected behavior
No error, or a clearer error message

Software versions

• monetdbe==0.11
• Ubuntu 21.10
• Installed via pip

[PedroTadim]

This happens because MonetDBe-Python has issues handling huge integers. Please look at the discussion here: MonetDBSolutions/MonetDBe-Python#141 A temporary solution is to cast the result into a bigint.

[dancromartie]

Thanks! Just to confirm it's the same issue, is the "huge integers" thing still relevant for relatively small sums? The sum in my case, if I cast to bigint, is 22,986,208 (the covid query), which would fit in smaller int types. Is the issue that Monet is trying to assign the result to a huge integer that Python isn't ready for?

[PedroTadim]

MonetDB stores the result of SUM in a hugeint to avoid overflow if huge integers are available. In MonetDBe-Python, the native library was compiled with huge integer support, while the Python side still doesn't read them. At the moment, MonetDBe-Python should be compiled without them.