Warn/work around Java issues with tr_TR locale

[gorkemgungormetu]

I installed message-ix and ixmp 3.4.0 with anaconda and have Java exception with the code below in Westeros tutorial.

base_inv_cost = {
    'node_loc': country,
    'year_vtg': model_horizon,
    'unit': 'USD/kW',
}

# Adding a new unit to the library
mp.add_unit('USD/kW','installation cost')     

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
File Platform.java:678, in at.ac.iiasa.ixmp.Platform.addUnitToDB()

File DbDAO.java:1327, in at.ac.iiasa.ixmp.database.DbDAO.addUnitToDB()

File DbDAO.java:836, in at.ac.iiasa.ixmp.database.DbDAO.getNextId()

Exception: Java Exception

The above exception was the direct cause of the following exception:

at.ac.iiasa.ixmp.exceptions.IxException   Traceback (most recent call last)
Input In [85], in <module>
      1 base_inv_cost = {
      2     'node_loc': country,
      3     'year_vtg': model_horizon,
      4     'unit': 'USD/kW',
      5 }
      7 # Adding a new unit to the library
----> 8 mp.add_unit('USD/kW','installation cost')

File ~\Anaconda3\envs\message_env\lib\site-packages\ixmp\core\platform.py:259, in Platform.add_unit(self, unit, comment)
    256     log.info(f"unit `{unit}` is already defined in the platform instance")
    257     return
--> 259 self._backend.set_unit(unit, comment)

File ~\Anaconda3\envs\message_env\lib\site-packages\ixmp\backend\jdbc.py:462, in JDBCBackend.set_unit(self, name, comment)
    461 def set_unit(self, name, comment):
--> 462     self.jobj.addUnitToDB(name, comment)

at.ac.iiasa.ixmp.exceptions.IxException: at.ac.iiasa.ixmp.exceptions.IxException: Cannot generate id for unıt

I checked the units in the ixmp instance and see that the unit is not already defined.

mp.units()

['USD/GWa',
 'km',
 'cases',
 '%',
 'USD/kWa',
 'USD/tC',
 'USD/tCO2',
 'USD',
 'tCO2',
 'MW',
 '-',
 'tC',
 'GWa',
 'GW',
 'T$',
 't',
 'USD/km',
 'MWa',
 'USD/kg',
 'y',
 'kg/kWa',
 'kg',
 'G$',
 '???']

I think there is a problem with my connection to the ixmp backend and would appreciate your help.

[khaeru]

Thanks for opening this issue! The bug report template asks you to provide the output of ixmp show-versions, because this information is required in order for us to diagnose. Can you please do that?

[gorkemgungormetu]

Thanks for opening this issue! The bug report template asks you to provide the output of ixmp show-versions, because this information is required in order for us to diagnose. Can you please do that?

ixmp.show_versions()

ixmp:        3.4.0
message_ix:  3.4.0
message_ix_models: None
message_data: None

click:       8.0.3
dask:        2022.01.1
genno:       installed
graphviz:    None
jpype:       1.3.0
… JVM path:  C:\Users\ggungor\Anaconda3\envs\message_env\Library\bin\server\jvm.dll
openpyxl:    3.0.9
pandas:      1.4.0
pint:        0.18
xarray:      0.21.1
yaml:        6.0

iam_units:   installed
jupyter:     installed
matplotlib:  3.5.1
plotnine:    0.8.0
pyam:        1.3.1

GAMS:        'gams' executable not in PATH

python:      3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:22:07) [MSC v.1929 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      ('Turkish_Turkey', '1254')

[khaeru]

Can you try both of the following and report the results?

1. From the command-line, run python -c "import ixmp; mp = ixmp.Platform(); mp.add_unit("foo").
2. In the tutorial notebook you are using, add a line mp.add_unit("foo") above the offending line (or in a new cell above that cell) and run it.

[gorkemgungormetu]

Can you try both of the following and report the results?

1. From the command-line, run python -c "import ixmp; mp = ixmp.Platform(); mp.add_unit("foo").

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\ggungor\Anaconda3\envs\message_env\lib\site-packages\ixmp\core\platform.py", line 92, in __init__
    self._backend = backend_class(**kwargs)
  File "C:\Users\ggungor\Anaconda3\envs\message_env\lib\site-packages\ixmp\backend\jdbc.py", line 287, in __init__
    _raise_jexception(e)
  File "C:\Users\ggungor\Anaconda3\envs\message_env\lib\site-packages\ixmp\backend\jdbc.py", line 138, in _raise_jexception
    raise RuntimeError(msg) from None
RuntimeError: unhandled Java exception: 
Unable to obtain connection from database (jdbc:hsqldb:file:C:/Users/ggungor/.local/share/ixmp/localdb/default) for user 'ixmp': Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@e517d533[file =C:\Users\ggungor\.local\share\ixmp\localdb\default.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2022-02-10 13:05:03 heartbeat - read: -7175 ms.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State  : S1000
Error Code : -451
Message    : Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@e517d533[file =C:\Users\ggungor\.local\share\ixmp\localdb\default.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2022-02-10 13:05:03 heartbeat - read: -7175 ms.

2. In the tutorial notebook you are using, add a line mp.add_unit("foo") above the offending line (or in a new cell above that cell) and run it.

mp.add_unit("foo")

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

File Platform.java:678, in at.ac.iiasa.ixmp.Platform.addUnitToDB()


File DbDAO.java:1327, in at.ac.iiasa.ixmp.database.DbDAO.addUnitToDB()


File DbDAO.java:836, in at.ac.iiasa.ixmp.database.DbDAO.getNextId()


Exception: Java Exception


The above exception was the direct cause of the following exception:


at.ac.iiasa.ixmp.exceptions.IxException   Traceback (most recent call last)

Input In [116], in <module>
----> 1 mp.add_unit("foo")


File ~\Anaconda3\envs\message_env\lib\site-packages\ixmp\core\platform.py:259, in Platform.add_unit(self, unit, comment)
    256     log.info(f"unit `{unit}` is already defined in the platform instance")
    257     return
--> 259 self._backend.set_unit(unit, comment)


File ~\Anaconda3\envs\message_env\lib\site-packages\ixmp\backend\jdbc.py:462, in JDBCBackend.set_unit(self, name, comment)
    461 def set_unit(self, name, comment):
--> 462     self.jobj.addUnitToDB(name, comment)


at.ac.iiasa.ixmp.exceptions.IxException: at.ac.iiasa.ixmp.exceptions.IxException: Cannot generate id for unıt

[khaeru]

I suspect the notebook error is a consequence of the database being locked; that, in turn, is the result of some code you ran earlier exiting prematurely.

Assuming you are only running the tutorials so far and do not have any other data, you can delete the files for the locked database (C:/Users/ggungor/.local/share/ixmp/localdb/default.*) and try again.

[gorkemgungormetu]

I suspect the notebook error is a consequence of the database being locked; that, in turn, is the result of some code you ran earlier exiting prematurely.

Assuming you are only running the tutorials so far and do not have any other data, you can delete the files for the locked database (C:/Users/ggungor/.local/share/ixmp/localdb/default.*) and try again.

I deleted the files and ran python python -c "import ixmp; mp = ixmp.Platform(); mp.add_unit("foo") and connected to ixmp but with Java Exception.

2022-02-11 11:23:03,188  INFO at.ac.iiasa.ixmp.Platform:146 - Welcome to the IX modeling platform!
2022-02-11 11:23:03,188  INFO at.ac.iiasa.ixmp.Platform:147 -  connected to database 'jdbc:hsqldb:file:C:/Users/ggungor/.local/share/ixmp/localdb/default' (user: ixmp)...
Traceback (most recent call last):
  File "Platform.java", line 678, in at.ac.iiasa.ixmp.Platform.addUnitToDB
  File "DbDAO.java", line 1327, in at.ac.iiasa.ixmp.database.DbDAO.addUnitToDB
  File "DbDAO.java", line 836, in at.ac.iiasa.ixmp.database.DbDAO.getNextId
Exception: Java Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\ggungor\Anaconda3\envs\message_env\lib\site-packages\ixmp\core\platform.py", line 259, in add_unit
    self._backend.set_unit(unit, comment)
  File "C:\Users\ggungor\Anaconda3\envs\message_env\lib\site-packages\ixmp\backend\jdbc.py", line 462, in set_unit
    self.jobj.addUnitToDB(name, comment)
at.ac.iiasa.ixmp.exceptions.at.ac.iiasa.ixmp.exceptions.IxException: at.ac.iiasa.ixmp.exceptions.IxException: Cannot generate id for unıt

It could be due to character encoding as there is usually mixing between english character "i" and turkish character "ı" which is also apparent in command output as unit is spelled as "unıt" at the last sentence. I changed my locale to English and encoding to UTF-8 but did not result in any difference.

ixmp:        3.4.0
message_ix:  3.4.0
message_ix_models: None
message_data: None

click:       8.0.3
dask:        2022.01.1
genno:       installed
graphviz:    None
jpype:       1.3.0
… JVM path:  C:\Users\ggungor\Anaconda3\envs\message_env\Library\bin\server\jvm.dll
openpyxl:    3.0.9
pandas:      1.4.0
pint:        0.18
xarray:      0.21.1
yaml:        6.0

iam_units:   installed
jupyter:     installed
matplotlib:  3.5.1
plotnine:    0.8.0
pyam:        1.3.1

GAMS:        35.2.0

python:      3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:22:07) [MSC v.1929 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder:   little
LC_ALL:      en_US.UTF-8
LANG:        en_US.UTF-8
LOCALE:      ('English_United States', '1254')

[khaeru]

It could be due to character encoding as there is usually mixing between english character "i" and turkish character "ı" which is also apparent in command output as unit is spelled as "unıt" at the last sentence.

Thanks for checking, and good eye. I was also going to guess about the encoding of strings, but more related to your "installation cost" argument getting altered somewhere in the code.

This is at least related (if not directly caused by) an old long-known bug in some Java code underlying the JDBCBackend (iiasa/ixmp_source#288, unfortunately not in a public repo). That bug report is just a link to https://garygregory.wordpress.com/2015/11/03/java-lowercase-conversion-turkey/. What happens is there is a database table named "UNIT", and to format the string, Java's String.toLowerCase() is called, which uses the current locale and produces "unıt". It's possible this is happening elsewhere in the code.

Can you please try the following—if possible with your system locale in each of tr_TR (as in the original report) and en_US (the second attempt)?

import ixmp
mp = ixmp.Platform(name="default", jvmargs="-Duser.language=en")
mp.add_unit("foo")

[gorkemgungormetu]

It could be due to character encoding as there is usually mixing between english character "i" and turkish character "ı" which is also apparent in command output as unit is spelled as "unıt" at the last sentence.

Thanks for checking, and good eye. I was also going to guess about the encoding of strings, but more related to your "installation cost" argument getting altered somewhere in the code.

This is at least related (if not directly caused by) an old long-known bug in some Java code underlying the JDBCBackend (iiasa/ixmp_source#288, unfortunately not in a public repo). That bug report is just a link to https://garygregory.wordpress.com/2015/11/03/java-lowercase-conversion-turkey/. What happens is there is a database table named "UNIT", and to format the string, Java's String.toLowerCase() is called, which uses the current locale and produces "unıt". It's possible this is happening elsewhere in the code.

Can you please try the following—if possible with your system locale in each of tr_TR (as in the original report) and en_US (the second attempt)?

import ixmp
mp = ixmp.Platform(name="default", jvmargs="-Duser.language=en")
mp.add_unit("foo")

This solves the issue for both tr_TR and en_US system locale. Thank you for your help.

[khaeru]

Great, glad to hear it.

I would still consider that an "unadvertised workaround," so we need to do at least 1 thing and maybe 2:

1. Improve our docs to warn users when this issue might occur and how to work around it.
2. Possibly put in an automatic workaround, e.g. when the locale is tr_TR then automatically add "-Duser.language=en" to the JVM arguments.

So I reopen the issue to track those actions, but nothing further needed from you! Thanks again for the report.