Missing import guards in tests

[karina-marques]

Hello!

I am installing Open Data Cube on Ubuntu 22.04.1. I have been following these instructions https://datacube-core.readthedocs.io/en/latest/installation/setup/ubuntu.html, but something is not working well. The message below appears when I run the integration tests within miniconda3 environment (I am using Python 3.10.9):

================================================================= short test summary info ===================================================================
SKIPPED [1] ../miniconda3/envs/odc_env/lib/python3.10/site-packages/_pytest/doctest.py:548: unable to import module PosixPath('/home/kmarques/datacube-core/tests/test_utils_dask.py')
SKIPPED [2] integration_tests/test_3d.py:26: could not import 'dcio_example.xarray_3d': No module named 'dcio_example'
ERROR tests/test_utils_dask.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I have tried to install "dcio_example" and "PosixPath", but I have got an error message:

(odc_env) kmarques@kmarques-VivoBook-ASUSLaptop-X515EA-F1500EA:~/datacube-core$ pip install dcio_example 
ERROR: Could not find a version that satisfies the requirement dcio_example (from versions: none) 
ERROR: No matching distribution found for dcio_example

[Kirill888]

@karina-marques thanks for the report. Real issue here is that test_3d.py does not include import guards even though it relies on optional dependency, also dev setup documentation is lacking.

Sample plugin is not on pypi, but is included in this repo in examples/io_plugin/, it can be installed from datacube checkout like so:

pip install examples/io_plugin

[Kirill888]

Hm, actually guards are present in test_3d.py, so some of the imports in test_utils_dask.py have optional dependency requirement that is not being checked.

Ideally we should run tests on some "minimal test environment" as well as "full test environment" to catch issues like that.

[karina-marques]

Hello @Kirill888

Thank you for your message and support.

I successfully installed the recommended plugin, but I still get an error message (see below).

================================================================== short test summary info =================================================================== SKIPPED [1] ../miniconda3/envs/odc_env/lib/python3.10/site-packages/_pytest/doctest.py:548: unable to import module PosixPath('/home/kmarques/datacube-core/tests/test_utils_dask.py') ERROR tests/test_utils_dask.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ========================================================== 1 skipped, 7 warnings, 1 error in 1.52s ===========================================================

[Kirill888]

@karina-marques

can you please try adding the following dependencies to your environment:

mamba install -n odc_env -c conda-forge pytest pytest-httpserver pytest-timeout pytest-cov moto 

Also can you point me to the documentation you were following, we might need to update it.

[karina-marques]

@Kirill888

I installed these dependencies, but I got more errors than before (154 in total). I suspect that it is a problem with the moto dependency, because when I uninstall it, it returns only an error, described below.

============================================================================================= short test summary info ============================================================================================= SKIPPED [1] ../miniconda3/envs/odc_env/lib/python3.10/site-packages/_pytest/doctest.py:555: unable to import module PosixPath('/home/kmarques/datacube-core/tests/test_utils_dask.py') ERROR tests/test_utils_dask.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ===================================================================================== 1 skipped, 3 warnings, 1 error in 1.72s =====================================================================================

However, it appears that it is a required dependency.

______________________ ImportError while importing test module '/home/kmarques/datacube-core/tests/test_utils_dask.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../miniconda3/envs/odc_env/lib/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/test_utils_dask.py:6: in <module> import moto E ModuleNotFoundError: No module named 'moto' ================================================================================================ warnings summary =================================================================================================

I am following this documentation https://datacube-core.readthedocs.io/en/latest/installation/setup/ubuntu.html. I installed the most recent version of postgresql (i.e. postgresql-14).