Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in WriteH5AD when SCE contains datetime objects #24

Open
giovp opened this issue Oct 9, 2020 · 5 comments
Open

Error in WriteH5AD when SCE contains datetime objects #24

giovp opened this issue Oct 9, 2020 · 5 comments
Labels
bug Something isn't working

Comments

@giovp
Copy link
Member

giovp commented Oct 9, 2020

Hi,

I've tried to use zellkonverter to write a sce to disk as anndata. This is the log and error (at the end). Any pointers on how to solve it? I'm also happy to drop problematic slots if helpful (technically just need obs+counts)

Log + error 
> writeH5AD(sce_noribo, file = "./Data_exchange_Giovanni/RData/1_sce_without-ribosomal-proteins.h5ad")
PREFIX=/Users/giovanni.palla/Library/Caches/basilisk/1.1.18/0
Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done                                       
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/giovanni.palla/Library/Caches/basilisk/1.1.18/0

  added / updated specs:
    - asn1crypto==1.3.0=py37_0
    - ca-certificates==2020.1.1=0
    - certifi==2019.11.28=py37_0
    - cffi==1.14.0=py37hb5b8e2f_0
    - chardet==3.0.4=py37_1003
    - conda-package-handling==1.6.0=py37h1de35cc_0
    - conda==4.8.2=py37_0
    - cryptography==2.8=py37ha12b0ac_0
    - idna==2.8=py37_0
    - libcxx==4.0.1=hcfea43d_1
    - libcxxabi==4.0.1=hcfea43d_1
    - libedit==3.1.20181209=hb402a30_0
    - libffi==3.2.1=h475c297_4
    - ncurses==6.2=h0a44026_0
    - openssl==1.1.1d=h1de35cc_4
    - pip==20.0.2=py37_1
    - pycosat==0.6.3=py37h1de35cc_0
    - pycparser==2.19=py37_0
    - pyopenssl==19.1.0=py37_0
    - pysocks==1.7.1=py37_0
    - python.app==2=py37_10
    - python==3.7.6=h359304d_2
    - readline==7.0=h1de35cc_5
    - requests==2.22.0=py37_1
    - ruamel_yaml==0.15.87=py37h1de35cc_0
    - setuptools==45.2.0=py37_0
    - six==1.14.0=py37_0
    - sqlite==3.31.1=ha441bb4_0
    - tk==8.6.8=ha441bb4_0
    - tqdm==4.42.1=py_0
    - urllib3==1.25.8=py37_0
    - wheel==0.34.2=py37_0
    - xz==5.2.4=h1de35cc_4
    - yaml==0.1.7=hc338f04_2
    - zlib==1.2.11=h1de35cc_3


The following NEW packages will be INSTALLED:

  asn1crypto         pkgs/main/osx-64::asn1crypto-1.3.0-py37_0
  ca-certificates    pkgs/main/osx-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/osx-64::certifi-2019.11.28-py37_0
  cffi               pkgs/main/osx-64::cffi-1.14.0-py37hb5b8e2f_0
  chardet            pkgs/main/osx-64::chardet-3.0.4-py37_1003
  conda              pkgs/main/osx-64::conda-4.8.2-py37_0
  conda-package-han~ pkgs/main/osx-64::conda-package-handling-1.6.0-py37h1de35cc_0
  cryptography       pkgs/main/osx-64::cryptography-2.8-py37ha12b0ac_0
  idna               pkgs/main/osx-64::idna-2.8-py37_0
  libcxx             pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
  libcxxabi          pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
  libedit            pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
  libffi             pkgs/main/osx-64::libffi-3.2.1-h475c297_4
  ncurses            pkgs/main/osx-64::ncurses-6.2-h0a44026_0
  openssl            pkgs/main/osx-64::openssl-1.1.1d-h1de35cc_4
  pip                pkgs/main/osx-64::pip-20.0.2-py37_1
  pycosat            pkgs/main/osx-64::pycosat-0.6.3-py37h1de35cc_0
  pycparser          pkgs/main/osx-64::pycparser-2.19-py37_0
  pyopenssl          pkgs/main/osx-64::pyopenssl-19.1.0-py37_0
  pysocks            pkgs/main/osx-64::pysocks-1.7.1-py37_0
  python             pkgs/main/osx-64::python-3.7.6-h359304d_2
  python.app         pkgs/main/osx-64::python.app-2-py37_10
  readline           pkgs/main/osx-64::readline-7.0-h1de35cc_5
  requests           pkgs/main/osx-64::requests-2.22.0-py37_1
  ruamel_yaml        pkgs/main/osx-64::ruamel_yaml-0.15.87-py37h1de35cc_0
  setuptools         pkgs/main/osx-64::setuptools-45.2.0-py37_0
  six                pkgs/main/osx-64::six-1.14.0-py37_0
  sqlite             pkgs/main/osx-64::sqlite-3.31.1-ha441bb4_0
  tk                 pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
  tqdm               pkgs/main/noarch::tqdm-4.42.1-py_0
  urllib3            pkgs/main/osx-64::urllib3-1.25.8-py37_0
  wheel              pkgs/main/osx-64::wheel-0.34.2-py37_0
  xz                 pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
  yaml               pkgs/main/osx-64::yaml-0.1.7-hc338f04_2
  zlib               pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3


Preparing transaction: ...working... done
Executing transaction: ...working... done
installation finished.
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/giovanni.palla/Library/Caches/basilisk/1.1.18/0

  added / updated specs:
    - nomkl


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-2.17                  |         openblas          11 KB  conda-forge
    ca-certificates-2020.7.22  |                0         124 KB
    certifi-2020.6.20          |           py37_0         156 KB
    conda-4.8.5                |           py37_0         2.9 MB
    libblas-3.8.0              |      17_openblas          11 KB  conda-forge
    libcblas-3.8.0             |      17_openblas          11 KB  conda-forge
    libgfortran-4.0.0          |      h50e675f_12          18 KB  conda-forge
    libgfortran4-7.5.0         |      h50e675f_12         1.0 MB  conda-forge
    liblapack-3.8.0            |      17_openblas          11 KB  conda-forge
    liblapacke-3.8.0           |      17_openblas          11 KB  conda-forge
    libopenblas-0.3.10         |openmp_h63d9170_3         8.2 MB  conda-forge
    llvm-openmp-10.0.0         |       h28b9765_0         236 KB
    nomkl-3.0                  |                0          45 KB
    openssl-1.1.1h             |       haf1e3a3_0         2.2 MB
    ------------------------------------------------------------
                                           Total:        15.0 MB

The following NEW packages will be INSTALLED:

  blas               conda-forge/osx-64::blas-2.17-openblas
  libblas            conda-forge/osx-64::libblas-3.8.0-17_openblas
  libcblas           conda-forge/osx-64::libcblas-3.8.0-17_openblas
  libgfortran        conda-forge/osx-64::libgfortran-4.0.0-h50e675f_12
  libgfortran4       conda-forge/osx-64::libgfortran4-7.5.0-h50e675f_12
  liblapack          conda-forge/osx-64::liblapack-3.8.0-17_openblas
  liblapacke         conda-forge/osx-64::liblapacke-3.8.0-17_openblas
  libopenblas        conda-forge/osx-64::libopenblas-0.3.10-openmp_h63d9170_3
  llvm-openmp        pkgs/main/osx-64::llvm-openmp-10.0.0-h28b9765_0
  nomkl              pkgs/main/osx-64::nomkl-3.0-0

The following packages will be UPDATED:

  ca-certificates                                2020.1.1-0 --> 2020.7.22-0
  certifi                                 2019.11.28-py37_0 --> 2020.6.20-py37_0
  conda                                        4.8.2-py37_0 --> 4.8.5-py37_0
  openssl                                 1.1.1d-h1de35cc_4 --> 1.1.1h-haf1e3a3_0



Downloading and Extracting Packages
blas-2.17            | 11 KB     | ########## | 100% 
nomkl-3.0            | 45 KB     | ########## | 100% 
llvm-openmp-10.0.0   | 236 KB    | ########## | 100% 
libgfortran4-7.5.0   | 1.0 MB    | ########## | 100% 
libcblas-3.8.0       | 11 KB     | ########## | 100% 
liblapacke-3.8.0     | 11 KB     | ########## | 100% 
conda-4.8.5          | 2.9 MB    | ########## | 100% 
ca-certificates-2020 | 124 KB    | ########## | 100% 
liblapack-3.8.0      | 11 KB     | ########## | 100% 
libblas-3.8.0        | 11 KB     | ########## | 100% 
certifi-2020.6.20    | 156 KB    | ########## | 100% 
libopenblas-0.3.10   | 8.2 MB    | ########## | 100% 
libgfortran-4.0.0    | 18 KB     | ########## | 100% 
openssl-1.1.1h       | 2.2 MB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/giovanni.palla/Library/Caches/basilisk/1.1.18/zellkonverter-0.99.5/anndata_env

  added / updated specs:
    - python=3.7.6


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.6.20  |       hecda079_0         146 KB  conda-forge
    certifi-2020.6.20          |   py37h2987424_2         151 KB  conda-forge
    libcxx-10.0.1              |       h5f48129_0         1.0 MB  conda-forge
    libffi-3.2.1               |    hb1e8313_1007          42 KB  conda-forge
    ncurses-6.2                |       hb1e8313_1         902 KB  conda-forge
    openssl-1.1.1h             |       haf1e3a3_0         1.9 MB  conda-forge
    pip-20.2.3                 |             py_0         1.1 MB  conda-forge
    python-3.7.6               |cpython_h1fd5dd1_6        23.7 MB  conda-forge
    python_abi-3.7             |          1_cp37m           4 KB  conda-forge
    readline-8.0               |       h0678c8f_2         255 KB  conda-forge
    setuptools-49.6.0          |   py37h2987424_2         939 KB  conda-forge
    sqlite-3.33.0              |       h960bd1c_1         1.7 MB  conda-forge
    tk-8.6.10                  |       hb0a8c7a_1         3.3 MB  conda-forge
    wheel-0.35.1               |     pyh9f0ad1d_0          29 KB  conda-forge
    xz-5.2.5                   |       haf1e3a3_1         228 KB  conda-forge
    zlib-1.2.11                |    h7795811_1009         102 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        35.3 MB

The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/osx-64::ca-certificates-2020.6.20-hecda079_0
  certifi            conda-forge/osx-64::certifi-2020.6.20-py37h2987424_2
  libcxx             conda-forge/osx-64::libcxx-10.0.1-h5f48129_0
  libffi             conda-forge/osx-64::libffi-3.2.1-hb1e8313_1007
  ncurses            conda-forge/osx-64::ncurses-6.2-hb1e8313_1
  openssl            conda-forge/osx-64::openssl-1.1.1h-haf1e3a3_0
  pip                conda-forge/noarch::pip-20.2.3-py_0
  python             conda-forge/osx-64::python-3.7.6-cpython_h1fd5dd1_6
  python_abi         conda-forge/osx-64::python_abi-3.7-1_cp37m
  readline           conda-forge/osx-64::readline-8.0-h0678c8f_2
  setuptools         conda-forge/osx-64::setuptools-49.6.0-py37h2987424_2
  sqlite             conda-forge/osx-64::sqlite-3.33.0-h960bd1c_1
  tk                 conda-forge/osx-64::tk-8.6.10-hb0a8c7a_1
  wheel              conda-forge/noarch::wheel-0.35.1-pyh9f0ad1d_0
  xz                 conda-forge/osx-64::xz-5.2.5-haf1e3a3_1
  zlib               conda-forge/osx-64::zlib-1.2.11-h7795811_1009



Downloading and Extracting Packages
python_abi-3.7       | 4 KB      | ########## | 100% 
python-3.7.6         | 23.7 MB   | ########## | 100% 
wheel-0.35.1         | 29 KB     | ########## | 100% 
libffi-3.2.1         | 42 KB     | ########## | 100% 
zlib-1.2.11          | 102 KB    | ########## | 100% 
openssl-1.1.1h       | 1.9 MB    | ########## | 100% 
xz-5.2.5             | 228 KB    | ########## | 100% 
sqlite-3.33.0        | 1.7 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
setuptools-49.6.0    | 939 KB    | ########## | 100% 
readline-8.0         | 255 KB    | ########## | 100% 
libcxx-10.0.1        | 1.0 MB    | ########## | 100% 
tk-8.6.10            | 3.3 MB    | ########## | 100% 
ca-certificates-2020 | 146 KB    | ########## | 100% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 
ncurses-6.2          | 902 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate /Users/giovanni.palla/Library/Caches/basilisk/1.1.18/zellkonverter-0.99.5/anndata_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/giovanni.palla/Library/Caches/basilisk/1.1.18/zellkonverter-0.99.5/anndata_env

  added / updated specs:
    - anndata=0.7.4
    - h5py=2.10.0
    - hdf5=1.10.5
    - natsort=7.0.1
    - numpy=1.18.5
    - packaging=20.4
    - pandas=1.0.4
    - python=3.7.6
    - scipy=1.4.1
    - sqlite=3.30.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    anndata-0.7.4              |   py37hc8dfbb8_0         187 KB  conda-forge
    h5py-2.10.0                |nompi_py37h106b333_102        1015 KB  conda-forge
    hdf5-1.10.5                |nompi_h3e39495_1104         3.0 MB  conda-forge
    importlib-metadata-2.0.0   |   py37hc8dfbb8_0          44 KB  conda-forge
    importlib_metadata-2.0.0   |                0           3 KB  conda-forge
    libopenblas-0.3.10         |openmp_h63d9170_4         8.2 MB  conda-forge
    llvm-openmp-10.0.1         |       h28b9765_0         265 KB  conda-forge
    natsort-7.0.1              |             py_0          32 KB  conda-forge
    numpy-1.18.5               |   py37h7687784_0         5.0 MB  conda-forge
    packaging-20.4             |     pyh9f0ad1d_0          32 KB  conda-forge
    pandas-1.0.4               |   py37h94625e5_0         9.6 MB  conda-forge
    pyparsing-2.4.7            |     pyh9f0ad1d_0          60 KB  conda-forge
    python-dateutil-2.8.1      |             py_0         220 KB  conda-forge
    pytz-2020.1                |     pyh9f0ad1d_0         227 KB  conda-forge
    scipy-1.4.1                |   py37hce1b9e5_3        18.9 MB  conda-forge
    six-1.15.0                 |     pyh9f0ad1d_0          14 KB  conda-forge
    sqlite-3.30.1              |       h93121df_0         2.5 MB  conda-forge
    zipp-3.3.0                 |             py_0          11 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        49.3 MB

The following NEW packages will be INSTALLED:

  anndata            conda-forge/osx-64::anndata-0.7.4-py37hc8dfbb8_0
  h5py               conda-forge/osx-64::h5py-2.10.0-nompi_py37h106b333_102
  hdf5               conda-forge/osx-64::hdf5-1.10.5-nompi_h3e39495_1104
  importlib-metadata conda-forge/osx-64::importlib-metadata-2.0.0-py37hc8dfbb8_0
  importlib_metadata conda-forge/noarch::importlib_metadata-2.0.0-0
  libblas            conda-forge/osx-64::libblas-3.8.0-17_openblas
  libcblas           conda-forge/osx-64::libcblas-3.8.0-17_openblas
  libgfortran        conda-forge/osx-64::libgfortran-4.0.0-h50e675f_12
  libgfortran4       conda-forge/osx-64::libgfortran4-7.5.0-h50e675f_12
  liblapack          conda-forge/osx-64::liblapack-3.8.0-17_openblas
  libopenblas        conda-forge/osx-64::libopenblas-0.3.10-openmp_h63d9170_4
  llvm-openmp        conda-forge/osx-64::llvm-openmp-10.0.1-h28b9765_0
  natsort            conda-forge/noarch::natsort-7.0.1-py_0
  numpy              conda-forge/osx-64::numpy-1.18.5-py37h7687784_0
  packaging          conda-forge/noarch::packaging-20.4-pyh9f0ad1d_0
  pandas             conda-forge/osx-64::pandas-1.0.4-py37h94625e5_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.1-py_0
  pytz               conda-forge/noarch::pytz-2020.1-pyh9f0ad1d_0
  scipy              conda-forge/osx-64::scipy-1.4.1-py37hce1b9e5_3
  six                conda-forge/noarch::six-1.15.0-pyh9f0ad1d_0
  zipp               conda-forge/noarch::zipp-3.3.0-py_0

The following packages will be DOWNGRADED:

  sqlite                                  3.33.0-h960bd1c_1 --> 3.30.1-h93121df_0



Downloading and Extracting Packages
importlib-metadata-2 | 44 KB     | ########## | 100% 
six-1.15.0           | 14 KB     | ########## | 100% 
libopenblas-0.3.10   | 8.2 MB    | ########## | 100% 
scipy-1.4.1          | 18.9 MB   | ########## | 100% 
sqlite-3.30.1        | 2.5 MB    | ########## | 100% 
hdf5-1.10.5          | 3.0 MB    | ########## | 100% 
numpy-1.18.5         | 5.0 MB    | ########## | 100% 
pandas-1.0.4         | 9.6 MB    | ########## | 100% 
importlib_metadata-2 | 3 KB      | ########## | 100% 
anndata-0.7.4        | 187 KB    | ########## | 100% 
python-dateutil-2.8. | 220 KB    | ########## | 100% 
packaging-20.4       | 32 KB     | ########## | 100% 
llvm-openmp-10.0.1   | 265 KB    | ########## | 100% 
pytz-2020.1          | 227 KB    | ########## | 100% 
natsort-7.0.1        | 32 KB     | ########## | 100% 
pyparsing-2.4.7      | 60 KB     | ########## | 100% 
h5py-2.10.0          | 1015 KB   | ########## | 100% 
zipp-3.3.0           | 11 KB     | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Note: using the 'counts' assay as the X matrix
... storing 'gene_type' as categorical
... storing 'gene_name' as categorical
... storing 'entrez_id' as categorical
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  TypeError: No conversion path for dtype: dtype('<M8[ns]')

Above error raised while writing key 'uns/tximetaInfo/importTime' of <class 'h5py._hl.files.File'> from /.

Thank you!
@lazappi
Copy link
Member

lazappi commented Oct 9, 2020

Hmmm... I'm guessing this is a problem with trying to convert something that doesn't have a matching Python type. Can you post the result of using str() on this object in the R environment (I think it will be somewhere like metadata(sce, "tximetaInfo")?

If you want to get it working ASAP easiest thing is probably to delete the troublesome object with something like metadata(sce, "tximetaInfo") <- NULL.

@LTLA
Copy link
Contributor

LTLA commented Oct 9, 2020

Yeah, we should probably throw warnings and ignore unconvertible objects. Not familiar enough with reticulate to know exactly how to do that, though I suppose we could just always try and handle the errors.

@giovp
Copy link
Member Author

giovp commented Oct 11, 2020

had to go for this cause same error popped up again in metadata
metadata(sce_noribo) <- list()

sorry for not providing reproducible example, here how the metadata object look like:

metadata 
$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 09:26:04 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 09:31:54 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 09:41:53 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 09:48:54 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 09:54:36 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 10:01:16 CEST"


$countsFromAbundance
[1] "no"

$tximetaInfo
$tximetaInfo$version
[1] ‘1.6.3’

$tximetaInfo$importTime
[1] "2020-08-28 10:05:46 CEST"


$countsFromAbundance
[1] "no"

@giovp giovp closed this as completed Oct 11, 2020
@lazappi
Copy link
Member

lazappi commented Oct 12, 2020

Thanks! I think it's probably the timestamps that are the issue. I'm going to reopen this as a reminder to add better checks to this part of the code.

@lazappi lazappi reopened this Oct 12, 2020
@lazappi
Copy link
Member

lazappi commented Oct 16, 2020 •
edited

Looking into this further the problem isn't anything to do with converting dates from R to Python, it's just that h5py can't really handle datetime objects. Despite this being a big scary error you still get a .h5ad file written to disk which seems to be perfectly fine, just missing the time data.

Not sure exactly how to handle this, possible options:

  1. Convert all dates to strings (but quite possible to miss something).
  2. Catch the Python error and give one in R that explains the problem a bit better.
  3. Catch the error, extract what is causing the problem and convert it automatically (not sure how possible that is)
  4. Open an issue in anndata and see if it can be handled there
  5. Ignore it and just wait for h5py v3.0 which should fix the problem (maybe ok because you still get a file)
  6. Some combination of the above?

@lazappi lazappi changed the title error occured while writing sce to anndata Error in WriteH5AD when SCE contains datetime objects Oct 16, 2020
@lazappi lazappi added the bug Something isn't working label Nov 16, 2020
@lazappi lazappi mentioned this issue Nov 16, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lazappi @LTLA @giovp