Cluster State Not Being Recognized

[mrworthington]

Hi Edgar + The sparklyr team!

Issue Overview

I've been following through this post and the docs here. I've created a cluster whose DRB is 13.3 and set the Access Mode to "Single User", which you can see a snapshot of below under "Screenshot". Despite these settings and following through both the docs and the blog post from posit, I'm getting this error which isn't listed in the docs "Reported Problems" section. For the life of me, I can't understand why it's saying my cluster doesn't have "Single User" access mode enabled when the portal says it is. FWIW, the cluster IDs have been scrubbed from here, but they match between the portal and what's stored in my R environment.

Here's what I run:

library(sparklyr)

sc <- spark_connect(method = "databricks_connect")

Here's what pops out each time:

! Retrieving version from cluster 'XXXX-XXXXXX-XXXXXXXX'
✔ Cluster version: '13.3'
✔ Using the 'r-sparklyr-databricks-13.3' Python environment (C:/Users/[USER_NAME]/OneDrive/Documents/.virtualenvs/r-sparklyr-databricks-13.3/Scripts/python.exe)
Error in `cluster_dbr_error()`:
! Spark connection error
• Possible cause = The cluster is initializing. Try again later
• status = StatusCode.FAILED_PRECONDITION
• details = "INVALID_STATE: cluster XXXX-XXXXXX-XXXXXXXX is not Shared or Single User Cluster. (requestId=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX)"

Screenshot

Session Info from session_info()

sessioninfo::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.1 (2023-06-16 ucrt)
 os       Windows 10 x64 (build 19045)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_United States.utf8
 ctype    English_United States.utf8
 tz       America/Chicago
 date     2023-10-26
 rstudio  2023.09.1+494 Desert Sunflower (desktop)
 pandoc   NA

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version    date (UTC) lib source
 askpass       1.2.0      2023-09-03 [1] CRAN (R 4.3.1)
 base64enc     0.1-3      2015-07-28 [1] CRAN (R 4.3.1)
 cli           3.6.1      2023-03-23 [1] CRAN (R 4.3.1)
 config        0.3.2      2023-08-30 [1] CRAN (R 4.3.1)
 curl          5.1.0      2023-10-02 [1] CRAN (R 4.3.1)
 DBI         * 1.1.3      2022-06-18 [1] CRAN (R 4.3.1)
 dbplyr        2.3.4      2023-09-26 [1] CRAN (R 4.3.1)
 digest        0.6.33     2023-07-07 [1] CRAN (R 4.3.1)
 dplyr         1.1.3      2023-09-03 [1] CRAN (R 4.3.1)
 fansi         1.0.5      2023-10-08 [1] CRAN (R 4.3.1)
 fastmap       1.1.1      2023-02-24 [1] CRAN (R 4.3.1)
 fs            1.6.3      2023-07-20 [1] CRAN (R 4.3.1)
 generics      0.1.3      2022-07-05 [1] CRAN (R 4.3.1)
 glue          1.6.2      2022-02-24 [1] CRAN (R 4.3.1)
 gt            0.10.0     2023-10-07 [1] CRAN (R 4.3.1)
 htmltools     0.5.6.1    2023-10-06 [1] CRAN (R 4.3.1)
 httr          1.4.7      2023-08-15 [1] CRAN (R 4.3.1)
 httr2         0.2.3      2023-05-08 [1] CRAN (R 4.3.1)
 janitor       2.2.0      2023-02-02 [1] CRAN (R 4.3.1)
 jsonlite      1.8.7      2023-06-29 [1] CRAN (R 4.3.1)
 lattice       0.21-8     2023-04-05 [2] CRAN (R 4.3.1)
 lifecycle     1.0.3      2022-10-07 [1] CRAN (R 4.3.1)
 lubridate     1.9.3      2023-09-27 [1] CRAN (R 4.3.1)
 magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.3.1)
 Matrix        1.5-4.1    2023-05-18 [2] CRAN (R 4.3.1)
 openssl       2.1.1      2023-09-25 [1] CRAN (R 4.3.1)
 pillar        1.9.0      2023-03-22 [1] CRAN (R 4.3.1)
 pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.3.1)
 png           0.1-8      2022-11-29 [1] CRAN (R 4.3.1)
 processx      3.8.2      2023-06-30 [1] CRAN (R 4.3.1)
 ps            1.7.5      2023-04-18 [1] CRAN (R 4.3.1)
 purrr         1.0.2      2023-08-10 [1] CRAN (R 4.3.1)
 pysparklyr    0.1.9002   2023-10-24 [1] Github (mlverse/pysparklyr@beb4b31)
 R6            2.5.1      2021-08-19 [1] CRAN (R 4.3.1)
 rappdirs      0.3.3      2021-01-31 [1] CRAN (R 4.3.1)
 Rcpp          1.0.11     2023-07-06 [1] CRAN (R 4.3.1)
 reticulate    1.34.0     2023-10-12 [1] CRAN (R 4.3.1)
 rlang         1.1.1      2023-04-28 [1] CRAN (R 4.3.1)
 rstudioapi    0.15.0     2023-07-07 [1] CRAN (R 4.3.1)
 sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.3.1)
 snakecase     0.11.1     2023-08-27 [1] CRAN (R 4.3.1)
 sparklyr    * 1.8.3.9001 2023-10-24 [1] Github (sparklyr/sparklyr@49c6852)
 stringi       1.7.12     2023-01-11 [1] CRAN (R 4.3.1)
 stringr       1.5.0      2022-12-02 [1] CRAN (R 4.3.1)
 tibble        3.2.1      2023-03-20 [1] CRAN (R 4.3.1)
 tidyr         1.3.0      2023-01-24 [1] CRAN (R 4.3.1)
 tidyselect    1.2.0      2022-10-10 [1] CRAN (R 4.3.1)
 timechange    0.2.0      2023-01-11 [1] CRAN (R 4.3.1)
 utf8          1.2.4      2023-10-22 [1] CRAN (R 4.3.1)
 vctrs         0.6.4      2023-10-12 [1] CRAN (R 4.3.1)
 withr         2.5.1      2023-09-26 [1] CRAN (R 4.3.1)
 xml2          1.3.5      2023-07-06 [1] CRAN (R 4.3.1)
 yaml          2.3.7      2023-01-23 [1] CRAN (R 4.3.1)

 [1] C:/Users/[USER_NAME]/AppData/Local/R/win-library/4.3
 [2] C:/Program Files/R/R-4.3.1/library

─ Python configuration ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 python:         C:/Users/[USER_NAME]/OneDrive/Documents/.virtualenvs/r-sparklyr-databricks-13.3/Scripts/python.exe
 libpython:      C:/Users/[USER_NAME]/AppData/Local/r-reticulate/r-reticulate/pyenv/pyenv-win/versions/3.9.13/python39.dll
 pythonhome:     C:/Users/[USER_NAME]/OneDrive/Documents/.virtualenvs/r-sparklyr-databricks-13.3
 version:        3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
 Architecture:   64bit
 numpy:          C:/Users/[USER_NAME]/OneDrive/Documents/.virtualenvs/r-sparklyr-databricks-13.3/Lib/site-packages/numpy
 numpy_version:  1.26.1
 databricks:     C:\Users\[USER_NAME]\ONEDRI~1\DOCUME~1\VIRTUA~1\R-SPAR~2.3\lib\site-packages\databricks\__init__.p
 
 NOTE: Python version was forced by use_python() function

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────   

[edgararuiz]

Hi @mrworthington, this error is new to me as well. I asked the team at Databricks that I'm in contact with, and will reply to you as soon as I have something to share. Thanks.

[mrworthington]

Thank you! Will hold for now.

[edgararuiz]

Hi @mrworthington , it seems that your Databricks Workspace is not Unity Catalog enabled, and thus I'm being told that it is not a valid single-user cluster that the underlying Python library can connect to. Can you try with an Unity Catalog enabled Workspace?