arrange desc cause error

[romangehrn]

Hi, I'm using sparklyr on Databricks DBR 14+, when i would like to arrange in descending order an error occurs, ascending is fine.

Here a small example:

library(magrittr)

sc <- sparklyr::spark_connect(method="databricks")

sdf_write <- sparklyr::copy_to(sc, mtcars, overwrite = T) %>%
  dbplyr::window_order(mpg) %>%
  dplyr::group_by(cyl) %>%
  dplyr::filter(row_number() == 1) %>%
  dplyr::ungroup() %>%
  dplyr::arrange(desc(cyl))

sparklyr::spark_write_table(sdf_write, name = "ch_lab.construction_project.test_project", mode = "overwrite")

Causing the following error:

Error in purrr::map_chr(order, as_name) : ℹ In index: 1.
Caused by error in `as_string()`:
! Can't convert a call to a string.
Error in `purrr::map_chr()`:

sessionInfo():

R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] magrittr_2.0.3

loaded via a namespace (and not attached):
 [1] jsonlite_1.8.7    sparklyr_1.8.1    dplyr_1.1.2       compiler_4.3.1   
 [5] tidyselect_1.2.0  blob_1.2.4        parallel_4.3.1    assertthat_0.2.1 
 [9] tidyr_1.3.0       yaml_2.3.7        fastmap_1.1.1     Rserve_1.8-11    
[13] R6_2.5.1          generics_0.1.3    htmlwidgets_1.6.2 tibble_3.2.1     
[17] SparkR_3.5.0      rprojroot_2.0.3   DBI_1.1.3         pillar_1.9.0     
[21] rlang_1.1.1       utf8_1.2.3        config_0.3.1      r2d3_0.2.6       
[25] forge_0.2.0       cli_3.6.1         withr_2.5.0       digest_0.6.33    
[29] rstudioapi_0.15.0 dbplyr_2.3.3      base64enc_0.1-3   lifecycle_1.0.3  
[33] vctrs_0.6.3       glue_1.6.2        fansi_1.0.4       purrr_1.0.1      
[37] httr_1.4.6        tools_4.3.1       pkgconfig_2.0.3   ellipsis_0.3.2   
[41] htmltools_0.5.5    

[edgararuiz]

Hi! For DBR 13+ please install the extension package pysparklyr, and use the new method of databricks_connect. Here are the full instructions: https://spark.rstudio.com/deployment/databricks-connect.html

[github-actions]

Automatically closed because there has not been a response for 30 days. When you're ready to work on this further, please comment here and the issue will automatically reopen.

[dabruehl]

Hi @edgararuiz

I got same issue on Databricks 14.3 LTS. I do not understand your provided solution?

Why should I use databricks_connect, if I am already on the cluster and logged in with my credentials. This databricks_connect is great, if I am in RStudio and need compute power or data from Databricks.