Running sparklyr with YARN Docker-mode in EMR cluster

[pvieito]

Hi! I am trying to set up an sparklyr job in EMR. I am using YARN Docker-mode (YARN_CONTAINER_RUNTIME_TYPE=docker) and deploying the job with spark-submit:

sudo spark-submit --master yarn --deploy-mode cluster \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf spark.yarn.maxAppAttempts=1 \
--packages org.apache.sedona:sedona-python-adapter-3.0_2.12:1.3.1-incubating,org.datasyslab:geotools-wrapper:1.3.0-27.2 \
CommandRunner.R

library(sparklyr)
config <- spark_config()
sc <- spark_connect(master = "yarn", conf = config)

When running the execution is stuck with this messages:

23/04/03 12:22:01 INFO sparklyr: Session (55821) is starting under 127.0.0.1 port 8880
23/04/03 12:22:01 INFO sparklyr: Session (55821) found port 8880 is available
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) is waiting for sparklyr client to connect to port 8880
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) accepted connection
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) is waiting for sparklyr client to connect to port 8880
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) received command 0
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) found requested session matches current session
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) is creating backend and allocating system resources
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) is using port 8881 for backend channel
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) created the backend
23/04/03 12:22:01 INFO sparklyr: Gateway (55821) is waiting for R process to end
23/04/03 12:22:02 INFO HiveConf: Found configuration file null
23/04/03 12:22:02 INFO SparkContext: Running Spark version 3.3.2
23/04/03 12:22:02 INFO ResourceUtils: ==============================================================
23/04/03 12:22:02 INFO ResourceUtils: No custom resources configured for spark.driver.
23/04/03 12:22:02 INFO ResourceUtils: ==============================================================
23/04/03 12:22:02 INFO SparkContext: Submitted application: sparklyr
23/04/03 12:22:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/04/03 12:22:02 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
23/04/03 12:22:02 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/04/03 12:22:02 INFO SecurityManager: Changing view acls to: hadoop,root
23/04/03 12:22:02 INFO SecurityManager: Changing modify acls to: hadoop,root
23/04/03 12:22:02 INFO SecurityManager: Changing view acls groups to: 
23/04/03 12:22:02 INFO SecurityManager: Changing modify acls groups to: 
23/04/03 12:22:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop, root); groups with view permissions: Set(); users  with modify permissions: Set(hadoop, root); groups with modify permissions: Set()
23/04/03 12:22:02 INFO Utils: Successfully started service 'sparkDriver' on port 40419.
23/04/03 12:22:02 INFO SparkEnv: Registering MapOutputTracker
23/04/03 12:22:02 INFO SparkEnv: Registering BlockManagerMaster
23/04/03 12:22:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/04/03 12:22:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/04/03 12:22:03 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/04/03 12:22:03 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/root/appcache/application_1680508493561_0044/blockmgr-49a3dae7-e16e-4332-8e4f-e92bfbed66dd
23/04/03 12:22:03 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/blockmgr-22600d14-035e-4ce8-aaaf-d05919f739ef
23/04/03 12:22:03 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
23/04/03 12:22:03 INFO SparkEnv: Registering OutputCommitCoordinator
23/04/03 12:22:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/04/03 12:22:03 INFO SparkContext: Added JAR file:/usr/local/lib/R/site-library/sparklyr/java/sparklyr-master-2.12.jar at spark://ip-10-5-78-242.eu-west-1.compute.internal:40419/jars/sparklyr-master-2.12.jar with timestamp 1680524522444
23/04/03 12:22:03 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
23/04/03 12:22:03 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
23/04/03 12:22:04 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:05 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:06 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:07 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:08 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:09 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:10 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:11 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:12 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:13 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:14 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:15 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:16 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:17 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:18 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:19 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:20 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:21 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:23 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/04/03 12:22:23 INFO RetryInvocationHandler: java.net.ConnectException: Your endpoint configuration is wrong; For more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ApplicationClientProtocolPBClientImpl.getNewApplication over null after 1 failover attempts. Trying to failover after sleeping for 22758ms.

The same deployment works using SparkR:

library(SparkR)
sparkR.session(appName = "app-name")
sqlContext <- sparkRSQL.init(spark.sparkContext);
df <- sql("SELECT 1 AS A")
print(df)

This are the environment variables in the Docker container deployed by YARN:

_R_CHECK_COMPILATION_FLAGS_KNOWN_
                        -Wformat -Werror=format-security -Wdate-time
APP_SUBMIT_TIME_ENV     1680524490796
APPLICATION_WEB_PROXY_BASE
                        /proxy/application_1680508493561_0044
CLASSPATH               /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001:/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/__spark_conf__:/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/__spark_libs__/*:/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/__spark_conf__/__hadoop_conf__
CONTAINER_ID            container_1680508493561_0044_01_000001
DATATOOLS_ENVIRONMENT   PRO
DATATOOLS_LOGLEVEL      DEBUG
EDITOR                  vi
EXISTING_SPARKR_BACKEND_PORT
                        41487
GPG_KEY                 E3FF2839C048B25C084DEBE9B26995E310250568
HADOOP_COMMON_HOME      /usr/lib/hadoop
HADOOP_CONF_DIR         /etc/hadoop/conf
HADOOP_HDFS_HOME        /usr/lib/hadoop-hdfs
HADOOP_MAPRED_HOME      /usr/lib/hadoop-mapreduce
HADOOP_TOKEN_FILE_LOCATION
                        /mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/container_tokens
HADOOP_YARN_HOME        /usr/lib/hadoop-yarn
HOME                    /home/
HOSTNAME                10.5.78.242
JAVA_HOME               /etc/alternatives/jre
JVM_PID                 1
KMP_INIT_AT_FORK        FALSE
LANG                    C.UTF-8
LD_LIBRARY_PATH         /usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/etc/alternatives/jre/lib/server:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native:
LN_S                    ln -s
LOCAL_DIRS              /mnt/yarn/usercache/root/appcache/application_1680508493561_0044,/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044
LOCAL_USER_DIRS         /mnt/yarn/usercache/root/,/mnt1/yarn/usercache/root/
LOCALIZATION_COUNTERS   371597366,0,13,0,4092
LOG_DIRS                /var/log/hadoop-yarn/containers/application_1680508493561_0044/container_1680508493561_0044_01_000001
LOGNAME                 root
MAKE                    make
MALLOC_ARENA_MAX        4
NM_AUX_SERVICE_spark_shuffle
                        
NM_HOST                 ip-10-5-78-242.eu-west-1.compute.internal
NM_HTTP_PORT            8042
NM_PORT                 8041
NOAWT                   1
PAGER                   /usr/bin/pager
PATH                    /src/FootFall/venv3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin
PRELAUNCH_ERR           /var/log/hadoop-yarn/containers/application_1680508493561_0044/container_1680508493561_0044_01_000001/prelaunch.err
PRELAUNCH_OUT           /var/log/hadoop-yarn/containers/application_1680508493561_0044/container_1680508493561_0044_01_000001/prelaunch.out
PWD                     /mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001
PYSPARK_DRIVER_PYTHON   /src/FootFall/venv3/bin/python
PYSPARK_PYTHON          /usr/bin/python3
PYTHON_EXECUTABLE       /src/FootFall/venv3/bin/python
PYTHON_GET_PIP_SHA256   394be00f13fa1b9aaa47e911bdb59a09c3b2986472130f30aa0bfaf7f3980637
PYTHON_GET_PIP_URL      https://github.com/pypa/get-pip/raw/d5cb0afaf23b8520f1bbcfed521017b4a95f5c01/public/get-pip.py
PYTHON_PIP_VERSION      22.0.4
PYTHON_SETUPTOOLS_VERSION
                        58.1.0
PYTHON_VENV             /src/FootFall/venv3
PYTHON_VERSION          3.9.16
PYTHONHASHSEED          0
PYTHONPATH              /usr/local/lib/R/site-library/reticulate/config:/src/FootFall:/mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001:/usr/local/lib/python39.zip:/usr/local/lib/python3.9:/usr/local/lib/python3.9/lib-dynload:/src/FootFall/venv3/lib/python3.9/site-packages:/usr/local/lib/R/site-library/reticulate/python
R_ARCH                  
R_BROWSER               xdg-open
R_BZIPCMD               /bin/bzip2
R_DOC_DIR               /usr/share/R/doc
R_GZIPCMD               /bin/gzip -n
R_HOME                  /usr/lib/R
R_INCLUDE_DIR           /usr/share/R/include
R_LIBS_SITE             /usr/local/lib/R/site-library:/usr/lib/R/site-library:/usr/lib/R/library
R_LIBS_USER             ~/R/x86_64-pc-linux-gnu-library/4.0
R_PAPERSIZE             letter
R_PAPERSIZE_USER        a4
R_PDFVIEWER             /usr/bin/xdg-open
R_PLATFORM              x86_64-pc-linux-gnu
R_PRINTCMD              /usr/bin/lpr
R_PROFILE_USER          /mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/sparkr/SparkR/profile/general.R
R_RD4PDF                times,inconsolata,hyper
R_SESSION_INITIALIZED   PID=52:NAME="reticulate"
R_SESSION_TMPDIR        /tmp/Rtmp3l9g78
R_SHARE_DIR             /usr/share/R/share
R_STRIP_SHARED_LIB      strip --strip-unneeded
R_STRIP_STATIC_LIB      strip --strip-debug
R_SYSTEM_ABI            linux,gcc,gxx,gfortran,gfortran
R_TEXI2DVICMD           /usr/bin/texi2dvi
R_UNZIPCMD              /usr/bin/unzip
R_ZIPCMD                /usr/bin/zip
SED                     /bin/sed
SHLVL                   1
SPARK_HOME              /src/FootFall/venv3/lib/python3.9/site-packages/pyspark
SPARK_PUBLIC_DNS        10.5.78.242
SPARK_USER              root
SPARK_YARN_STAGING_DIR
                        hdfs://ip-10-5-73-109.eu-west-1.compute.internal:8020/user/root/.sparkStaging/application_1680508493561_0044
SPARKR_BACKEND_AUTH_SECRET
                        251092ad552b2cd09eebd9e9e8cf58454e422026ab4e4bee618d824d884d1f9f
SPARKR_BACKEND_CONNECTION_TIMEOUT
                        6000
SPARKR_PACKAGE_DIR      /mnt1/yarn/usercache/root/appcache/application_1680508493561_0044/container_1680508493561_0044_01_000001/sparkr
TAR                     /bin/tar
USER                    root
VIRTUAL_ENV             /src/FootFall/venv3
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
                        686657584371.dkr.ecr.eu-west-1.amazonaws.com/tiendeodata-footfall-pro:latest
YARN_CONTAINER_RUNTIME_TYPE
                        docker

And the Spark config loaded by SparkR:

$spark.app.attempt.id
[1] "1"

$spark.app.id
[1] "application_1680508493561_0044"

$spark.app.name
[1] "app-name"

$spark.app.startTime
[1] "1680524505422"

$spark.app.submitTime
[1] "1680524482179"

$spark.blacklist.decommissioning.enabled
[1] "true"

$spark.blacklist.decommissioning.timeout
[1] "1h"

$spark.decommissioning.timeout.threshold
[1] "20"

$spark.default.parallelism
[1] "24"

$spark.driver.defaultJavaOptions
[1] "-XX:OnOutOfMemoryError='kill -9 %p'"

$spark.driver.extraClassPath
[1] "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar"

$spark.driver.extraJavaOptions
[1] "-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -XX:OnOutOfMemoryError='kill -9 %p'"

$spark.driver.extraLibraryPath
[1] "/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native"

$spark.driver.host
[1] "ip-10-5-78-242.eu-west-1.compute.internal"

$spark.driver.memory
[1] "11171M"

$spark.driver.port
[1] "46801"

$spark.dynamicAllocation.enabled
[1] "true"

$spark.emr.default.executor.cores
[1] "4"

$spark.emr.default.executor.instances
[1] "3"

$spark.emr.default.executor.memory
[1] "10356M"

$spark.emr.maximizeResourceAllocation
[1] "true"

$spark.eventLog.dir
[1] "hdfs:///var/log/spark/apps"

$spark.eventLog.enabled
[1] "true"

$spark.executor.cores
[1] "4"

$spark.executor.defaultJavaOptions
[1] "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'"

$spark.executor.extraClassPath
[1] "/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar:/docker/usr/lib/hadoop-lzo/lib/*:/docker/usr/lib/hadoop/hadoop-aws.jar:/docker/usr/share/aws/aws-java-sdk/*:/docker/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/docker/usr/share/aws/emr/security/conf:/docker/usr/share/aws/emr/security/lib/*:/docker/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/docker/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/docker/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/docker/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar"

$spark.executor.extraJavaOptions
[1] "-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'"

$spark.executor.extraLibraryPath
[1] "/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native"

$spark.executor.id
[1] "driver"

$spark.executor.instances
[1] "3"

$spark.executor.memory
[1] "10356M"

$spark.executor.memoryOverheadFactor
[1] "0.1"

$spark.executorEnv._SPARK_LOCAL_IP
[1] "127.0.0.1"

$spark.executorEnv._YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS
[1] "/etc/hosts:/etc/hosts"

$spark.executorEnv._YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER
[1] "true"

$spark.executorEnv.LD_LIBRARY_PATH
[1] "$LD_LIBRARY_PATH:/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/etc/alternatives/jre/lib/server:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native:"

$spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
[1] "686657584371.dkr.ecr.eu-west-1.amazonaws.com/tiendeodata-footfall-pro:latest"

$spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE
[1] "docker"

$spark.files.fetchFailure.unRegisterOutputOnHost
[1] "true"

$spark.hadoop.fs.s3.getObject.initialSocketTimeoutMilliseconds
[1] "2000"

$spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem
[1] "2"

$`spark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem`
[1] "true"

$spark.hadoop.mapreduce.output.fs.optimized.committer.enabled
[1] "true"

$`spark.hadoop.yarn.timeline-service.enabled`
[1] "false"

$spark.history.fs.logDirectory
[1] "hdfs:///var/log/spark/apps"

$spark.history.ui.port
[1] "18080"

$spark.home
[1] "/src/FootFall/venv3/lib/python3.9/site-packages/pyspark"

$spark.master
[1] "yarn"

$spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS
[1] "ip-10-5-73-109.eu-west-1.compute.internal"

$spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES
[1] "http://ip-10-5-73-109.eu-west-1.compute.internal:20888/proxy/application_1680508493561_0044"

$spark.r.sql.derby.temp.dir
[1] "/tmp/Rtmp3l9g78"

$spark.resourceManager.cleanupExpiredHost
[1] "true"

$spark.shuffle.service.enabled
[1] "true"

$spark.sql.catalogImplementation
[1] "hive"

$spark.sql.emr.internal.extensions
[1] "com.amazonaws.emr.spark.EmrSparkSessionExtensions"

$spark.sql.hive.metastore.sharedPrefixes
[1] "com.amazonaws.services.dynamodbv2"

$`spark.sql.parquet.fs.optimized.committer.optimization-enabled`
[1] "true"

$spark.sql.parquet.output.committer.class
[1] "com.amazon.emr.committer.EmrOptimizedSparkSqlParquetOutputCommitter"

$spark.sql.warehouse.dir
[1] "hdfs://ip-10-5-73-109.eu-west-1.compute.internal:8020/user/spark/warehouse"

$spark.stage.attempt.ignoreOnDecommissionFetchFailure
[1] "true"

$spark.submit.deployMode
[1] "cluster"

$spark.submit.pyFiles
[1] ""

$spark.ui.filters
[1] "org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter"

$spark.ui.port
[1] "0"

$spark.yarn.app.container.log.dir
[1] "/var/log/hadoop-yarn/containers/application_1680508493561_0044/container_1680508493561_0044_01_000001"

$spark.yarn.app.id
[1] "application_1680508493561_0044"

$spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS
[1] "$(hostname -f)"

$spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE
[1] "686657584371.dkr.ecr.eu-west-1.amazonaws.com/tiendeodata-footfall-pro:latest"

$spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE
[1] "docker"

$spark.yarn.dist.archives
[1] "file:/usr/lib/spark/R/lib/sparkr.zip#sparkr"

$spark.yarn.dist.files
[1] "file:/etc/hudi/conf.dist/hudi-defaults.conf,file:/home/hadoop/CommandRunner.R"

$spark.yarn.dist.jars
[1] "file:///root/.ivy2/jars/org.apache.sedona_sedona-python-adapter-3.0_2.12-1.3.1-incubating.jar,file:///root/.ivy2/jars/org.datasyslab_geotools-wrapper-1.3.0-27.2.jar,file:///root/.ivy2/jars/org.locationtech.jts_jts-core-1.18.2.jar,file:///root/.ivy2/jars/org.wololo_jts2geojson-0.16.1.jar,file:///root/.ivy2/jars/org.apache.sedona_sedona-core-3.0_2.12-1.3.1-incubating.jar,file:///root/.ivy2/jars/org.apache.sedona_sedona-sql-3.0_2.12-1.3.1-incubating.jar,file:///root/.ivy2/jars/org.apache.sedona_sedona-common-1.3.1-incubating.jar,file:///root/.ivy2/jars/org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar"

$spark.yarn.historyServer.address
[1] "ip-10-5-73-109.eu-west-1.compute.internal:18080"

$spark.yarn.maxAppAttempts
[1] "1"

$spark.yarn.secondary.jars
[1] "org.apache.sedona_sedona-python-adapter-3.0_2.12-1.3.1-incubating.jar,org.datasyslab_geotools-wrapper-1.3.0-27.2.jar,org.locationtech.jts_jts-core-1.18.2.jar,org.wololo_jts2geojson-0.16.1.jar,org.apache.sedona_sedona-core-3.0_2.12-1.3.1-incubating.jar,org.apache.sedona_sedona-sql-3.0_2.12-1.3.1-incubating.jar,org.apache.sedona_sedona-common-1.3.1-incubating.jar,org.scala-lang.modules_scala-collection-compat_2.12-2.5.0.jar"

How can I make sparklyr run on this set-up?