Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

对spark-submit上传jar的疑问 #58

Open
wloverine opened this issue Jan 21, 2022 · 0 comments
Open

对spark-submit上传jar的疑问 #58

wloverine opened this issue Jan 21, 2022 · 0 comments

Comments

@wloverine
Copy link

在Spark部署模式与作业提交.md一文中提到的:

需要注意的是:在集群环境下,application-jar 必须能被集群中所有节点都能访问,可以是 HDFS 上的路径;也可以是本地文件系统路> 径,如果是本地文件系统路径,则要求集群中每一个机器节点上的相同路径都存在该 Jar 包。

如果application-jar指定的是本地文件系统路径,那么spark会不会自动分发jar到每一台机器上。我在跑spark任务的时候,都是把jar上传到一台机器上就可以跑了,所以看到这里不免疑问。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant