Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

could support elasticsearch read time out configuration? #72

Open
wuyupengwoaini opened this issue Aug 9, 2019 · 3 comments · May be fixed by #73
Open

could support elasticsearch read time out configuration? #72

wuyupengwoaini opened this issue Aug 9, 2019 · 3 comments · May be fixed by #73

Comments

@wuyupengwoaini
Copy link

wuyupengwoaini commented Aug 9, 2019 •
edited

Recently,I got an error when run this spark-dependencies job in dokcer .
19/08/02 05:50:26 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2019-08-01T00:00Z, reading from jaeger-span-2019-08-01 index, result storing to jaeger-dependencies-2019-08-01 [Stage 0:> (1 + 8) / 3040]19/08/02 05:51:31 ERROR NetworkClient: Node [[ip1:port] failed (Read timed out); selected next node [[ip1:port] 19/08/02 05:51:31 ERROR NetworkClient: Node [ip1:port] failed (Read timed out); selected next node [ip2:port] 19/08/02 05:51:31 ERROR NetworkClient: Node [ip2:port] failed (Read timed out); selected next node [ip3:port]
So,I suppose could support timeout config.
`
The following is the specific configuration

es.http.timeout (default 1m)
Timeout for HTTP/REST connections to Elasticsearch.`

https://www.elastic.co/guide/en/elasticsearch/hadoop/6.6/configuration.html
@pavolloffay
Copy link
Member

+1 would you like to submit a PR for it?

@wuyupengwoaini
Copy link
Author

Not only the es.http.timeout configuration and also other the cofiguration about performance such as es.scroll.size (default 50).So,I suppose that we should provide the ability for users to configure performance parameters,but I can't find a good way to do this
@pavolloffay Do you have a good idea?

@wuyupengwoaini wuyupengwoaini linked a pull request Aug 12, 2019 that will close this issue
Open
@pavolloffay
Copy link
Member

The parameters are configurable via ENV vars, just follow the approach what we use for other parameters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support spark read performance configuration wuyupengwoaini/spark-dependencies
2 participants
@pavolloffay @wuyupengwoaini