Skip to content
This repository has been archived by the owner on Jun 8, 2023. It is now read-only.

Lambda: Failure opening selector #21

Open
davispalomino opened this issue Sep 22, 2020 · 2 comments · May be fixed by #22
Open

Lambda: Failure opening selector #21

davispalomino opened this issue Sep 22, 2020 · 2 comments · May be fixed by #22

Comments

@davispalomino
Copy link

Dear, there is an error in the execution of the lambda, when you see it, it may be an overload of the function for reading a large number of registers.

Error:

org.apache.http.nio.reactor.IOReactorException: Failure opening selector
java.lang.IllegalStateException: org.apache.http.nio.reactor.IOReactorException: Failure opening selector

Metric lambda:
https://ibb.co/tZpZmYt

Thank you for your reply.

Regards
@davispalomino davispalomino changed the title failure opening selector elasticsearch Lambda: Failure opening selecto Sep 22, 2020
@davispalomino davispalomino changed the title Lambda: Failure opening selecto Lambda: Failure opening selector Sep 22, 2020
@lobeck
Copy link

lobeck commented Oct 7, 2020 •
edited

We were just hitting the same issue:

Oct 07, 2020 12:41:25 PM org.apache.http.impl.nio.client.InternalHttpAsyncClient runorg.apache.http.impl.nio.client.InternalHttpAsyncClient run 
SEVERE: I/O reactor terminated abnormally 
org.apache.http.nio.reactor.IOReactorException: Failure opening selector
at org.apache.http.impl.nio.reactor.AbstractIOReactor.<init>(AbstractIOReactor.java:103)
at org.apache.http.impl.nio.reactor.BaseIOReactor.<init>(BaseIOReactor.java:85) 
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:321)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Too many open files 
at java.base/sun.nio.ch.EPoll.create(Native Method)
at java.base/sun.nio.ch.EPollSelectorImpl.<init>(Unknown Source)
at java.base/sun.nio.ch.EPollSelectorProvider.openSelector(Unknown Source)
at java.base/java.nio.channels.Selector.open(Unknown Source)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.<init>(AbstractIOReactor.java:101)
... 5 more
Finished processing; flushing any remaining logs...
Elasticsearch processor shut down

It seems, that it "crashed" while the cluster was under heavy pressure, but was not able to recover from this. It only started working again, after I've changed the lambda and it implicitly restarted.

The lambda was running fine for months now, so i think "too many open files" is a symptom/side effect, but not a actual issue. I don't think a lambda can run long enough to accumulate this many files.

lobeck added a commit to Sixt/cloudflare-elastic that referenced this issue Oct 8, 2020
@lobeck
reuse elasticsearch client and properly close it on error
bcf891f 
fixes cloudflarearchive#21
@lobeck lobeck linked a pull request Oct 8, 2020 that will close this issue
Open
lobeck added a commit to Sixt/cloudflare-elastic that referenced this issue Oct 8, 2020
@lobeck
reuse elasticsearch client and properly close it on error
c1dfb88 
fixes cloudflarearchive#21
@netanel246
Copy link

Hi,
I applied your solution in PR #22, and I still see these errors.
I tried to add more workers, but it doesn't help.
Do you know what else it can be?
Thanks.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

reuse elasticsearch client and properly close it on error Sixt/cloudflare-elastic
3 participants
@lobeck @davispalomino @netanel246