Suddently getting Error: failed to start plugin manager: too many open files

[farzadmf]

Describe the bug
My whole workflow is dependent on steampipe, and now it's failing with the following error:

Error: failed to start plugin manager: too many open files

Steampipe version (steampipe -v)
Steampipe v0.21.2

To reproduce
Not sure what happened TBH, but even doing steampipe plugin list doesn't work.

Expected behavior
steampipe should be working fine without me changing anything (except for maybe upgrading things)

Additional context
I tried removing ~/.steampipe directory, but still getting the error

[binaek]

@farzadmf sorry to see that you have hit this snag.

How are you using steampipe? If you find yourself doing steampipe query again and again, a steampipe service start at the start and a service stop at the end usually helps.

Can you check if there are any stray steampipe processes? I have seen a simple ps -ef | grep steampipe helps.

Does it help if you restart your system?

[farzadmf]

@binaek thank you for your reply.

I tried everything; killing the processes, deleting directories, etc., but nothing worked.

Restarting my computer did solve the issue, but I think you can agree that it's not a solution, rather a [hacky] workaround, given that it very well may be impossible to restart the system while you're actively working on something.

And regarding steampipe service, I think it would add a significant delay to my operations as the start and stop operations both take time, and that would basically render steampipe as "impossible to use because it takes too long to run" and just go with reliable native things (which, in my case at the moment, is aws CLI command)

[kaidaguerre]

Hey @farzadmf sorry you've hit this issue. I totally agree, restarting the computer is not an acceptable long term solution. However if it unblocks you for now that is good.

If the problem recurs, one thing to do is the check if you have any stray steampipe processes by running

ps aux | grep steampipe

I'll leave this issue open for now and do a bit of digging

[judell]

Hey @farzadmf can you please share details on which plugins you're loading, and roughly how many connections for each?

[farzadmf]

Hey @judell , right now, I only have the aws plugin, and my workflow I'd say is "light" (at least I hope it is).

I'm running stp query "<my-query>" and I'd say I have a max of 2-3 calls in the subshell I'm creating in my script (and it would be a single subshell created by me running my script).

I started to face some issues (I had a similar issue way back; I think it can be found in the issues - I searched and found #3297), so after that, I'm doing pkill -f steampipe in my script's entry point (another hacky thing).

And, as I mentioned before, the only solution that worked was restarting my computer (killing processes, deleting directories, etc. nothing worked)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[kaidaguerre]

hi @farzadmf sorry for the delay in getting back to this. Is this error still happening?

[farzadmf]

Hey @kaidaguerre no problem. TBH, it happens time to time, along with other weird errors, some requiring me to delete the ~/.steampipe directory, some requiring me to reboot my system.

When I discovered steampipe, I was "OMG, this is the best thing", and I started putting it everywhere in my scripts (my only use case right now is aws), but then I started getting all these weird errors, and I can't trust steampipe anymore and little by little, I'm' going back to using aws CLI.

Honestly, I think my initial impression was correct and steampipe has GREAT potential, but at least for me, the UX has been VERY dissapointing. Every time I run my scripts, I'm like "I hope I've converted it to aws CLI otherwise it will probably give some random error"

When steampipe does work, it's amazing, but ...

[kaidaguerre]

Hi @farzadmf I'm sorry to hear you continue to have issues. Intermittent issues such as this as particularly hard to track down - I have been unable to reproduce this locally.

What is the file limit per process on your system = ulimit -n

If it does recur, it would be interesting to get the results of running lsof to list the open files. In particular, finding the steampipe process using ps aux | grep steampipe, then passing that pid to lsof: lsof -p xxxx

[farzadmf]

Thank you @kaidaguerre for the reply; I'll keep that in mind for next time.

As for the file limit, I think I've increased to its max: 65535

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.