systemd-oomd on Fedora, large data - bazooka kills

[shoffmeister]

I love ripgrep. It's fast. It's intuitive. It works.

I love Fedora Linux (and KDE) and the konsole terminal application. It's powerful. It's intuitive. It works.

The combination of both, on large amounts of data - ouch!

The challenge is that by running the KDE desktop environment on Fedora Linux, and then starting a konsole terminal session, and running ripgrep in there, all memory consumption is attributed to that konsole process, which runs in a specific systemd slice.

Running rg on a wide and deep set of git repos will consume a lot of memory, will trigger memory pressure on systemd-oomd, will make that ... kill the complete konsole process (all of its tabs). Depending on how many konsole tabs were open, how many applications are still associated with those tabs, that's ... causing unhappiness.

I have no idea what to do.

systemd-oomd is "only" trying to protect my user session, and the system, so that it can stay responsive (Fedora Linux might have configuration which is by far too sensitive and too eager to kill). rg tries to work as fast and hard as possible. Both together lead into a bazooka kill and to unhappiness.

I could turn off systemd-oomd (and have actually done so), but then this is still a somewhat useful tool.

Any suggestions? Is there anything that ripgrep could do?

(FWIW, systemd-run is a thing, I have a wrapper script, but, being a lazy human, I don't want to think and remember any of that - Don't Make Me Think!).

[BurntSushi]

I don't know, sorry. This doesn't really seem like a ripgrep specific issue, but a problem with how systemd and konsole work? Otherwise it's not really clear what ripgrep can do. Presumably you're asking for a way to say, "give ripgrep this much memory and don't let it use more than that." But that sort feature requires deep and fundamental changes in the architecture of the program. And in particular, you have to answer the question, "what does ripgrep do when one of its components wants more memory but none is available?" This is a thorny issue and it's in part why the OOM killer exists in the first place. (Not directly, of course, the reason for the OOM killer is virtual memory and how malloc "always succeeds" on Linux by default.)

The only real knob that is useful to you here is probably the -j/--threads flag. You can use -j1 to force ripgrep into single threaded mode for example. And it is likely, although not guaranteed, that this will lead to less peak memory usage.

Otherwise I think you'll have to do something else. Such as a wrapper script or only searching a subset of your data.

[shoffmeister]

Many thanks for your response!

Well, yeah. :) Sigh. You are perfectly right.

Just after I posted, I thought of adding an alias for rg that simply remaps into that custom systemd-run wrapper script that I have, and that I apparently keep forgetting about. That might be the most "Don't make me think!" approach, as it will retain shell completion capabilities.

Long-term, I'd love to be running some good git repo tool (like Sourcegraph), but for now it's plain regular expressions, locally.