Best way to get a total number of all matches from all files

[Andrew15-5]

It looks like ripgrep can only show total number of matches per file or via the stats but with some post-processing.

Can I do better than this?

rg -q --stats search | head -n 2 | awk '{print$1}'
rg -q --stats search | head -n 2 | cut -d\  -f1

The other (less performant) options are:

rg -o search | wc -l
rg -q --stats --json search | jq '.data.stats.matches'

[BurntSushi]

Nope. You could do rg -o search | wc -l, but your method is almost certainly faster, and perhaps substantially so in some cases.

In general ripgrep doesn't really have any aggregate mode for an entire search. It only has aggregate mode for individual files.

[Andrew15-5]

Oh, it's nice that there is a much shorter way to get the same output.

In general ripgrep doesn't really have any aggregate mode for an entire search. It only has aggregate mode for individual files.

Is this something that won't change? I would like to hear why not support some additional options. And this is not a niche use case, I would say a big use case.

[BurntSushi]

For ripgrep, I am not a "why not" kind of maintainer. I am a "why yes" kind of maintainer. That means that the burden of a feature falls on justifying its existence rather than justifying its non-existence. This is in part a philosophical position, but it's also a pragmatic position. The "why not" approach is difficult to scale.

I'll indulge anyway:

1. I actually don't think this is a big use case or even a common one.
2. There are other solutions that work if you need a total count.
3. ripgrep already has a ton of flags, and I like to be careful about adding more.

[Andrew15-5]

I actually don't think this is a big use case or even a common one.

Then I'm expecting too much from ripgrep, I guess.

There are other solutions that work if you need a total count.

You mean like the ones I, you and @lespea showed?

ripgrep already has a ton of flags, and I like to be careful about adding more.

Yeah, about 72 flags is not small. I also don't know how to call this option if --count-matches and --count are already used. Only the --total-matches or something come to mind.

[BurntSushi]

You mean like the ones I, you and @lespea showed?

Yes.

Yeah, about 72 flags is not small.

If you count logically distinct flags (i.e., -o and --only-matching are treated as one flag), then ripgrep has 104 flags:

$ rg -h | rg -e '^\s+-' -c
104

[Andrew15-5]

Oh, so there are a few new flags in the new versions... (Debian packages suck)

[lespea]

You could use the json output and jq; it seems less likely to break if the stats output ever changes:

rg -q --stats --json search | jq '.data.stats.matches'

[Andrew15-5]

The JSON parsing is pretty straightforward AFAIK and the "amount" of text in plain/JSON version is kinda similar, so parsing one small thing or... grabbing the first line and then the first field... Yeah, I think the POSIX approach will be a bit faster.

[BurntSushi]

To clarify here, I didn't say JSON parsing would be faster. I said it shouldn't be a concern. In other words, if there's a perf difference, it's probably not one that a human would notice. For example, on my checkout of Chromium:

$ hyperfine -w5 \
    'rg -q --stats --json Openbox | jq .data.stats.matches' \
    'rg -q --stats Openbox | head -n2 | tail -n1 | awk "{print$1}"' \
    'rg -q --stats Openbox | head -n2 | tail -n1 | cut -d\  -f1'
Benchmark 1: rg -q --stats --json Openbox | jq .data.stats.matches
  Time (mean ± σ):     278.1 ms ±   2.9 ms    [User: 1203.1 ms, System: 1899.4 ms]
  Range (min … max):   273.5 ms … 283.1 ms    10 runs

Benchmark 2: rg -q --stats Openbox | head -n2 | tail -n1 | awk "{print$1}"
  Time (mean ± σ):     278.7 ms ±   3.9 ms    [User: 1210.0 ms, System: 1912.8 ms]
  Range (min … max):   274.8 ms … 286.3 ms    10 runs

Benchmark 3: rg -q --stats Openbox | head -n2 | tail -n1 | cut -d\  -f1
  Time (mean ± σ):     279.7 ms ±   4.0 ms    [User: 1222.6 ms, System: 1903.0 ms]
  Range (min … max):   273.9 ms … 286.9 ms    10 runs

Summary
  rg -q --stats --json Openbox | jq .data.stats.matches ran
    1.00 ± 0.02 times faster than rg -q --stats Openbox | head -n2 | tail -n1 | awk "{print$1}"
    1.01 ± 0.02 times faster than rg -q --stats Openbox | head -n2 | tail -n1 | cut -d\  -f1

There's no real difference. We can try again on a smaller repository where some kind of overhead might be noticeable:

$ hyperfine -w5 \
    'rg -q --stats --json regex | jq .data.stats.matches' \
    'rg -q --stats regex | head -n2 | tail -n1 | awk "{print$1}"' \
    'rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1'
Benchmark 1: rg -q --stats --json regex | jq .data.stats.matches
  Time (mean ± σ):       4.1 ms ±   0.5 ms    [User: 2.8 ms, System: 2.2 ms]
  Range (min … max):     2.8 ms …   5.5 ms    500 runs

Benchmark 2: rg -q --stats regex | head -n2 | tail -n1 | awk "{print$1}"
  Time (mean ± σ):       3.9 ms ±   0.5 ms    [User: 4.1 ms, System: 2.7 ms]
  Range (min … max):     2.7 ms …   5.8 ms    521 runs

Benchmark 3: rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1
  Time (mean ± σ):       4.0 ms ±   0.5 ms    [User: 2.9 ms, System: 2.4 ms]
  Range (min … max):     2.8 ms …   6.0 ms    553 runs

Summary
  rg -q --stats regex | head -n2 | tail -n1 | awk "{print$1}" ran
    1.03 ± 0.18 times faster than rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1
    1.05 ± 0.18 times faster than rg -q --stats --json regex | jq .data.stats.matches

So there is maaaaaaaybe a difference here where the jq variant is slower, but:

1. It's a difference of tenths of a single millisecond. You won't notice it.
2. The times are so quick here that the benchmark results are likely suscpetible to noise.

[Andrew15-5]

Oh man, I didn't see the "n't" in the "I wouldn't expect perf to be a concern here".

'rg -q --stats regex | head -n2 | tail -n1 | awk "{print$1}"'

FYI, I'm pretty sure this is a bad way to quote the thing. Because if the whole command is run in a shell, then the "{print$1}" would actually be just {print} instead of {print$1}. So the correct way to quote here would be:

sh -c 'rg -q --stats regex | head -n 2 | awk "{print\$1}"'
sh -c 'rg -q --stats regex | head -n 2 | awk '\''{print$1}'\' # This is the safest way.

[BurntSushi]

Aye:

$ hyperfine -w5 \
    'rg -q --stats --json regex | jq .data.stats.matches' \
    'rg -q --stats regex | head -n2 | tail -n1 | awk '\''{print$1}'\' \
    'rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1'
Benchmark 1: rg -q --stats --json regex | jq .data.stats.matches
  Time (mean ± σ):       4.0 ms ±   0.5 ms    [User: 3.0 ms, System: 2.1 ms]
  Range (min … max):     2.8 ms …   5.4 ms    500 runs

Benchmark 2: rg -q --stats regex | head -n2 | tail -n1 | awk '{print$1}'
  Time (mean ± σ):       3.9 ms ±   0.5 ms    [User: 4.3 ms, System: 2.7 ms]
  Range (min … max):     2.7 ms …   5.3 ms    503 runs

Benchmark 3: rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1
  Time (mean ± σ):       4.0 ms ±   0.5 ms    [User: 3.1 ms, System: 2.2 ms]
  Range (min … max):     2.7 ms …   5.8 ms    492 runs

Summary
  rg -q --stats regex | head -n2 | tail -n1 | awk '{print$1}' ran
    1.01 ± 0.17 times faster than rg -q --stats regex | head -n2 | tail -n1 | cut -d\  -f1
    1.02 ± 0.17 times faster than rg -q --stats --json regex | jq .data.stats.matches

[lespea]

You could also switch to gojq but that's even less likely to be easily available.

Benchmark 1: rg -q --stats --json test | jq '.data.stats.matches'
  Time (mean ± σ):      39.1 ms ±   4.1 ms    [User: 36.7 ms, System: 49.1 ms]
  Range (min … max):    33.8 ms …  58.7 ms    60 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 2: rg -q --stats --json test | gojq '.data.stats.matches'
  Time (mean ± σ):      35.9 ms ±   3.0 ms    [User: 12.0 ms, System: 49.2 ms]
  Range (min … max):    29.6 ms …  45.5 ms    60 runs
 
Benchmark 3: rg -q --stats test | head -n2 | tail -n1 | cut -d\  -f1
  Time (mean ± σ):      35.9 ms ±   2.8 ms    [User: 11.8 ms, System: 51.3 ms]
  Range (min … max):    29.7 ms …  45.5 ms    63 runs
 
Summary
  rg -q --stats test | head -n2 | tail -n1 | cut -d\  -f1 ran
    1.00 ± 0.11 times faster than rg -q --stats --json test | gojq '.data.stats.matches'
    1.09 ± 0.14 times faster than rg -q --stats --json test | jq '.data.stats.matches'