How to make rg _always_ look at all files under Git control

[nkiesel]

I generally love how rg interacts with Git, but I recently ran into one issue where it does not behave like git ls-files | grep -P: rg does not look at "hidden" files under version control. So if I have a .npmrc file under Git version control and a .other file which is not under Git version control, is there a flag to rg which searches through .npmrc but not through .other?

[BurntSushi]

ripgrep generally has three different types of automatic filtering, as documented in the guide: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#automatic-filtering

The first is filtering based on .gitignore, .ignore and .rgignore. The second is hidden files/directories. The third is binary files.

Generally, they are not connected with one another. So when you pass --hidden to ripgrep to squash the second type of automatic filtering, it does so for all hidden files.

With that said, you can make ripgrep search .npmrc and not .other by explicitly whitelisting it. For example, echo '!.npmrc' > .ignore will tell ripgrep to search that file even though it's hidden.

For example:

ripgrep/.ignore

Line 1 in 44fb9fc

!/.github/

Also note that this isn't quite the same as the question you asked. Or at least, the question you asked is a little broader. git ls-files reports all files checked into source control, where as ripgrep merely respects your .gitignore. git is happy to check in files that are in your .gitignore. So if your .gitignore is out of sync with what's checked in, ripgrep might not match what you expect. The fix for this is to fix your .gitignore files.

[nkiesel]

So seems that the answer is: ripgrep does honor .gitignore, but does not understand / uses Git otherwise. I guess I do have to go back to git ls-files, given the amount of "hidden" files I have under version control. Or add a post-commit Git hook which automates white-listing all "hidden" files under version control. Something as simple as git ls-files --format='!%(path)' -- '.*' > .rgignore might do the trick

[BurntSushi]

Yeah if you have a lot of hidden files and the set is changing frequently, it would be easy to forget to update .rgignore.

[nkiesel]

I just tried this, but does not seem to work. I have a directory containing a Dockerfiile and a .circleci/config.yml and a rg -. node16 finds matches in both files, while a rg node16 will not (as expected). I then used echo '!.circleci/config.yml' > .rgignore, but rg node16 still does not show the matches in .circleci/config.yml. What am I doing wrong?

[BurntSushi]

Have you looked at the output of --debug? Otherwise, you need to provide instructions to reproduce. For example, here's the setup:

$ git init
Initialized empty Git repository in /tmp/rg2436/.git/
$ echo node16 > Dockerfile
$ mkdir .circleci
$ echo node16 > .circleci/config.yml
$ git add Dockerfile .circleci/config.yml
$ git commit -m "initial commit"
[master (root-commit) 2dc85bc] initial commit
 2 files changed, 2 insertions(+)
 create mode 100644 .circleci/config.yml
 create mode 100644 Dockerfile
$ rg node16
Dockerfile
1:node16
$ rg -. node16
Dockerfile
1:node16

.circleci/config.yml
1:node16

Now we try to create a .rgignore:

$ echo '!.circleci/config.yml' > .rgignore
$ rg node16
Dockerfile
1:node16

Doesn't work. So as ripgrep's guide suggests, run with --debug:

$ rg node16 --debug
DEBUG|rg::config|crates/core/config.rs:43: /home/andrew/.ripgreprc: arguments loaded from config file: ["--max-columns-preview", "--colors=match:bg:0xff,0x7f,0x00", "--colors=match:fg:white", "--colors=line:none", "--colors=line:fg:magenta", "--colors=path:fg:green", "--type-add=got:*_test.go"]
DEBUG|rg::args|crates/core/args.rs:525: final argv: ["rg", "--max-columns-preview", "--colors=match:bg:0xff,0x7f,0x00", "--colors=match:fg:white", "--colors=line:none", "--colors=line:fg:magenta", "--colors=path:fg:green", "--type-add=got:*_test.go", "node16", "--debug"]
DEBUG|grep_regex::literal|crates/regex/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(node16)], limit_size: 250, limit_class: 10 }
DEBUG|rg::args|crates/core/args.rs:549: running with 8 threads for parallelism
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 1 literals, 0 basenames, 0 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1740: ignoring ./.rgignore: Ignore(IgnoreMatch(Hidden))
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1740: ignoring ./.circleci: Ignore(IgnoreMatch(Hidden))
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1740: ignoring ./.git: Ignore(IgnoreMatch(Hidden))
Dockerfile
1:node16
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
DEBUG|globset|crates/globset/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes

Notice this line:

DEBUG|ignore::walk|crates/ignore/src/walk.rs:1740: ignoring ./.circleci: Ignore(IgnoreMatch(Hidden))

So whatever you put in your .rgignore doesn't work because it doesn't match .circleci. Which it doesn't. That's correct. You need to whitelist the directory itself, otherwise ripgrep doesn't descend into it. (Consider the alternative. ripgrep would have to descend into every directory, even if it was supposed to be ignored.)

So let's just try whitelisting the top-level .circleci directory:

$ echo '!/.circleci/' > .rgignore
$ rg node16
Dockerfile
1:node16

.circleci/config.yml
1:node16

If you want to ignore other stuff inside of .circleci, then you'll need to add more rules.

[nkiesel]

yeah, now ended up using git ls-files --format='!/%(path)' -- '.*' | sed -e 's,\(/.*/\).*,\1,' > .rgignore. This does not handle your valid "If you want to ignore other stuff inside of .circleci, then you'll need to add more rules." comment, but luckily that does not apply to my repos.

[nkiesel]

BTW: thanks so much for your answers!