definitions mixup across distinct files

[vladak]

In a project which is part of internal deployment running 1.12.12, some users noticed a strange thing: for a C file (fem.c), the definitions in the file (as reported by the Navigate pane - clicking on these definitions lead to various lines in the file; this is also visible when searching) clearly come from a different file. The other file that should have these definitions (devfs_vnops.c) has no definitions listed in the Navigate pane. It seems as if the ctags definitions got mixed up and broken during indexing. The initial indexing was done using OpenGrok 1.12.x and Universal Ctags 6.0.0(20c45ce9).

It would be worthwhile to add a check for this at the end of indexing (i.e. for each document in the index take its definitions and see if they are actually present in the corresponding file) and then run heavily parallelized indexing for the project multiple times to see if the issue can be reproduced.

[vladak]

Ran 100 iterations of reindex from scratch followed by the definitions check introduced in PR #4402 using the following script:

#!/bin/bash

set -e

ROOT=/var/tmp/indexCheck

for arg in `seq 100`; do

    echo $arg

    /usr/bin/nice -n 30 \
       /jdk/openjdk11/bin/java \
       -Djava.util.logging.config.file=$ROOT/logging.properties \
       -XX:-UseGCOverheadLimit -Xmx16g -server \
       -jar $ROOT/opengrok/lib/opengrok.jar \
       -P \
       -s /test/src \
       -d /test/data \
       -G -m '256' --leadingWildCards 'on' \
       -c /usr/local/bin/ctags -o /opengrok/etc/ctags.config \
       -W $ROOT/config.xml \
       --threads 32

    /jdk/openjdk11/bin/java \
       -jar $ROOT/opengrok/lib/opengrok.jar \
       -R $ROOT/config.xml \
       --checkIndex definitions

    rm -rf /test/data/*
done

however all the checks were successful. I suspect this might happen when one of the ctags processes crashes or hangs.