CodeChecker doesn't fully capture the errors/warnings reported by clang-tidy

[w3ntao]

Describe the bug
For the same project, clang-tidy alone report 4 warnnings but CodeChecker (that use clang-tidy as backend) produces only 1.

CodeChecker version

CodeChecker version:
Base package version | 6.23.1                                  
Package build date   | 2023-12-14T14:38                        
Git commit ID (hash) | 2a8fa6e711a4ff591280a79fe8798dee2507d984
Git tag information  | 6.23.1

clang-tidy version:
LLVM (http://llvm.org/):
  LLVM version 18.0.0git
  Optimized build.

CodeChecker was installed via pip
clang-tidy was installed via apt

To Reproduce

1. clone the code project, and generate compile_commands.json:

$ git clone https://github.com/w3ntao/codechecker-bug.git
$ cd codechecker-bug
$ mkdir build; cd build
$ cmake ..

2. analyze the project with clang-tidy binary alone and report 4 warnings (3 in calculator.cpp and 1 in main.cpp):

$ cd codechecker-bug
$ clang-tidy -p ./build/compile_commands.json *.cpp

/root/codechecker-bug/calculator.cpp:5:3: warning: avoid using 'goto' for flow control [cppcoreguidelines-avoid-goto]
    5 |   goto final;
      |   ^~~~~~~~~~
/root/codechecker-bug/calculator.cpp:10:1: note: label defined here
   10 | final:
      | ^
/root/codechecker-bug/calculator.cpp:7:7: warning: variable 'k' is not initialized [cppcoreguidelines-init-variables]
    7 |   int k;
      |       ^
      |         = 0
/root/codechecker-bug/calculator.cpp:8:27: warning: do not use 'std::endl' with streams; use '\n' instead [performance-avoid-endl]
    8 |   std::cout << "hello" << std::endl;
      |                           ^~~~~~~~~
      |                           '\n'
/root/codechecker-bug/main.cpp:5:7: warning: variable 'unused_var_in_main' is not initialized [cppcoreguidelines-init-variables]
    5 |   int unused_var_in_main;
      |       ^                 
      |                          = 0

3. analyze with CodeChecker and parse the result:

$ cd codechecker-bug
$ CodeChecker analyze ./build/compile_commands.json --enable sensitive --output ./reports --analyzer-config 'clang-tidy:take-config-from-directory=true'
$ CodeChecker parse --export html --output ./reports_html ./reports

Now open codechecker-bug/reports_html/index.html there is only 1 warning in main.cpp

Expected behaviour
CodeChecker should capture and present all 4 warnings reported by clang-tidy.

Desktop
OS: Debian 12.5
Browser: Firefox/Chrome

Additional context
file .clang-tidy is important to reproducing this bug.

[whisperity]

Could you please try this with an older version (e.g., LLVM 16.0 or so?) of clang-tidy? It could be that the changed diagnostic format around the fixits (#4063) are causing problems in the HTML view. What happens if you use CodeChecker parse directly, without the HTML conversion?

[w3ntao]

With commands

$ CodeChecker analyze ./build/compile_commands.json --enable sensitive --output ./reports --analyzer-config 'clang-tidy:take-config-from-directory=true'
$ CodeChecker parse ./reports

I got

----==== File Statistics ====----
-------------------------------------
File name         | Number of reports
-------------------------------------
main.cpp          |                 2
functional_hash.h |                 1
-------------------------------------
----=================----

Still, 2 errors detected on main.cpp, all of 3 warnings from calculator.cpp missed.

Does this proves that, this is not a HTML rendering bug?

[whisperity]

Yes, the raw output files do not contain the warnings. So this must be an issue with how the analysis is executed. And the problem is likely the conflict between what the .clang-tidy file says the checker list should be, vs. the --enable sensitive passed on the command-line.

[w3ntao]

So regarding this specific bug, is there a workaround solution or do I have to wait for a fixing patch?

[whisperity]

We have to investigate what is causing this exactly, why is .clang-tidy files synergising badly with other CLI arguments. Unfortunately, we are in a release cycle as of right now, so it is likely that this will only be fixed in the following release, not the current one.

Meanwhile, what you can do, is not use the .clang-tidy file and instead specify the enabled checkers through CodeChecker directly. --enable takes multiple kinds of arguments: individual checkers, or checker "groups" (such as cppcoreguidelines), or the profiles like sensitive. So in your case, "Checks": "cppcoreguidelines-*,performance-*,clang-diagnostic-*" would likely correspond to --enable cppcoreguidelines --enable performance --enable clang-diagnostic.