Unclear buildbot failure email

[asb]

Reported by @jayfoad for an email from the rv64gc-qemu-user-single-stage bot (administered by me). The relevant build failure is this one.

Issues with the email:

• It doesn't include the reason for test failure that is present in the web ui (stdout vs stderr issue?). Namely these lines

command timed out: 1200 seconds without output running [b'ninja', b'check-all'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=11612.025770

• The summary line "BUILD FAILED: failed 72275 expected passes 195 expected failures 1311 unsupported tests (failure)" is confusing/misleading.

As this bot doesn't have anything exotic in terms of its buildbot config (ref) I'd assume this is a generic buildbot/llvm-zorg issue, but of course would love to hear if there's anything I'm missing on my end.

As I noted in a separate email thread, the mooted "gatekeeper" bot support would be a huge help in avoiding builder runs that identify an issue not specific to that builder that's been fixed in the mean time. Though that wouldn't directly fix the fact the generated email was confusing in this case.

[jayfoad]

Thanks for following up on this!

[andreil99]

Looks like llvm-zorg does not parse logs well in the case of a build cancelled because of timeout.

The relevant chunk of log:

PASS: lit :: usage.py (73765 of 73782)
PASS: lit :: use-llvm-tool.py (73766 of 73782)
PASS: lit :: test-output-resultdb.py (73767 of 73782)
PASS: lit :: lit-opts.py (73768 of 73782)
PASS: lit :: shtest-env.py (73769 of 73782)
PASS: lit :: shtest-if-else.py (73770 of 73782)
PASS: lit :: shtest-run-at-line.py (73771 of 73782)
PASS: lit :: xfail-cl.py (73772 of 73782)
PASS: lit :: shtest-recursive-substitution.py (73773 of 73782)
PASS: lit :: googletest-timeout.py (73774 of 73782)
PASS: lit :: allow-retries.py (73775 of 73782)
PASS: lit :: shtest-timeout.py (73776 of 73782)
PASS: lit :: shtest-not.py (73777 of 73782)
PASS: lit :: discovery.py (73778 of 73782)
PASS: lit :: selecting.py (73779 of 73782)
PASS: lit :: shtest-define.py (73780 of 73782)
PASS: lit :: shtest-shell.py (73781 of 73782)
command timed out: 1200 seconds without output running [b'ninja', b'check-all'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=11612.025770

test-build-unified-tree-check-all step summary is 72275 expected passes 195 expected failures 1311 unsupported tests (failure), but should mention step interruption because of timeout.

The same issue is with the email notifier. That has a wrong summary, as well as the quoted chunk of log is not actually relevant to the issue.

[andreil99]

For the sake of completeness, here is the quoted log from email notification:

BUILD FAILED: failed 72275 expected passes 195 expected failures 1311 unsupported tests (failure)

Step 6 (test-build-unified-tree-check-all) failure: 72275 expected passes 195 expected failures 1311 unsupported tests (failure)
...
PASS: Clangd Unit Tests :: ./ClangdTests/26/38 (4244 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/performance/move-const-arg-const-ref.cpp (4245 of 73782)
PASS: Clang :: CodeGen/RISCV/rvv-intrinsics-autogenerated/policy/non-overloaded/vlseg8e8.c (4246 of 73782)
PASS: LLVM :: CodeGen/AMDGPU/reduction.ll (4247 of 73782)
PASS: Clang :: CodeGenCXX/inheriting-constructor.cpp (4248 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/abseil/no-internal-dependencies.cpp (4249 of 73782)
PASS: Clang :: PCH/cxx11-exception-spec.cpp (4250 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/swapped-arguments.cpp (4251 of 73782)
PASS: Clang :: OpenMP/taskloop_simd_loop_messages.cpp (4252 of 73782)
PASS: LLVM :: CodeGen/X86/combine-add-usat.ll (4253 of 73782)
XFAIL: Clang :: Analysis/reinterpret-cast-pointer-to-member.cpp (4254 of 73782)
PASS: LLVM :: CodeGen/AMDGPU/flat-for-global-subtarget-feature.ll (4255 of 73782)
PASS: LLVM :: Analysis/CostModel/X86/arith-overflow.ll (4256 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/readability/redundant-string-cstr-msvc.cpp (4257 of 73782)
PASS: Clang :: OpenMP/target_parallel_for_loop_messages.cpp (4258 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/modernize/use-auto-new.cpp (4259 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/argument-comment-gmock.cpp (4260 of 73782)
PASS: LLVM :: CodeGen/X86/vector-tzcnt-512.ll (4261 of 73782)
PASS: Clang :: OpenMP/target_teams_distribute_parallel_for_simd_simdlen_messages.cpp (4262 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/not-null-terminated-result-in-initialization-strlen.c (4263 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/not-null-terminated-result-memcpy-safe-other.c (4264 of 73782)
PASS: Clang :: OpenMP/target_teams_distribute_loop_messages.cpp (4265 of 73782)
PASS: Clang :: CodeGen/RISCV/rvv-intrinsics-autogenerated/policy/non-overloaded/vloxseg3ei8.c (4266 of 73782)
PASS: Clang :: CodeGen/aarch64-v8.5a-neon-frint3264-intrinsic.c (4267 of 73782)
PASS: Clang :: utils/update_cc_test_checks/resolve-tmp-conflict.test (4268 of 73782)
PASS: Clang :: SemaCXX/cxx2a-initializer-aggregates.cpp (4269 of 73782)
PASS: lld :: COFF/implib-name.test (4270 of 73782)
PASS: Clang :: SemaOpenCL/[extension-version.cl](http://extension-version.cl/) (4271 of 73782)
PASS: Clang :: CodeGen/RISCV/rvv-intrinsics-autogenerated/policy/overloaded/vfmadd.c (4272 of 73782)
PASS: lld :: MinGW/lib.test (4273 of 73782)
PASS: Clang :: CodeGen/aarch64-v8.6a-neon-intrinsics.c (4274 of 73782)
PASS: lld :: MachO/lc-linker-option-order.ll (4275 of 73782)
PASS: LLVM :: CodeGen/AMDGPU/si-sgpr-spill.ll (4276 of 73782)
PASS: Clang :: Preprocessor/ext-pp-directive.c (4277 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/suspicious-memset-usage.cpp (4278 of 73782)
PASS: Clang :: Preprocessor/riscv-cmodel.c (4279 of 73782)
PASS: Clang :: CodeGen/RISCV/rvv-intrinsics-autogenerated/non-policy/overloaded/vsoxseg6ei8.c (4280 of 73782)
PASS: Clangd Unit Tests :: ./ClangdTests/15/38 (4281 of 73782)
PASS: Clang :: OpenMP/target_teams_distribute_simd_is_device_ptr_ast_print.cpp (4282 of 73782)
PASS: Clang :: CodeGen/catch-pointer-overflow.c (4283 of 73782)
PASS: lld :: ELF/linkerscript/symbols.s (4284 of 73782)
PASS: Clang Tools :: clang-tidy/checkers/bugprone/misplaced-pointer-arithmetic-in-alloc.cpp (4285 of 73782)
PASS: Clang :: CodeGen/RISCV/rvv-intrinsics-autogenerated/policy/non-overloaded/vrgather.c (4286 of 73782)
PASS: LLVM :: Analysis/CostModel/X86/mul-latency.ll (4287 of 73782)
PASS: LLVM :: CodeGen/PowerPC/p8-scalar_vector_conversions.ll (4288 of 73782)
PASS: lld :: ELF/why-extract.s (4289 of 73782)
PASS: LLVM :: CodeGen/AMDGPU/sra.ll (4290 of 73782)
PASS: LLVM :: CodeGen/X86/win32_sret.ll (4291 of 73782)
PASS: Clang :: Driver/aarch64-v91a.c (4292 of 73782)