Release 4.8.0 - RC 1 - Footprint Metrics - ALL-EXCEPT-ACTIVE-RESPONSE (2.5d)

[wazuhci]

Footprint metrics information

Main release stage issue #	#23246
Main footprint metrics issue #	#23254
Version	4.8.0
Release stage #	RC 1
Tag	https://github.com/wazuh/wazuh/tree/v4.8.0-rc1

Stress test documentation

Packages used

• Repository: packages-dev.wazuh.com

• Package path: pre-release

• Package revision: 1

• Jenkins build: https://ci.wazuh.info/job/Test_stress/5118/

---

Manager

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5118_manager_2024-05-06.zip

• CSV

  monitor-manager-Test_stress_B5118_manager-pre-release.csv
  Test_stress_B5118_manager_analysisd_events.csv
  Test_stress_B5118_manager_analysisd_state.csv
  Test_stress_B5118_manager_remoted_state.csv

Centos agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5118_centos_2024-05-06.zip

• CSV

  monitor-agent-Test_stress_B5118_centos-pre-release.csv
  Test_stress_B5118_centos_agentd_state.csv

Ubuntu agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5118_ubuntu_2024-05-06.zip

• CSV

  monitor-agent-Test_stress_B5118_ubuntu-pre-release.csv
  Test_stress_B5118_ubuntu_agentd_state.csv

Windows agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5118_windows_2024-05-06.zip

• CSV

  monitor-winagent-Test_stress_B5118_windows-pre-release.csv
  Test_stress_B5118_windows_agentd_state.csv

macOS agent

• Plots
• Logs and configuration
• CSV

Solaris agent

• Plots
• Logs and configuration
• CSV

Conclusion 🔴

Graphs 🔴

New issues

• Increase of memory usage by wazuh-modulesd #23304

Logs 🔴

New issue

• Wazuh-db errors logs in Footprint tests #23301

Known issues

• https://github.com/wazuh/wazuh-jenkins/issues/5287
• Real time Windows callback process error in Syscheck stress test #12074
• Errors when reading files in footprint tests  #18995
• https://github.com/wazuh/wazuh-jenkins/issues/4867
• https://github.com/wazuh/wazuh-jenkins/issues/4469
• Internal error in modulesd:oscap in stress tests logs #9311
• Vulnerability Detector module is enabled at the start of the stress test #22565
• https://github.com/wazuh/wazuh-jenkins/issues/4481
• https://github.com/wazuh/wazuh-jenkins/issues/6368

[Rebits]

Analysis report

Plots compared to #23174

Graphs 🔴

Centos

• https://github.com/wazuh/wazuh-jenkins/issues/5939

Ubuntu

• https://github.com/wazuh/wazuh-jenkins/issues/5939

Windows

• No abnormal behaviour detected

Manager

• VM/RSS/PSS/USS increased: Reported in Increase of memory usage by wazuh-modulesd #23304
• FD increased: Reported in Increase in differents metrics by Wazuh-modulesd #22841

Logs 🔴

Manager 🔴

• Expected in stress test

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 02:26:10 sca: WARNING: Interval overtaken.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 11:33:00 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 11:39:36 wazuh-logcollector: WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files".
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 02:14:19 wazuh-analysisd: WARNING: Rootcheck decoder queue is full.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:19:58 wazuh-logcollector: WARNING: Target 'agent' message queue is full (1024). Log lines may be lost.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-05.log:2024/05/05 08:24:06 wazuh-analysisd: WARNING: Syscollector decoder queue is full.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-05.log:2024/05/05 08:24:32 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 22:13:20 wazuh-analysisd: WARNING: Security Configuration Assessment decoder queue is full.

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4481

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 02:27:50 wazuh-modulesd:azure-logs: ERROR: azure-activity: Returned error code: '1'.

• Reported Wazuh-db errors logs in Footprint tests #23301

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 11:36:50 wazuh-db: ERROR: Socket 39 error: Broken pipe (32)
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 11:36:50 wazuh-db: ERROR: at run_worker(): wnotify_add(39): Bad file descriptor (9)

• Reported https://github.com/wazuh/wazuh-jenkins/issues/5287

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 18:00:08 wazuh-syscheckd: WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir1584': No such file or directory

• Reported Internal error in modulesd:oscap in stress tests logs #9311

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:38 wazuh-modulesd:oscap: ERROR: Internal error. Exiting...

• Reported Vulnerability Detector module is enabled at the start of the stress test #22565

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:17:52 indexer-connector: WARNING: No username and password found in the keystore, using default values.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:17:52 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities', retrying until the connection is successful.
./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:19:04 indexer-connector: WARNING: Failed to sync agent '000' with the indexer.

• Reported https://github.com/wazuh/wazuh-jenkins/issues/6368

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:38 wazuh-modulesd:vulnerability-scanner: WARNING: The 'feed-update-interval' option at module 'vulnerability-detection' must be at least 1 hour. Automatically set to 60 minutes.

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4867

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:20:43 wazuh-remoted: WARNING: Agent key already in use: agent ID '002'

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4469

./ossec_Test_stress_B5118_manager_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:38 wazuh-modulesd:ciscat: WARNING: No evals defined. Exiting...

Windows 🟡

• Expected in Stress Tests

./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:27:14 wazuh-agent: WARNING: Agent buffer is full: Events may be lost.
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:30:00 sca: WARNING: Interval overtaken.
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:30:01 wazuh-agent: WARNING: Agent buffer at 90 %.
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:29:54 wazuh-agent: WARNING: (6906): Real time process: no data. Probably buffer overflow.
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:30:03 wazuh-agent: WARNING: (1960): File limit has been reached (200).
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:24:44 wazuh-agent: WARNING: Target 'agent' message queue is full (1024). Log lines may be lost.

• Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287

./ossec_Test_stress_B5118_windows_2024-05-06/fimError.log:[2024-05-04_02:14:38] [ERROR] (create_delete): files\fimStress.2000558 file cannot be deleted.

• Reported Real time Windows callback process error in Syscheck stress test #12074

./footprint_output/logs/footprint_logs_B5118_windows/logs\2024\May\ossec-05_ERROR.log:2024/05/05 15:23:33 wazuh-agent: ERROR: (6613): Real time Windows callback process: 'Access is denied.' (5).

• Reported Errors when reading files in footprint tests  #18995

./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:25:08 wazuh-agent: ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.12860'. Error code: 2
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:25:08 wazuh-agent: WARNING: At get_user(c:\tmp\syscheck_test\files\fimstress.12860): CreateFile(): The system cannot find the file specified. (2)
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:25:08 wazuh-agent: ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.12877'. Error code: 2

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4469

./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 21:27:04 wazuh-modulesd:ciscat: ERROR: Timeout expired executing 'C:\cis-cat\benchmarks\CIS_Microsoft_Windows_Server_2016_Benchmark_v1.0.0-xccdf.xml'.
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 17:26:05 wazuh-modulesd:ciscat: ERROR: Report result file 'tmp\ciscat-report.txt' missing: No such file or directory
./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 17:26:05 wazuh-modulesd:ciscat: ERROR: Failed reading scan results for policy 'C:\cis-cat\benchmarks\CIS_Microsoft_Windows_Server_2016_Benchmark_v1.0.0-xccdf.xml'

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4867

./ossec_Test_stress_B5118_windows_2024-05-06/logs\2024\May\ossec-03.log:2024/05/03 15:22:23 wazuh-agent: ERROR: (1216): Unable to connect to '[172.31.6.10]:1514/tcp': 'No connection could be made because the target machine actively refused it.'.

Ubuntu 🟡

• Expected in Stress Tests

./footprint_output/logs/footprint_logs_B5118_ubuntu/var/ossec/logs/ossec_WARNING.log:2024/05/06 00:00:12 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.
./footprint_output/logs/footprint_logs_B5118_ubuntu/var/ossec/logs/ossec_WARNING.log:2024/05/06 00:01:05 sca: WARNING: Interval overtaken.
./footprint_output/logs/footprint_logs_B5118_manager/var/ossec/logs/wazuh/2024/May/ossec-05_WARNING.log:2024/05/05 00:01:05 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.
./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 00:09:20 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.
./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 03:24:40 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.

• Reported https://github.com/wazuh/wazuh-jenkins/issues/5633

./footprint_output/logs/footprint_logs_B5118_ubuntu/var/ossec/logs/wazuh/2024/May/ossec-04_ERROR.log:2024/05/04 01:52:06 wazuh-logcollector: ERROR: Error on lstat '/tmp/logcollector_test/flood-3-9.log' due to [(2)-(No such file or directory)]

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4469

./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:13 wazuh-modulesd:ciscat: ERROR: CIS-CAT tool not found at '/var/ossec/wodles/cis-cat'.

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4867

./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:30 wazuh-agentd: ERROR: (1216): Unable to connect to '[172.31.6.10]:1514/tcp': 'Connection refused'.
./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-05.log:2024/05/05 12:46:56 wazuh-modulesd: WARNING: Process locked due to agent is offline. Waiting for connection...

• Reported Internal error in modulesd:oscap in stress tests logs #9311

./ossec_Test_stress_B5118_ubuntu_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:43 wazuh-modulesd:oscap: ERROR: Internal error. Exiting...

Centos 🟡

• Expected in stress test

./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 00:00:11 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 00:01:16 sca: WARNING: Interval overtaken.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 01:04:12 wazuh-agentd: WARNING: Agent buffer is flooded: Producing too many events.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 00:06:00 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 00:09:10 wazuh-logcollector: WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files".

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4469

./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 00:00:36 wazuh-modulesd:ciscat: ERROR: Report result file 'tmp/ciscat-report.txt' missing: No such file or directory
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/ossec.log:2024/05/06 00:00:36 wazuh-modulesd:ciscat: ERROR: Failed reading scan results for policy '/var/ossec/wodles/cis-cat/benchmarks/CIS_Google_Chrome_Benchmark_v1.2.0-xccdf.xml'

• Reported https://github.com/wazuh/wazuh-jenkins/issues/5287

./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-04.log:2024/05/04 02:33:46 wazuh-syscheckd: WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir1581': No such file or directory

• Reported Internal error in modulesd:oscap in stress tests logs #9311

./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:20:26 wazuh-modulesd:oscap: ERROR: Internal error. Exiting...

• Reported https://github.com/wazuh/wazuh-jenkins/issues/4867

./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:23 wazuh-agentd: ERROR: Connection socket: Connection reset by peer (104)
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:23 wazuh-agentd: ERROR: (1137): Lost connection with manager. Setting lock.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:23 wazuh-agentd: ERROR: (1216): Unable to connect to '[172.31.6.10]:1514/tcp': 'Connection refused'.
./ossec_Test_stress_B5118_centos_2024-05-06/var/ossec/logs/wazuh/2024/May/ossec-03.log:2024/05/03 15:22:23 wazuh-modulesd: WARNING: Process locked due to agent is offline. Waiting for connection...

[santipadilla]

LGTM

[juliamagan]

LGTM