4.6.0 - Cannot index event publisher.Event mapper_parsing_exception (with Watchtower container)

[Lefuneste83]

Hi there

Following the last upgrade to 4.6.0 image for manager, indexer and dashboard, all freshly reinstalled from scratch, I am facing a weird problem which is actually quite annoying.

I am monitoring half a dozen of hosts (VM and BM). On most of them I run a Watchtower image to automatically update their local images for various application compose.

Since yesterday, as soon as I deploy the Watchtower compose, my wazuh-manager logs gets totally flooded with the following message :

WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc14b5332759ad2b8, ext:6457425512642, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"954aa80c-7ff6-4c19-be2d-80e909549893","hostname":"wazuh.manager","id":"bda90485-6c1d-4308-b2fb-f7e37ae8987f","name":"wazuh.manager","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"wazuh.manager"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":24312186},"message":"{"timestamp":"2023-11-09T12:49:12.816+0000","rule":{"level":3,"description":"Docker: Command launched in container watchtower. Action: \"exec_start: /watchtower --health-check\"","id":"87907","firedtimes":2431,"mail":false,"groups":["docker"],"gdpr":["IV_32.2"]},"agent":{"id":"012","name":"portainer","ip":"192.168.0.5"},"manager":{"name":"wazuh.manager"},"id":"1699534152.39968595","decoder":{"name":"json"},"data":{"integration":"docker","docker":{"status":"exec_start: /watchtower --health-check","id":"e6908f3a8791945ede85f76aa2bd3f441a99b28633a096123a46675e7e401277","from":"containrrr/watchtower:latest","Type":"container","Action":"exec_start: /watchtower --health-check","Actor":{"ID":"e6908f3a8791945ede85f76aa2bd3f441a99b28633a096123a46675e7e401277","Attributes":{"com":{"centurylinklabs":{"watchtower":"true"},"docker":{"compose":{"config-hash":"ddcbc895893665bf51fce751eb9bcc7837d3e3a486a012e2809dd5878773435a","container-number":"1","image":"sha256:4e5375761c77e7a2997517265919c1bd36dc5a2e0ea570b89e0d84c9984ef379","oneoff":"False","project":"watchtower","service":"watchtower","version":"2.20.2"}}},"execID":"2473613645b4720ac275fa39d2cf7d44d24481a85b29b023361d338e01b413ed","image":"containrrr/watchtower:latest","name":"watchtower"}},"scope":"local","time":"1699534152","timeNano":"1699534152816337920.000000"}},"location":"Wazuh-Docker"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::437090-65026", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000175c70), Source:"/var/ossec/logs/alerts/alerts.json", Offset:24313542, Timestamp:time.Time{wall:0xc14b4ce425064caa, ext:147343542, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x6ab62, Device:0xfe02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [data.docker.Actor.Attributes.com.centurylinklabs.watchtower] tried to parse field [watchtower] as object, but found a concrete value"}

The number of these messages is really insane.

As I have a registered agent running on each of these hosts, these logs add up quite fast and I end up with thousands of these logs very fast. Despite these warning messages, Wazuh manager runs flawlessly.

I have tried all sorts of tunings on the compose definition of Watchtower itself, without any effect.

According to my searches, the issue seems linked to a filebeat template definition, whch is unable to parse the content of data.docker.Actor.Attributes.com.centurylinklabs.watchtower.

I am not sure why I see this occuring now. Is it a problem with Watchtower docker file or a problem with the filebeat template, and if so why do I only see this happening for Watchtower while I have dozens of other images running alongside.

I have been running the Wazuh stack for the last 8 months with no major issues, and Watchtower does not seem to have updated their image recently.

Any idea on how to bypass these messages ? At the moment I must chose between running either the Wazuh stack or the Watchtower stack...

Any help would be appreciated ! Thanks !