Very frustrating situation: fields not not shown

[celesteking]

So I had VERY complex events indexed, now I want to display them to users.
I had "objects in arrays are not supported" message, so I had to use copy_to: hits.messages, but that brings another issue.

It appears Kibana can't display doc_values: true fields. Actually, kibana can't display fields that are returned by ordinary query through ES REST API.

                "fields": {
                    "hits.messages": [
                        "Access denied with code 402 (phase 2). Pattern match \"(.*)\" at TX:960009-OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER-REQUEST_HEADERS.",
                        "Operator EQ matched 0 at REQUEST_HEADERS.",
                        "Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host."
                    ]
                }

Despite hits.messages being mapped as:

"messages" : {
"index" : "not_analyzed",
"type" : "string",
"doc_values" : true
},

it's not shown in Kibana. I can visualize it, but can't display. This is contradicting what a normal human would expect. Also, if you're able to search on it, you should be able to see it -- and it doesn't matter whether additional 8 bytes of RAM would be wasted.

For hits.messages, Kibana telling me: "This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.".

I read the docs, but nowhere I've seen mention of the behaviour. If kibana displays ONLY _source fields, please, write this with big red letters on main kibana manual page as this is very important.

There are so many open bugs at ELK trio that I'm not sure whether I should log any additional or just skip the software.

[rashidkpc]

Yep. Kibana only displays stuff stored in _source. Sorry for the confusion, that is why you can aggregate on it, but not display it.

Cheers

[celesteking]

What's the workaround other than using logstash to rearrange the original event and inject new field[s] with mangled/adjusted data?

[Lifang13]

facing the issue now. Kibana is not user friendly at all