Problems with stagemonitor-metrics index mapping in ElasticSearch5.4.3 / Kibana5.4.3

[yacota]

While playing with the following combo : Stagemonitor-0.82.0, ElasticSearch5.4.3, Kibana5.4.3 I came across that the following index definitions : stagemonitor-metrics and stagemonitor-spans are not exactly the same regarding the "@timestamp" field/property definition.
The metrics definition also add ""format": "epoch_millis""

https://github.com/stagemonitor/stagemonitor/blob/master/stagemonitor-tracing-elasticsearch/src/main/resources/stagemonitor-elasticsearch-span-index-template.json

https://github.com/stagemonitor/stagemonitor/blob/master/stagemonitor-core/src/main/resources/stagemonitor-elasticsearch-metrics-index-template.json

I think that because of this mappings of those indexes are not the same in ES

"stagemonitor-spans-2017.08.25" : {
"mappings" : {
"spans" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},

"stagemonitor-metrics-2017.08.25" : {
"mappings" : {
"metrics" : {
"properties" : {
"@timestamp" : {
"type" : "long"
},

In Kibana I can perfectly define the index pattern for spans as [stagemonitor-spans-]YYYY.MM.DD and Kibana founds timestamp field as a "Time-field name" BUT I am unable to do the same for metrics.

[felixbarny]

You're right! But I don't quite understand whats wrong with the mapping. Do you have any clue? The strange thing is that stagemonitor manually creates the index pattern for stagemonitor-metrics @timestamp and there seem to be no problems.

Why do you manually want to create the index pattern?

[yacota]

You point me in the right direction .... "stagemonitor should do that work" ... so I checked my logs and found this

moviesui_1        | 11:59:13.643 [main] INFO org.stagemonitor.core.Stagemonitor - System information: Java 1.8.0_111 (Oracle Corporation) Linux 4.9.36-moby
moviesui_1        | 11:59:13.666 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - Agent attachment
moviesui_1        | 11:59:13.666 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - AlertingPlugin (version 0.82.0)
moviesui_1        | 11:59:13.673 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - CorePlugin (version 0.82.0)
moviesui_1        | 11:59:13.676 [main] INFO org.stagemonitor.core.Stagemonitor - FAIL - DataSource instrumentation (No DataSource instrumented)
**moviesui_1        | 11:59:13.680 [main] INFO org.stagemonitor.core.Stagemonitor - FAIL - Elasticsearch (Elasticsearch is not available)**
moviesui_1        | 11:59:13.680 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - ElasticsearchTracingPlugin (version 0.82.0)
moviesui_1        | 11:59:13.680 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - JdbcPlugin (version 0.82.0)
moviesui_1        | 11:59:13.693 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - JvmPlugin (version 0.82.0)
moviesui_1        | 11:59:13.694 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - ServletPlugin (version 0.82.0)
moviesui_1        | 11:59:13.704 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - SoapTracingPlugin (version 0.82.0)
moviesui_1        | 11:59:13.705 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - TracingPlugin (version 0.82.0)
moviesui_1        | 11:59:13.706 [main] INFO org.stagemonitor.core.Stagemonitor - # stagemonitor configuration, listing non-default values:
moviesui_1        | 11:59:13.710 [main] INFO org.stagemonitor.core.Stagemonitor - stagemonitor.applicationName: moviesui (source: Environment Variables)
moviesui_1        | 11:59:13.711 [main] INFO org.stagemonitor.core.Stagemonitor - stagemonitor.instanceName: moviesui (source: Environment Variables)
moviesui_1        | 11:59:13.716 [main] INFO org.stagemonitor.core.Stagemonitor - stagemonitor.reporting.elasticsearch.url: http://elasticsearch:9200 (source: Environment Variables)

I "forgot" to mention that I am using docker-compose in order to launch all services involved, thus the problem is that I forgot to control the boot sequence, so I need to ensure that ES is completely up&running before launching other apps
The strange thing is that ElasticsearchTracingPlugin does not complain, isn't it ?

For anyone interested check docker/compose#374
Although one must take into account docker/compose#4369

Now logs are nicer :)

**moviesui_1        | 13:35:32.797 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - Elasticsearch**
moviesui_1        | 13:35:32.810 [main] INFO org.stagemonitor.core.Stagemonitor - OK   - ElasticsearchTracingPlugin (version 0.82.0)

And metric's mappings are the ones defined in your json template

stagemonitor-metrics-2017.08.25" : {
    "mappings" : {
      "metrics" : {
        "_all" : {
          "enabled" : false
        },
        "_source" : {
          "enabled" : false
        },
        "dynamic_templates" : [
          {
            "strings" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "keyword"
              }
            }
          }
        ],
        **"properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "epoch_millis"
          },**

Kibana's dashboards and indexes patterns are also there (stagemonitor-metrics-* and stagemonitor-spans-*)

[felixbarny]

One more case where the startup logs saved a lot of headache :)

The strange thing is that ElasticsearchTracingPlugin does not complain, isn't it ?

The ElasticsearchTracingPlugin does not really care if ES is up or not as it can handle ES being down and up again. But when the mapping is not already present (when starting the stack for the first time) it can lead to problems.