Add UI to configure Data Tier defaults in system configuration

[AntonEbel]

At present, the default Index set configuration in System>Configuration does not reflect the existence of the warm tier.

We should add the appropriate UI elements, as from the Index Set Configuration page.

eg.

This toggle which alters which subsequent options are shown. by default, set to data tiering.

The standard data tiering rotation controls

The standard retention controls, offering "Delete" or "Archive". Default the strategy to Delete.

And lastly, check if the user has set up the back end repo for warm tier - if they have everything needed to use warm tier, also show the warm tier config options. Set use of warm tier off by default.

[mpfz0r]

And lastly, check if the user has set up the back end repo for warm tier - if they have everything needed to use warm tier, also show the warm tier config options. Set use of warm tier off by default.

We need to include the index set defaults in the "repo is in use" check then, to prevent deletion of a here referenced repo.

[AntonEbel]

It's already there.

[tellistone]

This seems like something we need in 6.0

Currently customer has no way of using Data Tiering option as default strategy and customers will be actively steered away from it :-(

[AntonEbel]

@grotlue As discussed, the endpoints and the user data for the standard configuration:

GET /api/system/cluster_config/org.graylog2.configuration.IndexSetsDefaultConfiguration

{
    "index_analyzer": "standard",
    "shards": 4,
    "replicas": 0,
    "index_optimization_max_num_segments": 1,
    "index_optimization_disabled": false,
    "field_type_refresh_interval": 5,
    "field_type_refresh_interval_unit": "SECONDS",
    "rotation_strategy_class": "org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategy",
    "rotation_strategy_config": {
        "type": "org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig",
        "max_docs_per_index": 20000000
    },
    "retention_strategy_class": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy",
    "retention_strategy_config": {
        "type": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig",
        "max_number_of_indices": 20
    },
    "data_tiering": {
        "type": "hot_only",
        "index_lifetime_min": "P30D",
        "index_lifetime_max": "P40D"
    },
    "use_legacy_rotation": false,
    "retention_strategy": {
        "type": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig",
        "max_number_of_indices": 20
    },
    "rotation_strategy": {
        "type": "org.graylog2.indexer.rotation.strategies.MessageCountRotationStrategyConfig",
        "max_docs_per_index": 20000000
    }
}

the endpoint for updates is: PUT /api/system/indices/index_set_defaults

As with the index set configuration, the use_legacy_rotation property decides whether we use data tiering or not.

As for the configuration of the index sets, we also have to deal with different configurations of data tiering here. For Graylog open:

    "data_tiering": {
        "type": "hot_only",
        "index_lifetime_min": "P30D",
        "index_lifetime_max": "P40D"
    },

Graylog enterprise:

    "data_tiering": {
        "type": "hot_warm",
        "index_lifetime_min": "P30D",
        "index_lifetime_max": "P40D"
        "warm_tier_enabled": true,
        "warm_tier_repository_name": "minio",
        "index_hot_lifetime_min": "P5D",
        "archive_before_deletion": false
    },

[AntonEbel]

fixed by #18956