How to ingest logs from on-premises to OpenSearch using CLO solution

[JoeShi]

I have applications running on-premises, and would like to ingest logs to the OpenSearch on the AWS cloud.

[JoeShi]

Since Centralized Logging with OpenSearch v2.0, there is a feature called S3 Connector is designed for this purse. The basic idea is sending the logs from on-premise to an Amazon S3 bucket, and use the CLO to create a pipeline to read, parse, and move the data from S3 to OpenSearch clusters.

Here are the overall steps:

• Step 1. Create A Log Config to parse the logs
• Step 2. Create an application log pipeline
• Step 3. Configure Fluent Bit to upload logs to the S3 bucket
• Step 4. Create Index Pattern in OpenSearch Dashboards.
• Step 5. Check the logs in OpenSearch Dashboards - Discover.

Step 1. Create A Log Config to parse the logs

Please follow this documentation to create a Log Config, which can parse your logs.

Step 2. Create an application log pipeline

Please follow this documentation.

Please ignore the Buffer settings, ingest logs from the S3 bucket do not need a Buffering layer. Do remember to choose a bucket (with prefix) without S3 Event Notification configured, otherwise, you will encounter an error. Choose the Log Config, you have created in Step 1.

Once the pipeline is being active, it is ready to ingest logs from S3 to OpenSearch. We should suggest you to choose Gzip for data compression, because it will save both storage, and bandwidth, and your compute (Lambda) cost.

Step 3. Configure Fluent Bit to upload logs to the S3 bucket

By default, Fluent Bit supports upload logs from anywhere to Amazon S3 bucket. If you need to upload to on-premises, then you should configure the aws_access_key_id, and aws_secret_access_key on the on-premise instances. Here is the guide.

Here is an example of uploading Nginx logs to Amazon S3. Please remember to adjust the parameters using this guide

fluent-bit.conf

[SERVICE]
    Flush                       5
    Daemon                      off
    Log_level                   Info
    Http_server                 On
    Http_listen                 0.0.0.0
    Http_port                   2022
    storage.path                /var/fluent-bit/state/flb-storage/
    storage.sync                normal
    storage.checksum            off
    storage.backlog.mem_limit   5M
    Parsers_File                /opt/fluent-bit/etc/applog_parsers.conf

[INPUT]
    Name                tail
    Alias               a243860c-7f85-4867-9ac2-2e107110570e_input
    Tag                 log.a243860c-7f85-4867-9ac2-2e107110570e.6ecc688a-85ac-4a77-81c2-2deb326a0bcb.v1.*
    DB                  /var/fluent-bit/state/flb_container-a243860c-7f85-4867-9ac2-2e107110570e.6ecc688a-85ac-4a77-81c2-2deb326a0bcb.v1.db
    DB.locking          True
    Mem_Buf_Limit       30M
    Buffer_Chunk_Size   512k
    Buffer_Max_Size     5M
    Skip_Long_Lines     On
    Skip_Empty_Lines    On
    Refresh_Interval    10
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      False
    Path_Key            file_name
    Path                /var/log/nginx/
    Parser              nginx_6ecc688a-85ac-4a77-81c2-2deb326a0bcb_v1

[OUTPUT]
    Name                        s3
    Alias                       a243860c-7f85-4867-9ac2-2e107110570e_output
    Match                       log.a243860c-7f85-4867-9ac2-2e107110570e.6ecc688a-85ac-4a77-81c2-2deb326a0bcb.v1.*
    bucket                      centralizedlogging-clloggingbucket5f34e4eb-1xwffab3k2b8d
    region                      us-east-1
    total_file_size             50M
    upload_timeout              60s
    use_put_object              true
    s3_key_format               /AppLogs/osi-pipe-test/year=%Y/month=%m/day=%d/%H-%M-%S-$UUID.gz
    compression                 gzip
    storage_class               INTELLIGENT_TIERING
    json_date_key               time
    json_date_format            iso8601
    tls.verify                  False

applog_parsers.conf

[PARSER]
    Name        nginx_6ecc688a-85ac-4a77-81c2-2deb326a0bcb_v1
    Format      regex
    Time_Key    time
    Regex       (?<remote_addr>\S+)\s+-\s+(?<remote_user>\S+)\s+\[(?<time_local>\d+/\S+/\d+:\d+:\d+:\d+\s+\S+)\]\s+"(?<request_method>\S+)\s+(?<request_uri>\S+)\s+\S+"\s+(?<status>\S+)\s+(?<body_bytes_sent>\S+)\s+"(?<http_referer>[^"]*)"\s+"(?<http_user_agent>[^"]*)"\s+"(?<http_x_forwarded_for>[^"]*)".*

Step 4. Create Index Pattern in OpenSearch Dashboards.

If you are using Apache HTTP Sever / Nginx Log Config, and you have selected to allow CLO to create a sample dashboard for you. You can skip this step.

1. Go to OpenSearch Dashboards, and choose Stack Management in the left navigation bar.
2. Choose Index Patterns, and define the index pattern.
3. Choose the Time field, and select Create index pattern.

Step 5. Check the logs in OpenSearch Dashboards - Discover

1. Go to OpenSearch Dashboards, and choose Discover in the left navigation bar.
2. Choose the index in the left drop-down list.