How to ingest ECS Log into OpenSearch (CLO) v2.0.0

[YikaiHu]

This doc provides guidance of using Centralized Logging with OpenSearch (CLO) to collect logs from Elastic Container Service (ECS) into OpenSearch.

[YikaiHu]

In the following guide, we will config the AWS FileLens and Centralized Logging with OpenSearch to collect the application log.

Step 1 - Using FireLens to send logs to S3

1. Add the required policy

Add Create Log group policy to your ECS Task execution role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "*"
        }
    ]
}

Add S3 putObject policy to your ECS Task role, please replace YOUR_BUCKET_NAME

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

2. Create a new version for your Task Definition

We recommend to use Create revision with JSON mode, and please modify the containerDefinitions part.

What we will do:

1. We add a new container, which called log_router. The new container will collect logs from the task container (This new container is for FireLens).
2. And we update the logConfiguration of our current app container to output the logs to a specified S3 location.

Please replace the following parameters in your task definition:

• YOUR_LOG_BUCKET_NAME
• YOUR_LOG_BUCKET_REGION
• YOUR_S3_LOG_PREFIX
• YOUR_CURRENT_REGION

"containerDefinitions": [
        {
            "name": "app",
            "image": "httpd",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "app-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Name": "s3",
                    "bucket": "YOUR_LOG_BUCKET_NAME",
                    "region": "YOUR_LOG_BUCKET_REGION",
                    "retry_limit": "2",
                    "s3_key_format": "/YOUR_S3_LOG_PREFIX/fluent-bit-logs/%Y/%m/%d/%H-%M-%S",
                    "total_file_size": "1M",
                    "upload_timeout": "1m",
                    "use_put_object": "On"
                }
            }
        },
        {
            "name": "log_router",
            "image": "amazon/aws-for-fluent-bit:stable",
            "cpu": 0,
            "memoryReservation": 50,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "YOUR_CURRENT_REGION",
                    "awslogs-stream-prefix": "firelens"
                }
            },
            "firelensConfiguration": {
                "type": "fluentbit"
            }
        }
    ],

3. Using new Task Definition to update your ECS task or service.

You can see the log in your s3 bucket.

You can debug the FireLens in Task → Logs page, if you encounter any issues.

Step 2 - Config the CLO S3 connector

1. Create a Json Log Config

In this guidance, we use Apache log as example. If your log are in different format, please refer to Log Config documentation to create corresponding log config.

Sample Log:

{"date":"2023-09-07T03:28:34.049166Z","source":"stderr","log":"[Thu Sep 07 03:28:34.048874 2023] [mpm_event:notice] [pid 1:tid 139657060620160] AH00489: Apache/2.4.57 (Unix) configured -- resuming normal operations","container_id":"12b85918e9264362bca5517477a5ec51-527074092","container_name":"app","ecs_cluster":"clo-v2-0-0-ECSClusterStackCLClusterBCB8AA1C-LROMvHv7S4E7","ecs_task_arn":"arn:aws:ecs:us-west-1:428529255974:task/clo-v2-0-0-ECSClusterStackCLClusterBCB8AA1C-LROMvHv7S4E7/12b85918e9264362bca5517477a5ec51","ecs_task_definition":"clo-ecs-log-dev:2"}

Time format

%Y-%m-%dT%H:%M:%S.%LZ

Timezone

UTC+00:00

2. Open CLO portal and create a S3 connector pipeline

1. Choose Amazon S3 as log source
2. Choose the S3 bucket, which you specified in Create a new version for your Task Definition Step.
3. Enter the Prefix filter, which you have configured in Create a new version for your Task Definition Step.
4. Choose On-going mode
5. Compression format choose Uncompressed

3. Create the index pattern

After the pipeline is created, we need to go to OpenSearch Dashboard to create the index pattern.

Enter the index prefix

Choose the time filed

4. Search the log in Discover page

You now should be able to view the logs from ECS tasks in the Discovery of OpenSearch Dashboard.

Limitation

This guide does not include log parsing using custom regex. If you want to parse the log by custom regex, please refer to: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#tutorial-replicate-an-ecs-firelens-task-setup-locally

Reference

https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/s3