Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

instance_profile_credentials are not allowing uploads to my S3 bucket #432

Open
NickBrinsmade opened this issue Oct 17, 2023 · 0 comments
Open

Comments

@NickBrinsmade
Copy link

NickBrinsmade commented Oct 17, 2023

Describe the bug

As my logs are being written to stdout, they are not uploading to the S3 bucket. It works correctly if I specify the aws_key_id and aws_sec_key parameters, but when I remove those parameters to instead use the IAM role attached to the EC2 instance, no files are uploaded to the bucket.

It says in the documentation, to use EC2 IAM role, you need to have the instance_profile_credentials block in your match block, so not sure why it isn't working. Direct uploads to the bucket work via powershell Write-S3Object command, so the role and bucket are both configured correctly.

To Reproduce

Download the latest version of fluent-package from their website. Update C:\opt\fluent\etc\fluent\fluentd.conf to the configuration below. Run fluentd --config C:\opt\fluent\etc\fluent\fluentd.conf in administrator command prompt from the C:\opt\fluent\bin directory.

Expected behavior

Upload windows event logs to the S3 bucket specified.

Your Environment

- Fluentd version: 1.16.2
- TD Agent version: 
- fluent-plugin-s3 version: 1.7.2
- aws-sdk-s3 version: 
- aws-sdk-sqs version:
- Operating system: Windows 10 
- Kernel version:

Your Configuration

<system>
<log>
    rotate_age 30
</log>
</system>

  <source>
    @type windows_eventlog2
    @id windows_eventlog2
    channels security # Also be able to use `<subscribe>` directive.
    read_existing_events false
    read_interval 2
    tag winevt.raw
    <storage>
      @type local
      persistent true
      path ./tmp/storage.json
    </storage>
  </source>

  <match winevt.raw>
    @type stdout
  </match>

  <match **>
    @type s3
    s3_bucket my_bucket_name
    s3_region us-east-1 # region the bucket is in
    path path/inside_my_bucket/
    <instance_profile_credentials>
      # I've tried just leaving the instance_profile_credentials empty and also specifying the defaults ip of 16.254.169.254 and port 80. Neither option worked.
    </instance_profile_credentials>
    <buffer tag,time>
      @type file
      path C:\opt\fluent\etc\fluent\buffer
      timekey 60 # 1 hour partition
      timekey_wait 10s
      timekey_use_utc true # use utc
      chunk_limit_size 256m
    </buffer>
    <format>
      @type json
    </format>
  </match>

Your Error Log

It's not erroring out, it's just not uploading to the bucket.

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant