Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentd-kubernetes-daemonset segmentation fault #1406

Open
rpurdel opened this issue Jan 19, 2023 · 5 comments
Open

fluentd-kubernetes-daemonset segmentation fault #1406

rpurdel opened this issue Jan 19, 2023 · 5 comments
Labels
bug

Comments

@rpurdel
Copy link

rpurdel commented Jan 19, 2023

Using fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-amd64-1 and sometimes getting a segmentation fault in the k8s metadata filter. Any idea why this is?

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371: [BUG] Segmentation fault at 0x0000000000000030
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0169 e:000168 CFUNC  :dump
c:0029 p:0143 s:0164 e:000162 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_meta
c:0028 p:0010 s:0148 e:000147 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96
c:0027 p:0022 s:0141 e:000140 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0026 p:---- s:0137 e:000136 CFUNC  :each
c:0025 p:0023 s:0133 e:000132 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0024 p:0031 s:0123 e:000122 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94
c:0023 p:0008 s:0116 e:000115 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218 [FINISH]
c:0022 p:---- s:0110 e:000109 IFUNC 
c:0021 p:---- s:0107 e:000106 CFUNC  :each
c:0020 p:---- s:0104 e:000103 CFUNC  :reduce
c:0019 p:0021 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217
c:0018 p:0009 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC 
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `reduce'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `filter'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `dump'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f61e7da0b24 RBP: 0x00007f61da821928 RSP: 0x00007f61dacfb828
 RAX: 0x00007f61da805460 RBX: 0x0000000000000000 RCX: 0x000000000000000d
 RDX: 0x0000000000000000 RDI: 0x00007f61da803bf8 RSI: 0x00007f61da821900
  R8: 0x000000000000000c  R9: 0x00000000000019c8 R10: 0x0000000000000000
 R11: 0x00007f61da821900 R12: 0x000000000000d000 R13: 0x000000000000000a
 R14: 0x0000000000000000 R15: 0x000000000000000a EFL: 0x0000000000010246

-- C level backtrace information -------------------------------------------
@ad-m-ss
Copy link

ad-m-ss commented Feb 23, 2023

I have the following segmentation fault in Kubernetes pods when using Docker image fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.0:

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180: [BUG] Segmentation fault at 0x0000000000000048
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0175 e:000174 CFUNC  :write
c:0029 p:0036 s:0170 e:000169 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180
c:0028 p:0010 s:0163 e:000162 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039
c:0027 p:0022 s:0157 e:000156 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0026 p:---- s:0153 e:000152 CFUNC  :each
c:0025 p:0023 s:0149 e:000148 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0024 p:0058 s:0139 e:000138 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038
c:0023 p:0014 s:0127 e:000126 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922
c:0022 p:0020 s:0119 e:000118 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852
c:0021 p:0011 s:0113 e:000112 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0020 p:0009 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852:in `emit_buffered'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922:in `execute_chunking'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038:in `handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039:in `block in handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `format'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `write'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f4261b69922 RBP: 0x00007f4260e03b18 RSP: 0x00007f42524fb8b0
 RAX: 0x00007f4260eb9101 RBX: 0x00007f4260ea7c00 RCX: 0x0000000000000006
 RDX: 0xffffffffffff9000 RDI: 0xffffffffffffffbf RSI: 0x00007f4260ea7c00
  R8: 0x000000000000000c  R9: 0x0000000000001988 R10: 0x00007f4260eb91a8
 R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
 R14: 0x0000000000000000 R15: 0x0000000000000006 EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------

It's hard for me to tell if it's related.

@emil2k
Copy link

emil2k commented Apr 12, 2023 •
edited

We have also experienced a similar segmentation fault using the image v1.15-debian-cloudwatch-1:

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371: [BUG] Segmentation fault at 0x0000000000010047
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0045 p:---- s:0266 e:000265 CFUNC  :dump
c:0044 p:0143 s:0261 e:000259 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_meta
c:0043 p:0010 s:0245 e:000244 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96
c:0042 p:0011 s:0238 e:000237 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110
c:0041 p:0031 s:0231 e:000230 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94
c:0040 p:0008 s:0224 e:000223 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218 [FINISH]
c:0039 p:---- s:0218 e:000217 IFUNC
c:0038 p:---- s:0215 e:000214 CFUNC  :each
c:0037 p:---- s:0212 e:000211 CFUNC  :reduce
c:0036 p:0021 s:0207 e:000206 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217
c:0035 p:0009 s:0201 e:000200 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193
c:0034 p:0011 s:0193 e:000192 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0033 p:0028 s:0185 e:000184 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106
c:0032 p:0011 s:0178 e:000177 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103
c:0031 p:0038 s:0173 e:000172 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338
c:0030 p:0073 s:0164 e:000163 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288
c:0029 p:0122 s:0156 e:000155 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107 [FINISH]
c:0028 p:---- s:0151 e:000150 CFUNC  :synchronize
c:0027 p:0014 s:0147 e:000146 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137
c:0026 p:0004 s:0142 e:000141 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90
c:0025 p:0009 s:0135 e:000134 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82
c:0024 p:0022 s:0130 e:000129 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0023 p:---- s:0126 e:000125 CFUNC  :each
c:0022 p:0023 s:0122 e:000121 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0021 p:0005 s:0112 e:000111 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81
c:0020 p:0016 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82:in `block in emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90:in `process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107:in `block in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288:in `push'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338:in `update_buffer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103:in `block (2 levels) in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `reduce'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `filter'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `dump'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f3651497922 RBP: 0x00007f3637a03a98 RSP: 0x00007f3637efb220
 RAX: 0x00007f3637ab5401 RBX: 0x00007f3637a9bb00 RCX: 0x0000000000000006
 RDX: 0x00007f3637aa7d00 RDI: 0xffffffffffffffbf RSI: 0x00007f3637a9bb00
  R8: 0x0000000000000000  R9: 0x00007f3628752000 R10: 0x00007f3637ab54a8
 R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
 R14: 0x000000000000ffff R15: 0x0000000000000006 EFL: 0x0000000000010206

-- C level backtrace information -------------------------------------------
2023-04-11 18:11:12 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 18:46:53 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 19:30:29 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:06:03 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:49:46 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Jul 12, 2023
@github-actions
Copy link

This issue was automatically closed because of stale in 30 days

@ad-m-ss
Copy link

ad-m-ss commented Aug 13, 2023

FTR: As problem resolution we migrated from Fluentd to Fluentbit.

@ashie ashie removed the stale label Aug 14, 2023
@ashie ashie reopened this Aug 14, 2023
@ashie ashie added the bug label Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ashie @emil2k @ad-m-ss @rpurdel