Not able to fetch metadata filename from the log files. #5379
Comments
|
log the message with the template RSYSLOG_DebugFormat so we can see all the
properties and variables for the log. In a large percentage of cases, it ends up
being a typo or something like that.
Also check for any config complaints with rsyslogd -N1
Now, 8.24 is very old (released Jan 2017) with some bugfixes backported (by
Amazon or RedHat that Amazon bases their OS on), if you can upgrade to a more
current version, the odds of us being able to help go up a lot.
I know that imfile has had a complete rewrite since 8.24, so it's very possible
that a new version will just work
David Lang
…On Sun, 12 May 2024, harishkm1234 wrote:
Date: Sun, 12 May 2024 12:00:53 -0700
From: harishkm1234 ***@***.***>
Reply-To: rsyslog/rsyslog
***@***.***>
To: rsyslog/rsyslog ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [rsyslog/rsyslog] Not able to fetch metadata filename from the log
files. (Issue #5379)
Hello,
The $!metadata!filename is not able to print the log filename. I'm using below config for fetching the log filenames of pod logs from eks nodes. There are no errors and other content of the message is able print.
/etc/rsyslog.d/30-microservice-client-rsyslog.conf:
```
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 20500
# Load the imfile module
$ModLoad imfile
# Reliable Event Logging Protocol - network transport that we use for logs
module(load="omrelp")
input(type="imfile"
File="/var/log/containers/*cwf-ss_cwf-public-service-web*.log"
Tag="cwf-public-service-web"
addMetadata="on"
Severity="info"
Facility="local0"
)
template(name="docker_apps_remote" type="string"
string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %$!metadata!filename% %syslogtag%:%msg%\n"
)
action(name="docker_apps_remote" type="omrelp" target="xxxxxxxxxx.com" port="20500" template="docker_apps_remote" tls="off"
# TLS directives only when encrypting traffic
queue.filename="docker_app_queue"
queue.type="linkedlist"
queue.spoolDirectory="/var/spool/rsyslog"
queue.highwatermark="8000"
queue.lowwatermark="6000"
queue.maxdiskspace="1g"
queue.timeoutenqueue="0"
queue.saveonshutdown="on"
queue.size="10000" )
*.* @@xxxxxxxxxx.com:20500
```
Rsyslog version:
```
rsyslogd 8.24.0-57.amzn2.2.0.2, compiled with:
PLATFORM: x86_64-koji-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
```
See http://www.rsyslog.com for more information.
Thank you for the help.
|
|
I tried to upgrade the version by following steps here but don't think its going to work for amazon linux 2. Seeing the same logs no change in it. Not sure I'm doing it in correct way but still not working. |
|
debug output in case it helps. rsyslogd -dn |
|
don't define RSYSLOG_DebugFormat, that's a predefined format, use it to write to
a file and send a sample message.
David Lang
|
|
Able to get few logs. config: |
|
Ok, this shows that $!metadata!filename is populated, so the question is what is
wrong with your template.
please make a new version of the template you are sending with and add a \n to
the end of the line, then use it to write the logs to a local file so we can see
what we get.
David Lang
On Sun, 12 May 2024, harishkm1234 wrote:
… Date: Sun, 12 May 2024 13:07:04 -0700
From: harishkm1234 ***@***.***>
Reply-To: rsyslog/rsyslog
***@***.***>
To: rsyslog/rsyslog ***@***.***>
Cc: David Lang ***@***.***>, Comment ***@***.***>
Subject: Re: [rsyslog/rsyslog] Not able to fetch metadata filename from the
log files. (Issue #5379)
Able to get few logs.
```
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972800678Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run''
escaped msg: '2024-05-12T20:04:10.972800678Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972800678Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504119" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972804698Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker''
escaped msg: '2024-05-12T20:04:10.972804698Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in`block in run_worker''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972804698Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504278" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972812228Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process''
escaped msg: '2024-05-12T20:04:10.972812228Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in`main_process''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972812228Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504457" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972816179Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker''
escaped msg: '2024-05-12T20:04:10.972816179Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in`run_worker''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972816179Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504629" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972819859Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>''
escaped msg: '2024-05-12T20:04:10.972819859Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972819859Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504799" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972823519Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
escaped msg: '2024-05-12T20:04:10.972823519Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972823519Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6504980" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972827429Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
escaped msg: '2024-05-12T20:04:10.972827429Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972827429Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6505143" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972831169Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>''
escaped msg: '2024-05-12T20:04:10.972831169Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972831169Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6505306" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972834889Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `load''
escaped msg: '2024-05-12T20:04:10.972834889Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `load''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972834889Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `load''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6505468" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.972838669Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `<main>''
escaped msg: '2024-05-12T20:04:10.972838669Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `<main>''
inputname: imfile rawmsg: '2024-05-12T20:04:10.972838669Z stdout F 2024-05-12 20:04:10 +0000 [error]: #0 /usr/bin/fluentd:25:in `<main>''
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6505578" } }
$.:
$/:
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'ip-10-223-207-43', PRI: 134,
syslogtag 'cwf-public-service-web', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 12 20:04:11', STRUCTURED-DATA: '-',
msg: '2024-05-12T20:04:10.97668624Z stdout F 2024-05-12 20:04:10 +0000 [error]: Worker 0 exited unexpectedly with status 1'
escaped msg: '2024-05-12T20:04:10.97668624Z stdout F 2024-05-12 20:04:10 +0000 [error]: Worker 0 exited unexpectedly with status 1'
inputname: imfile rawmsg: '2024-05-12T20:04:10.97668624Z stdout F 2024-05-12 20:04:10 +0000 [error]: Worker 0 exited unexpectedly with status 1'
$!:{ "metadata": { "filename": "\/var\/log\/containers\/fluentd-daemonset-78w99_default_fluentd-c21fb9cbf69daf53b310642d42b023d62166bd52a48e4bd30a4a2e44bcb094d8.log", "fileoffset": "6505690" } }
$.:
$/:
```
config:
```
action(name="RSYSLOG_DebugFormat" type="omfile" file="/var/log/rsyslog_debug.log" template="RSYSLOG_DebugFormat")
```
|
|
Thank You David Lang. Its working now. Not sure what I was doing wrong. Is there any way I can fetch only till the first underscore from the filename? I only need to append pod name. |
|
On Sun, 12 May 2024, harishkm1234 wrote:
Is there any way I can fetch only till the first underscore from the filename?
```
/var/log/containers/cwf-admin-service-web-cwf-ss-6d86ddf4b5-rfvqp_cwf-ss_cwf-admin-service-web-exporter-7aed136048047f336fd3de1a2e9cd7d5a322a19d08a7abdd59f499825e69e12d.log
```
I only need to append pod name.
```
cwf-admin-service-web-cwf-ss-6d86ddf4b5-rfvqp
```
see the functions at
https://www.rsyslog.com/doc/rainerscript/functions/index.html
what you will want to do is create a new variable (say $!podname) and use set to
set it to what you want, and then use $!podname in your template
David Lang
|
|
I'm able to forward to a local file but not able to send to the remote server. In local file able to see the pod name. |
|
able to get the logs on the remote server after changing type="omfwd" to type="omrelp" |
|
Sorry it seems unrelated for this issue. I'm trying to add few brackets on the client side but in the remote server its considering the whole as syslogtag. How can I use brackets inside config? On the remote server side, the syslog tag is the whole. ex: cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr]: In the debug format also syslogtag is same. |
|
Changed it to programname. It resolved. But the entries are still having the TIMESTAMP and FROMHOST. How to avoid this kind of cache? |
|
Hi @davidelang , could you please give me few suggestions if you have some info about this? I understand this is not related, but any help would be greatly appreciated. |
|
what is the template you are using to send the message and please post a sample
of the message you are sending (write it to a file so we can see it)
if things are ending up in syslogtag, that means that you are not formatting the
message correctly and the parser at the far end is seeing your podname in the
place in the message where syslogtag is supposed to be.
David Lang
…On Tue, 14 May 2024, harishkm1234 wrote:
Hi @davidelang , could you please give me few suggestions if you have some info about this? I understand this is not related, but any help would be greatly appreciated.
Thank You.
|
|
Thank you for the quick response. Right now the server logs are like below with TimeStamp and FromHost entries.: Client side logs don't has TimeStamp and FromHost entries. Server side Rsyslog_DebugFormat: Client side template: rsyslog.conf: cwf-public-service.conf Server side template: |
|
the thing in the [] brackes for a syslogtag is supposed to be a pid, not a long
string, so I would suggest that you move the podname past the syslogtag
however it looks like this is getting parsed reasonably on the server side, and
the podname is in $procid
what is it that you want to take place?
David Lang
…On Tue, 14 May 2024, harishkm1234 wrote:
Date: Tue, 14 May 2024 11:36:06 -0700
From: harishkm1234 ***@***.***>
Reply-To: rsyslog/rsyslog
***@***.***>
To: rsyslog/rsyslog ***@***.***>
Cc: David Lang ***@***.***>, Mention ***@***.***>
Subject: Re: [rsyslog/rsyslog] Not able to fetch metadata filename from the
log files. (Issue #5379)
Thank you for the quick response.
Right now the server logs are like below with TimeStamp and FromHost entries.:
```
May 14 18:19:19 ip-10-223-207-43.ec2.internal cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:19:16.511125796Z stdout F mongoid-5.2.1/lib/mongoid/contextual/mongo.rb:248:in `first'
May 14 18:19:19 ip-10-223-207-43.ec2.internal cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:19:16.511129526Z stdout F mongoid-5.2.1/lib/mongoid/contextual/mongo.rb:248:in `block in first'
May 14 18:19:19 ip-10-223-207-43.ec2.internal cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:19:16.511169226Z stdout F mongoid-5.2.1/lib/mongoid/contextual/mongo.rb:504:in `try_cache'
```
Client side logs don't has TimeStamp and FromHost entries.
```
cwf-public-service-web-istio-proxy[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:25:09.708483035Z stdout F [2024-05-14T18:25:08.996Z] "GET /metrics HTTP/1.1" 200 - via_upstream - "-" 0 820 4 3 "-" "Prometheus/2.45.0" "a76f6051-4d56-4008-8a98-6115f9a5a73d" "10.223.206.228:8080" "10.223.206.228:8080" inbound|8080|| 127.0.0.6:46547 10.223.206.228:8080 10.223.202.17:57028 - default
cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:25:02.436045174Z stdout F 2024-05-14 18:25:02 +0000: [dd49f115-8f62-4383-ac85-968211932360] Started POST "/projects/60073/execution_refresh/6449432d36957e006b19440d" (for 127.0.0.6 ), Session ID: 477bcc6c-121f-11ef-8793-ac72a3883f25, Session Data: {},Shard: default
cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:25:02.436756146Z stdout F 2024-05-14 18:25:02 +0000: [dd49f115-8f62-4383-ac85-968211932360] Completed in 0ms | 404 Not Found | 69 bytes
cwf-public-service-web-istio-proxy[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:25:18.706751271Z stdout F [2024-05-14T18:25:18.020Z] "GET /metrics HTTP/1.1" 200 - via_upstream - "-" 0 820 4 4 "-" "Prometheus/2.45.0" "4a8797a6-8068-4952-ab5c-8ef55ddbb1f3" "10.223.206.228:8080" "10.223.206.228:8080" inbound|8080|| 127.0.0.6:49443 10.223.206.228:8080 10.223.202.17:57028 - default
```
Server side Rsyslog_DebugFormat:
```
Debug line with all properties:
FROMHOST: 'ip-10-223-207-43.ec2.internal', fromhost-ip: '10.223.207.43', HOSTNAME: 'ip-10-223-207-43.ec2.internal', PRI: 13,
syslogtag 'cwf-public-service-web-istio-proxy[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr]', programname: 'cwf-public-service-web-istio-proxy', APP-NAME: 'cwf-public-service-web-istio-proxy', PROCID: 'cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr', MSGID: '-',
TIMESTAMP: 'May 14 18:32:50', STRUCTURED-DATA: '-',
msg: ' 2024-05-14T18:32:48.911723682Z stdout F [2024-05-14T18:32:48.020Z] "GET /metrics HTTP/1.1" 200 - via_upstream - "-" 0 822 4 4 "-" "Prometheus/2.45.0" "ba3307f5-e6f4-4316-b64e-097e9a9183cc" "10.223.206.228:8080" "10.223.206.228:8080" inbound|8080|| 127.0.0.6:50205 10.223.206.228:8080 10.223.202.17:57028 - default'
escaped msg: ' 2024-05-14T18:32:48.911723682Z stdout F [2024-05-14T18:32:48.020Z] "GET /metrics HTTP/1.1" 200 - via_upstream - "-" 0 822 4 4 "-" "Prometheus/2.45.0" "ba3307f5-e6f4-4316-b64e-097e9a9183cc" "10.223.206.228:8080" "10.223.206.228:8080" inbound|8080|| 127.0.0.6:50205 10.223.206.228:8080 10.223.202.17:57028 - default'
inputname: imrelp rawmsg: 'cwf-public-service-web-istio-proxy[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:32:48.911723682Z stdout F [2024-05-14T18:32:48.020Z] "GET /metrics HTTP/1.1" 200 - via_upstream - "-" 0 822 4 4 "-" "Prometheus/2.45.0" "ba3307f5-e6f4-4316-b64e-097e9a9183cc" "10.223.206.228:8080" "10.223.206.228:8080" inbound|8080|| 127.0.0.6:50205 10.223.206.228:8080 10.223.202.17:57028 - default'
$!:
$.:{ "service_name": "cwf-public-service-web", "container_name": "istio-proxy" }
$/:
FROMHOST: 'ip-10-223-207-43.ec2.internal', fromhost-ip: '10.223.207.43', HOSTNAME: 'ip-10-223-207-43.ec2.internal', PRI: 13,
syslogtag 'cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr]', programname: 'cwf-public-service-web', APP-NAME: 'cwf-public-service-web', PROCID: 'cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr', MSGID: '-',
TIMESTAMP: 'May 14 18:33:40', STRUCTURED-DATA: '-',
msg: ' 2024-05-14T18:33:34.501949713Z stdout F 2024-05-14 18:33:34 +0000: **Airbrake: HTTP error: Failed to open TCP connection to errbit.rightscale.com:443 (getaddrinfo: Name or service not known)'
escaped msg: ' 2024-05-14T18:33:34.501949713Z stdout F 2024-05-14 18:33:34 +0000: **Airbrake: HTTP error: Failed to open TCP connection to errbit.rightscale.com:443 (getaddrinfo: Name or service not known)'
inputname: imrelp rawmsg: 'cwf-public-service-web[cwf-public-service-web-cwf-ss-5c87b996ff-v5hhr] 2024-05-14T18:33:34.501949713Z stdout F 2024-05-14 18:33:34 +0000: **Airbrake: HTTP error: Failed to open TCP connection to errbit.rightscale.com:443 (getaddrinfo: Name or service not known)'
$!:
$.:{ "service_name": "cwf-public-service-web", "container_name": "cwf-public-service-web" }
$/
```
Client side template:
rsyslog.conf:
```
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 20500
# Load the imfile module
$ModLoad imfile
# Reliable Event Logging Protocol - network transport that we use for logs
module(load="omrelp")
# Extract the pod-name
set $!podname_path = re_extract($!metadata!filename, "([^_]*)_", 0, 1, "failedToFetchPodname");
set $!podname = re_extract($!podname_path, "([^/]*)\$", 0, 1, "failedToFetchPodname");
# Log entries format.
template(name="docker_apps_remote" type="string"
string="%syslogtag%[%$!podname%] %msg%\n"
)
# Forward logs to remote
action(name="docker_apps_remote" type="omrelp" target="xxxxxxxxxxx.com" port="20500" template="docker_apps_remote" tls="off"
queue.filename="docker_app_queue"
queue.type="linkedlist"
queue.spoolDirectory="/var/spool/rsyslog"
queue.highwatermark="8000"
queue.lowwatermark="6000"
queue.maxdiskspace="1g"
queue.timeoutenqueue="0"
queue.saveonshutdown="on"
queue.size="10000" )
# Rsyslog remote server DNS
*.* @@xxxxxxxxxxx.com:20500
```
cwf-public-service.conf
```
# cwf-public-service-web rsyslog config
# Define input file for cwf-public-service-web logs
input(type="imfile"
File="/var/log/containers/cwf-public-service-web*_cwf-ss_cwf-public-service-web*.log"
Tag="cwf-public-service-web"
Severity="info"
Facility="local0"
addMetadata="on"
)
# Define input file for istio-init logs
input(type="imfile"
File="/var/log/containers/cwf-public-service-web*_cwf-ss_istio-init*.log"
Tag="cwf-public-service-web-istio-init"
Severity="info"
Facility="local0"
addMetadata="on"
)
# Define input file for istio-proxy logs
input(type="imfile"
File="/var/log/containers/cwf-public-service-web*_cwf-ss_istio-proxy*.log"
Tag="cwf-public-service-web-istio-proxy"
Severity="info"
Facility="local0"
addMetadata="on"
)
```
Server side template:
```
set $.service_name = "none";
set $.container_name = "none";
# Check programname and update service_name and container_name accordingly
if ($programname contains "istio-proxy") then {
set $.service_name = re_extract($programname, '^(.*)-istio-proxy$', 0, 1, 0);
set $.container_name = "istio-proxy";
} else if ($programname contains "istio-init") then {
set $.service_name = re_extract($programname, '^(.*)-istio-init$', 0, 1, 0);
set $.container_name = "istio-init";
} else if not ($programname contains "istio-init") and not ($programname contains "istio-proxy") then {
if ( $programname startswith "cwf-") or ($programname startswith "ss-") or ($programname startswith "cm-") then {
set $.service_name = $programname;
set $.container_name = $programname;
}
}
template(name="Dyn_AppLogs" type="string" string="/var/log/rs_ops/eks/%$.service_name%/%$.container_name%.log")
if ( $.service_name != "none" ) and ( $.container_name != "none" ) then {
action(name="Dyn_AppLogs" type="omfile" DirCreateMode="0755" FileCreateMode="0640" FileGroup="syslog" FileOwner="syslog" DynaFile="Dyn_AppLogs" dynaFileCacheSize="500")
stop
}
```
|
|
I see in the client side PROCID is empty. Not able to use in the client Rsyslog template. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
The $!metadata!filename is not able to print the log filename. I'm using below config for fetching the log filenames of pod logs from eks nodes. There are no errors and other content of the message is able print.
/etc/rsyslog.d/30-microservice-client-rsyslog.conf:
Rsyslog version:
See http://www.rsyslog.com for more information.
Thank you for the help.
The text was updated successfully, but these errors were encountered: