WIP: syslog-ng-ctl: add start and stop internal log message command. #2840

mehul-m-prajapati · 2019-07-21T05:37:12Z

Hi,

This patch will add 2 new syslog-ng-ctl commands,

syslog-ng-ctl internal-logs --start (Start collecting internal log messages)
syslog-ng-ctl internal-logs --stop (Display collected internal log messages and stop further collection of messages)

Later, I will use these commands in debun to save internal log messages in a file.

To set message levels,

syslog-ng-ctl debug --set 1
syslog-ng-ctl internal-logs start
#wait 
syslog-ng-ctl stop
syslog-ng-ctl debug --set 0

resolves mehul-m-prajapati/gsoc-debun-tool#5

Thanks,
Mehul

kira-syslogng · 2019-07-21T05:37:14Z

This user does not have permission to start the build. Can one of the admins verify this patch and start the build?
(admin: you have the next options (make sure you checked the code):
"ok to test" to accept this pull request (and further changes) for testing
"test this please" for a one time test run
"add to whitelist" add author of a Pull Request to whitelist (globally, be careful, it means this user can trigger kira for any PR)
do nothing -> CI won't start)

kira-syslogng · 2019-07-21T05:55:49Z

Build FAILURE

kira-syslogng · 2019-07-22T05:44:41Z

Build FAILURE

kira-syslogng · 2019-07-22T22:16:15Z

Build FAILURE

lib/afinter.c

mehul-m-prajapati · 2019-07-23T13:04:39Z

``

Could you please explain the reason for this change?

With this change, it will collect logs at higher levels. I think it was one of the requirements of this feature.

mehul-m-prajapati · 2019-07-23T17:26:15Z

This is inheritly racy, you get messages and stop collection. Any number of message could arrive between the two call. First you have to stop the collection and than collect the logs. (or done in one step)

ok. I will first stop the collection and then get the internal message logs

kira-syslogng · 2019-07-23T18:08:21Z

Build FAILURE

Kokan · 2019-07-24T06:02:27Z

``

Could you please explain the reason for this change?

With this change, it will collect logs at higher levels. I think it was one of the requirements of this feature.

Yes, it is. But the question how much higher should it be is still open, well it gave an answer as highest. But imho that is not good enough. So it should be up to the user to decide, it could be configured like syslog-ng-ctl internal-logs start --debug/--warning.
On the other hand this is kinda a possibility already via syslog-ng-ctl:

Syntax: /tmp/install/sbin/syslog-ng-ctl <command> [options]
Possible commands are:
...
    verbose              Enable/query verbose messages
    debug                Enable/query debug messages
    trace                Enable/query trace messages
...

Which is good enough for now, one could simply do:

syslog-ng-ctl debug --set 1
syslog-ng-ctl internal-logs start
#wait 
syslog-ng-ctl stop
syslog-ng-ctl debug --set 0

lib/afinter.c

kira-syslogng · 2019-07-24T12:42:56Z

Build SUCCESS

kira-syslogng · 2019-07-24T13:04:15Z

Build SUCCESS

kira-syslogng · 2019-07-24T13:34:13Z

Build SUCCESS

kira-syslogng · 2019-07-24T15:35:16Z

Build SUCCESS

kira-syslogng · 2020-01-04T02:38:34Z

Build FAILURE

kira-syslogng · 2020-01-04T03:10:34Z

Build SUCCESS

kira-syslogng · 2020-01-05T04:31:17Z

Build SUCCESS

kira-syslogng · 2020-01-20T04:09:19Z

Build SUCCESS

kira-syslogng · 2020-01-30T05:24:21Z

Build SUCCESS

kira-syslogng · 2020-02-09T00:00:46Z

Build SUCCESS

kira-syslogng · 2020-02-20T05:13:42Z

Build SUCCESS

kira-syslogng · 2020-02-29T23:54:19Z

Build SUCCESS

furiel · 2020-03-03T11:55:11Z

lib/afinter.c

+  while (!g_queue_is_empty(internal_msg_queue))
+    {
+      msg = g_queue_pop_head(internal_msg_queue);
+      msg_text = log_msg_get_value(msg, LM_V_MESSAGE, NULL);


I suggest doing something similar that is used in dqtool (dqtool.c). Dqtool allows specifying a template function as cli parameter. You can use that input to format the message. If template function is not provided, I think the same default can be fine that is used in dqtool: log_template_compile(template, "$DATE $HOST $MSGHDR$MSG\n", NULL);.

log_writer_format_log would be somewhat difficult to use, as you would need a (at least partially) initialized LogWriter instance.

furiel · 2020-03-03T11:57:34Z

lib/mainloop-control.c

+control_internal_logs(ControlConnection *cc, GString *command, gpointer user_data)
+{
+  GString *result = g_string_new("");
+  gchar **arguments = g_strsplit(command->str, " ", 0);


This should be freed in the end with g_strfreev().

furiel · 2020-03-03T12:02:51Z

syslog-ng-ctl/syslog-ng-ctl.c

+slng_internal(int argc, char *argv[], const gchar *mode, GOptionContext *ctx)
+{
+  GString *cmd = g_string_new("");
+


If neither start or stop are provided, I think we should write an error. Now the cli exits silently. Well the error code is not zero, so not entirely silently. Still, we could write something out.

furiel · 2020-03-03T12:03:26Z

syslog-ng-ctl/syslog-ng-ctl.c

+  else if (config_options_stop)
+    g_string_assign(cmd, "INTERLOGS STOP");
+  else
+    return 1;


We should not allow that start and stop are provided at the same time.

furiel · 2020-03-03T12:07:44Z

lib/mainloop-control.c

+  else if (g_str_equal(arguments[1], "STOP"))
+    {
+      afinter_stop_live_collection();
+      afinter_get_collected_messages(result);


I get assertion error with the following case:

@version: 3.25 log { source { example-msg-generator(num(1)); }; destination { file(/dev/stdout); }; };

Start syslog-ng with ./syslog-ng -Fe -f ../etc/internal.conf.

After calling start, and then stop, syslog-ng emits:

[2020-03-03T13:05:58.048621] g_queue_is_empty: assertion 'queue != NULL' failed

furiel · 2020-03-03T12:10:54Z

lib/afinter.c

+    }
+
+  if (!strlen(result->str))
+    g_string_assign(result, "No live messages collected");


The log collection does not work for me some reason. I am generating messages continuously with:

@version: 3.25 log { source { example-msg-generator(); }; destination { file(/dev/stdout); }; };

Starting syslog-ng with -Fevdt, I can see internal logs generating continuously. However when I start the cli, wait a few seconds, then stop it, I only see this error message: No live messages collected.

My bad. If I start syslog-ng in foreground, then there will be no internal messages.

Could you please handle as error if someone wants to start the log collection while syslog-ng is running in foreground? Others can run into this as well.

furiel · 2020-03-03T12:35:48Z

lib/afinter.c

+
+  g_static_mutex_lock(&internal_msg_lock);
+
+  while (!g_queue_is_empty(internal_msg_queue))


I steadily get deadlock in the following case:

@version: 3.25 log { source { example-msg-generator(); }; destination { file(/dev/stdout); }; };

Start syslog-ng with ./syslog-ng -F -f ../etc/internal.conf (! just -F, not -Fdt. With -Fdt, I did not see deadlock yet).

After start and stop, syslog-ng hangs. I crashed syslog-ng with segv (kill -segv ), then checked the core dump with gdb: I see syslog-ng hangs while trying to grab a lock during pushing internal message. Moving up in the stack, the message is triggered by g_queue_is_empty (through a g_log call). The hint is internal_msg_queue is empty for some reason, and I think the code wants to emit the same assert that I mentioned in the other comment. So maybe the root cause and the fix is the same.

furiel · 2020-03-03T12:44:34Z

lib/afinter.c

+
+  while (!g_queue_is_empty(internal_msg_queue))
+    {
+      msg = g_queue_pop_head(internal_msg_queue);


This message needs to be unreffed using log_msg_unref.

furiel · 2020-03-16T10:14:23Z

@mehul-m-prajapati I think this PR is close to the finish line. Did you have time to look into the review comments? Is there anything to be clarified? Let me know if you need any help.

mehul-m-prajapati · 2020-03-16T20:21:14Z

@mehul-m-prajapati I think this PR is close to the finish line. Did you have time to look into the review comments? Is there anything to be clarified? Let me know if you need any help.

Hi @furiel ,

Thanks for taking time to review my PR. I will make changes in this week.

afinter: add functions to collect internal messages at higher level. Signed-off-by: Mehul Prajapati <mehul.encs@gmail.com>

kira-syslogng · 2021-05-11T02:25:19Z

Build SUCCESS

mehul-m-prajapati changed the title ~~syslog-ng-ctl: add start and stop internal log messgages command.~~ [WIP] syslog-ng-ctl: add start and stop internal log messgages command. Jul 22, 2019

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from dc23973 to 2e032e7 Compare July 22, 2019 05:25

gaborznagy added the community-driven-merge label Jul 22, 2019

mehul-m-prajapati force-pushed the gsoc-internal-logs branch 2 times, most recently from ce14eec to 499ca1e Compare July 22, 2019 21:59

mehul-m-prajapati mentioned this pull request Jul 23, 2019

requesting a feature: internal ring buffer for internal() messages #2654

Open

Kokan reviewed Jul 23, 2019

View reviewed changes

lib/afinter.c Outdated Show resolved Hide resolved

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 499ca1e to 0849bb3 Compare July 23, 2019 17:49

Kokan added the user-visible-feature label Jul 24, 2019

Kokan reviewed Jul 24, 2019

View reviewed changes

lib/afinter.c Show resolved Hide resolved

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 0849bb3 to db3e2f8 Compare July 24, 2019 12:18

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from db3e2f8 to 2aeaa5f Compare July 24, 2019 12:43

mehul-m-prajapati closed this Jul 24, 2019

mehul-m-prajapati reopened this Jul 24, 2019

mehul-m-prajapati closed this Jul 24, 2019

mehul-m-prajapati reopened this Jul 24, 2019

mehul-m-prajapati changed the title ~~[WIP] syslog-ng-ctl: add start and stop internal log messgages command.~~ syslog-ng-ctl: add start and stop internal log messgages command. Jul 28, 2019

lbudai self-requested a review July 31, 2019 08:38

mehul-m-prajapati changed the title ~~syslog-ng-ctl: add start and stop internal log message command.~~ WIP: syslog-ng-ctl: add start and stop internal log message command. Jan 4, 2020

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from c9c13fb to 1ee4e6a Compare January 4, 2020 02:50

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 1ee4e6a to 0d5dcb9 Compare January 5, 2020 04:07

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 0d5dcb9 to d42e9d1 Compare January 20, 2020 03:46

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from d42e9d1 to e2c2a8e Compare January 30, 2020 05:04

Kokan self-requested a review February 3, 2020 11:59

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from e2c2a8e to 243a0e6 Compare February 8, 2020 23:37

Kokan changed the title ~~WIP: syslog-ng-ctl: add start and stop internal log message command.~~ syslog-ng-ctl: add start and stop internal log message command. Feb 10, 2020

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 243a0e6 to acef9ee Compare February 20, 2020 04:50

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from acef9ee to 4fada10 Compare February 29, 2020 23:30

furiel reviewed Mar 3, 2020

View reviewed changes

mehul-m-prajapati changed the title ~~syslog-ng-ctl: add start and stop internal log message command.~~ WIP: syslog-ng-ctl: add start and stop internal log message command. Mar 22, 2020

syslog-ng-ctl: add start and stop internal log messgages command.

c9f6bf8

afinter: add functions to collect internal messages at higher level. Signed-off-by: Mehul Prajapati <mehul.encs@gmail.com>

mehul-m-prajapati force-pushed the gsoc-internal-logs branch from 4fada10 to c9f6bf8 Compare May 11, 2021 02:01

MrAnno removed the request for review from lbudai February 12, 2022 20:40

MrAnno marked this pull request as draft February 12, 2022 20:40

MrAnno removed the user-visible-feature label May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: syslog-ng-ctl: add start and stop internal log message command. #2840

WIP: syslog-ng-ctl: add start and stop internal log message command. #2840

mehul-m-prajapati commented Jul 21, 2019 •

edited

kira-syslogng commented Jul 21, 2019

kira-syslogng commented Jul 21, 2019

kira-syslogng commented Jul 22, 2019

kira-syslogng commented Jul 22, 2019

mehul-m-prajapati commented Jul 23, 2019

mehul-m-prajapati commented Jul 23, 2019 •

edited

kira-syslogng commented Jul 23, 2019

Kokan commented Jul 24, 2019 •

edited

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jan 4, 2020

kira-syslogng commented Jan 4, 2020

kira-syslogng commented Jan 5, 2020

kira-syslogng commented Jan 20, 2020

kira-syslogng commented Jan 30, 2020

kira-syslogng commented Feb 9, 2020

kira-syslogng commented Feb 20, 2020

kira-syslogng commented Feb 29, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel Mar 3, 2020

furiel commented Mar 16, 2020

mehul-m-prajapati commented Mar 16, 2020 •

edited

kira-syslogng commented May 11, 2021


		g_static_mutex_lock(&internal_msg_lock);

		while (!g_queue_is_empty(internal_msg_queue))

WIP: syslog-ng-ctl: add start and stop internal log message command. #2840

Are you sure you want to change the base?

WIP: syslog-ng-ctl: add start and stop internal log message command. #2840

Conversation

mehul-m-prajapati commented Jul 21, 2019 • edited

kira-syslogng commented Jul 21, 2019

kira-syslogng commented Jul 21, 2019

kira-syslogng commented Jul 22, 2019

kira-syslogng commented Jul 22, 2019

mehul-m-prajapati commented Jul 23, 2019

mehul-m-prajapati commented Jul 23, 2019 • edited

kira-syslogng commented Jul 23, 2019

Kokan commented Jul 24, 2019 • edited

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jul 24, 2019

kira-syslogng commented Jan 4, 2020

kira-syslogng commented Jan 4, 2020

kira-syslogng commented Jan 5, 2020

kira-syslogng commented Jan 20, 2020

kira-syslogng commented Jan 30, 2020

kira-syslogng commented Feb 9, 2020

kira-syslogng commented Feb 20, 2020

kira-syslogng commented Feb 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

furiel commented Mar 16, 2020

mehul-m-prajapati commented Mar 16, 2020 • edited

kira-syslogng commented May 11, 2021

mehul-m-prajapati commented Jul 21, 2019 •

edited

mehul-m-prajapati commented Jul 23, 2019 •

edited

Kokan commented Jul 24, 2019 •

edited

mehul-m-prajapati commented Mar 16, 2020 •

edited