Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rotate log files based on size #2964

Open
amdrsantos opened this issue Oct 7, 2019 · 7 comments · May be fixed by #4145
Open

Rotate log files based on size #2964

amdrsantos opened this issue Oct 7, 2019 · 7 comments · May be fixed by #4145
Labels
enhancement

Comments

@amdrsantos
Copy link

Description of the problem

When using the logrotate with common configuration, log entries can be lost between the time in which the logrotate copies the file and the truncate of the file.

Proposed solution

I suggest to rotate the log within syslog-ng based in a file size limit configuration parameter. Additionally one parameter with the desired suffix can be added.
I try was already done in https://github.com/pld-linux/syslog-ng/blob/master/log-file-size-limit.patch

Alternatives

Add a disk buffer and reliable properties in syslog-ng configuration for each log file, and configure in logrotate the restart of syslog-ng in the end of each rotation. This solution has the disadvantage of periodically have to restart the syslog-ng. Potentially it can be too many restarts of syslog-ng depending on number of logs, size limits, and events recorded.

Additional context

Already have a patch ready
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 7, 2019
@amdrsantos
block-name: First implementation of feature syslog-ng#2964
e2557ea 
Rotation of log files when the file reaches a size limit.

Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 8, 2019
@amdrsantos
Implementation of feature syslog-ng#2964
d80e8f4 
Corrected unsed variable error.
Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 8, 2019
@amdrsantos
block-name: Implementation of Feature syslog-ng#2964
c2f23bb 
Correcte identation
Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 8, 2019
@amdrsantos
block-name: First implementation of feature syslog-ng#2964
c5a38b2 
Identation Corrected.
Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 8, 2019
@amdrsantos
block-name: First implementation of feature syslog-ng#2964
1df3316 
Rotation of log files when the file reaches a size limit.

Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
amdrsantos added a commit to amdrsantos/syslog-ng that referenced this issue Oct 8, 2019
@amdrsantos
block-name: First implementation of feature syslog-ng#2964
097ccf9 
Corrected identation.

Signed-off-by: Alexandre Santos <alexandre.rosas.santos@gmail.com>
@amdrsantos amdrsantos mentioned this issue Nov 19, 2019
Closed
@sudheerreddys
Copy link

HI @amdrsantos

We are trying similar patch for log roatation . however we are facing issue for the new option size_limit

Observation is new options added in modules/affile/affile-grammar.ym or modules/affile/affile-grammar.y files
modules/affile/affile-grammar.c file new code is not generated for the new option size_limit code .
Can you please help me how to refelct the new code from .y file to .c generate .

@amdrsantos
Copy link
Author

Hi @sudheerreddys,
I think this enhancement is being developed by Shikhar Vashistha at GSoC 2022, maybe he can help you with that.

@sudheerreddys
Copy link

Thanks @amdrsantos

Hi @shikharvashistha Please help on this

@sudheerreddys
Copy link

Hi ,
Its generic question when we add new option in syslog-ng.conf file we are adding new field in ..grammer.y file .
I assumed once compile it will generate the ..grammer.c file . my case i have added new code in ..grammer.y file but
the new code is not coming into the ..grammer.c file . for taking the new code from ..grammer.y file shell i need
to pass any compilation options ? please let me know

@MrAnno
Copy link
Collaborator

MrAnno commented Aug 19, 2022 •
edited

@sudheerreddys Executing make should trigger the regeneration of those Bison grammar files.

Please note that whenever a .ym file exists next to its .y version, the .ym file needs to be edited.

I think this enhancement is being developed by Shikhar Vashistha at GSoC 2022, maybe he can help you with that.

That is correct, we have some initial work on this feature with Shikhar. It can be publicly followed in my fork, but it is currently far from a working prototype.

For example, MrAnno#17

@sudheerreddys
Copy link

sudheerreddys commented Aug 21, 2022 •
edited

Thanks @MrAnno you are reply .

Yes i have added new option in the .ym file after compile i have seen in .y its automatically adding new option
However .C (affile-grammar.c )file i did not see the new option code generated. i think due to that i am getting the error .
Please let me known any thing missing in compilation or additional place code need.

please see the below size_lmit option i have added getting the below error when run syslog bin

=======
Error parsing affile, syntax error, unexpected $undefined in /etc/syslog-ng/syslog-ng.conf:23:61-23:71:
21      destination d_local_notice { file("/var/log/local7notice"  template(t_log_format) ); };
22      destination d_event_fifo { file("/tmp/eventlog.fifo"  template(t_log_format) ); };
23----> destination d_messages { file("/var/log/messages" fsync(no) size_limit(7168) template(t_log_format) ); };
23---->                                                             ^^^^^^^^^^
24      filter f_rdklogs { message(lvl=WARN) or message(lvl=ERROR) or message(lvl=FATAL);};
25      filter f_syslogs { level(warn..emerg);};

===========


my diffs are : 
************
Index: syslog-ng-3.24.1/modules/affile/affile-grammar.ym
===================================================================
--- syslog-ng-3.24.1.orig/modules/affile/affile-grammar.ym
+++ syslog-ng-3.24.1/modules/affile/affile-grammar.ym
@@ -102,6 +102,7 @@ affile_grammar_set_wildcard_file_source_
 %token KW_FORCE_DIRECTORY_POLLING

 %token KW_STDIN
+%token KW_SIZE_LIMIT

 %type  <ptr> source_affile
 %type  <ptr> source_affile_params
@@ -289,6 +290,7 @@ dest_affile_option
        | KW_OVERWRITE_IF_OLDER '(' nonnegative_integer ')'     { affile_dd_set_overwrite_if_older(last_driver, $3); }
        | KW_FSYNC '(' yesno ')'                { affile_dd_set_fsync(last_driver, $3); }
        | KW_TIME_REAP '(' nonnegative_integer ')'              { affile_dd_set_time_reap(last_driver, $3); }
+       | KW_SIZE_LIMIT '(' LL_NUMBER ')'       { affile_dd_set_file_size_limit(last_driver, $3); }
        ;

Index: syslog-ng-3.24.1/modules/affile/affile-grammar.h
===================================================================
--- syslog-ng-3.24.1.orig/modules/affile/affile-grammar.h
+++ syslog-ng-3.24.1/modules/affile/affile-grammar.h
@@ -210,7 +210,8 @@ extern int affile_debug;
     KW_MAX_FILES = 10527,
     KW_MONITOR_METHOD = 10528,
     KW_FORCE_DIRECTORY_POLLING = 10529,
-    KW_STDIN = 10530
+    KW_STDIN = 10530,
+    KW_SIZE_LIMIT = 10531
   };
 #endif
 /* Tokens.  */
@@ -380,6 +381,7 @@ extern int affile_debug;
 #define KW_MONITOR_METHOD 10528
 #define KW_FORCE_DIRECTORY_POLLING 10529
 #define KW_STDIN 10530
+#define KW_SIZE_LIMIT 10531

 /* Value type.  */

Index: syslog-ng-3.24.1/modules/affile/affile-parser.c
===================================================================
--- syslog-ng-3.24.1.orig/modules/affile/affile-parser.c
+++ syslog-ng-3.24.1/modules/affile/affile-parser.c
@@ -53,6 +53,7 @@ static CfgLexerKeyword affile_keywords[]
   { "multi_line_garbage", KW_MULTI_LINE_GARBAGE },
   { "multi_line_suffix",  KW_MULTI_LINE_GARBAGE },
   { "time_reap",          KW_TIME_REAP },
+  { "size_limit",         KW_SIZE_LIMIT },
   { NULL }
 };

Index: syslog-ng-3.24.1/modules/affile/affile-dest.h
===================================================================
--- syslog-ng-3.24.1.orig/modules/affile/affile-dest.h
+++ syslog-ng-3.24.1/modules/affile/affile-dest.h
@@ -47,6 +47,7 @@ typedef struct _AFFileDestDriver
   GHashTable *writer_hash;

   gint overwrite_if_older;
+  off_t size_limit;
   gboolean use_time_recvd;
   gint time_reap;
 } AFFileDestDriver;
@@ -60,5 +61,6 @@ void affile_dd_set_overwrite_if_older(Lo
 void affile_dd_set_local_time_zone(LogDriver *s, const gchar *local_time_zone);
 void affile_dd_set_time_reap(LogDriver *s, gint time_reap);
 void affile_dd_global_init(void);
+void affile_dd_set_file_size_limit(LogDriver *s, off_t file_size_limit);

 #endif

@sudheerreddys
Copy link

@MrAnno please help on above one .this for new option under destination affile for .conf

@MrAnno MrAnno linked a pull request Sep 20, 2022 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
3 participants
@MrAnno @amdrsantos @sudheerreddys