4.5 pulls a huge (80MB) dependency

[number492]

Description of the problem

After upgrading from 4.3.1 to 4.5.0 I noticed that Docker images containing syslog-ng grew by about 100 MB (this is a 25% increase in our case). It seems that this is caused mostly by the botocore library, which has a roughly 80 MB data directory. If syslog-ng is used as part of the logging infrastructure in a large containerized deployment, then this can quickly add up.

Proposed solution

If it is only required for typing (and in syslog-ng itself), then there are techniques to hack it out and replace the necessary parts with a stub library: https://blog.frank-mich.com/minimizing-the-size-of-lambdas-by-avoiding-having-boto3-and-its-stubs-in-the-deployment-package/.

Alternatives

Maybe separating the dependent parts of syslog-ng into a different package? I'm using core and mod-python with a fixed config, I'm pretty sure I won't be needing boto for now.

Additional context

During this investigation, I tried grepping for dependencies in https://github.com/syslog-ng/syslog-ng/blob/master/requirements.txt through this repository, and I found few libraries imported via conventional means. I don't know much about syslog-ng's internals, could you please check if these dependencies are actually used, and maybe cut them down if not?

[bazsi]

Thanks for the report and the pointer to that blog article. I didn't recognize that boto was this huge. I guess this van be attributed to our s3 destination that was added recently

[bazsi]

It indeed is caused by S3.

Our module handles the case where the python dependencies are not installed and then bails out with an error.

So it's actually possible to just remove the boto libraries. It will work as long as you don't use s3

Which container is this?

[number492]

I'm using syslog-ng in a private project. Right now I'm deleting the data directory of the botocore library in the same Docker layer that installs syslog-ng from the deb package. This kind of works, but it's a bit hacky. Anyway, this is not an urgent issue, but this might come up later with other users as people start upgrading to 4.5.0, so please keep the install's size in mind when looking for optimization opportunities.

[czanik]

The idea of a per Python module requirements.txt came up already a couple of times: for example installing Kubernetes related modules does not make sense at all on FreeBSD. Right now there is only a global requirements.txt. As a workaround, I separated Python support from modules written in Python in my RPM and FreeBSD packaging. System python packages are pulled in where available, otherwise syslog-ng-update-virtualenv needs to be used.