Log Cache Syslog server trims leading and trailing whitespace in the log messages

[chombium]

The Log Cache Syslog Server trims leading and trailing whitespace in the log messages which is changing the structure of the log messages from the apps. This must not be done. I've also checked and made sure that this doesn't happen on the Firehose path. Debugging on the Syslog Path (Forwarder Agent -> Loggregator Agent -> Syslog Agent -> Log Cache -> CF CLI), I've found out that in Log Cache when the Syslog formatted messages are converted to Loggregator V2 envelopes the leading and trailong whitespace in the log message is trimmed.
This should not be done and there are also some valid use cases for having leading whitespace for indentation of

Here is an example of a CF app build with the NodeJS Cloud Native Buildpack. In this case the leading whitespace is used for indentation and formatting of the log output for better visibility.

Output of the pack build command:

===> BUILDING
[builder]
[builder] VCAP Compatibility Buildpack 2.5.1
[builder]   https://github.tools.sap/unified-runtime/vcap-compatibility
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/unified-runtime_vcap-compatibility/helper/exec.d/vcap-application-compatibility-helper
[builder]     Creating /layers/unified-runtime_vcap-compatibility/helper/exec.d/vcap-services-compatibility-helper
[builder]
[builder] Paketo Buildpack for CA Certificates 3.7.0
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] Paketo Buildpack for Node Engine 3.2.2
[builder]   Resolving Node Engine version
[builder]     Candidate version sources (in priority order):
[builder]                 -> ""
[builder]       <unknown> -> ""
[builder]
[builder]     Selected Node Engine version (using ): 20.11.0
[builder]
[builder]   Executing build process
[builder]     Installing Node Engine 20.11.0
[builder]       Completed in 5.552s
[builder]
[builder]   Generating SBOM for /layers/paketo-buildpacks_node-engine/node
[builder]       Completed in 0s
[builder]
[builder]   Configuring build environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_OPTIONS -> "--use-openssl-ca"
[builder]     NODE_VERBOSE -> "false"
[builder]
[builder]   Configuring launch environment
[builder]     NODE_ENV     -> "production"

Output of the staging phase of cf push --lifecycle cnb:

===> BUILDING
Timer: Builder started at 2024-04-19T13:18:12Z
Running build for buildpack paketo-buildpacks/node-engine@3.2.1
Looking up buildpack
Finding plan
Creating plan directory
Preparing paths
Running build command
Paketo Buildpack for Node Engine 3.2.1
Resolving Node Engine version
Candidate version sources (in priority order):
-> ""
<unknown> -> ""
Selected Node Engine version (using ): 20.10.0
Executing build process
Installing Node Engine 20.10.0
Completed in 6.915s
Generating SBOM for /home/vcap/layers/paketo-buildpacks_node-engine/node
Completed in 0s
Configuring build environment
NODE_ENV     -> "production"
NODE_HOME    -> "/home/vcap/layers/paketo-buildpacks_node-engine/node"
NODE_OPTIONS -> "--use-openssl-ca"
NODE_VERBOSE -> "false"
Configuring launch environment
NODE_ENV     -> "production"

As it can be seen, in the current state the log output when the staging happens is not indented and it is hard to follow what is happening.

Although, the current behavior is wrong, I've created a PR in which, in order to leave the current bahavior untouched and not introduce a breaking change, I've added a config parameter which controls if the whitespace should be trimmed or not.
The other option would be to simply change the code so that the leading and trailing whitespace will never be trimmed.

I'm opened for suggestions on how to solve this

[ctlong]

There appears to be no recorded reason for us to be trimming whitespace.

I went spelunking in the code, but didn't turn up any useful information about why the whitespace is being trimmed. Here's the original commit that adds the whitespace trimming, which is also the original commit that adds the Log Cache Syslog Server. No information is recorded in the commit or the tracker story about trimming whitespace.

Additionally, there's no tests related to this behaviour.