libcrypto.so error on initial password hashing

[vl-federico]

splunk/splunk:9.2 shows the following error on initialisation:

splunk  | TASK [splunk_common : Hash the password] ***************************************
splunk  | fatal: [localhost]: FAILED! => {
splunk  |     "changed": false,
splunk  |     "cmd": [
splunk  |         "/opt/splunk/bin/splunk",
splunk  |         "hash-passwd",
splunk  |         "!234Qwer"
splunk  |     ],
splunk  |     "delta": "0:00:00.026730",
splunk  |     "end": "2024-02-15 08:35:52.678832",
splunk  |     "rc": 127,
splunk  |     "start": "2024-02-15 08:35:52.652102"
splunk  | }
splunk  |
splunk  | STDERR:
splunk  |
splunk  | /opt/splunk/bin/splunkd: error while loading shared libraries: libcrypto.so.1.0.0: cannot enable executable stack as shared object requires: Invalid argument

This error was not present on splunk/splunk:9.0.7.

[wAuner]

Can confirm that this does currently not work on macOS (Rosetta emulation).
Tested with this command:

docker run --platform linux/amd64 -p 8000:8000 -e "SPLUNK_PASSWORD=mysupersecurepassword" \
             -e "SPLUNK_START_ARGS=--accept-license" \
             -it --name so1 splunk/splunk:latest
             

[dd-Splunk]

It appeared with the latest MacOS Sonoma security patch ( 14.3.1 ).

[outcoldman]

@dd-Splunk I don't believe it is related that much to the MacOS Sonoma security patch, but to the distribution of the Linux image. Splunk version 9.1.2 works without any issues, but all the images 9.1.3+ have issues.

It seems like Splunk have changed how they build lib/libcrypto.so.1.0.0

More than that, if you copy lib/libcrypto.so.1.0.0 from 9.1.2 release to the 9.2.0.1 release - it starts fine.

More than that, if you compare the two libraries:

execstack -q splunk-9.1.2/lib/libcrypto.so.1.0.0
- splunk-9.1.2/lib/libcrypto.so.1.0.0

execstack -q splunk-9.2.0.1/lib/libcrypto.so.1.0.0
X splunk-9.2.0.1/lib/libcrypto.so.1.0.0

So this library libcrypto.so.1.0.0 marked with the executable stack flag in the latest releases, and the system is actually doing a right thing by blocking this library to load, as this seems like a security issue. Why would this library need to have executable memory? Is that a security issue?

I am sure, if you try to run Splunk on any linux with more secure protection like NX and DEP, you probably will see the same issue. And my guess if you would play with apparmor on Docker for Mac, you probably can run it without any issues.

And to workaround the issue, you can do the following, install execstack, and clear the flag with execstack -c $SPLUNK_HOME/lib/libcrypto.so.1.0.0 - after that you can start latest Splunk images just fine.

I do believe, that this is a security issue in Splunk distribution, and I am honestly surprised that it is not caught by now.

[dd-Splunk]

Yep for me it appeared right after the Sec update and as my compose always download the "latest" it just downloaded the 9.1.3 and I didn't notice the version change.

[kmahyyg]

Same problem here for Linux version when running splunk-dashboard-studio addon (I'm running 9.0.6 due to multiple reasons),

Created multiple symbolic links to solve:

$ pwd 
/usr/lib64
$ ls -alh
<SNIP>
lrwxrwxrwx  1 root root   28 Feb 25 09:42 libcrypto.so -> /opt/splunk/lib/libcrypto.so
lrwxrwxrwx  1 root root   34 Feb 25 09:42 libcrypto.so.1.0.0 -> /opt/splunk/lib/libcrypto.so.1.0.0
lrwxrwxrwx  1 root root   27 Feb 25 09:40 libexslt.so -> /opt/splunk/lib/libexslt.so
lrwxrwxrwx  1 root root   29 Feb 25 09:40 libexslt.so.0 -> /opt/splunk/lib/libexslt.so.0
lrwxrwxrwx  1 root root   25 Feb 25 09:41 libssl.so -> /opt/splunk/lib/libssl.so
lrwxrwxrwx  1 root root   31 Feb 25 09:41 libssl.so.1.0.0 -> /opt/splunk/lib/libssl.so.1.0.0
lrwxrwxrwx  1 root root   26 Feb 25 09:36 libxslt.so -> /opt/splunk/lib/libxslt.so
lrwxrwxrwx  1 root root   28 Feb 25 09:36 libxslt.so.1 -> /opt/splunk/lib/libxslt.so.1
<SNIP>

[adityapinglesf]

this command works perfectly fine for me on Sonoma mac with Intel chip. what machine are you using? also for @kmahyyg, are you facing the same error for your linux instance? can you share some more information about the instance? If that information is sensitive, maybe try raising a ticket with the splunk support team.

docker run -p 8000:8000 -e "SPLUNK_PASSWORD=mysupersecurepassword" -e "SPLUNK_START_ARGS=--accept-license"  -it --name so1 splunk/splunk:latest

[vl-federico]

this command works perfectly fine for me on Sonoma mac with Intel chip. what machine are you using? also for @kmahyyg, are you facing the same error for your linux instance? can you share some more information about the instance? If that information is sensitive, maybe try raising a ticket with the splunk support team.

docker run -p 8000:8000 -e "SPLUNK_PASSWORD=mysupersecurepassword" -e "SPLUNK_START_ARGS=--accept-license"  -it --name so1 splunk/splunk:latest

I'm using a 16-inch MacBook Pro (M1) 2021.

[dd-Splunk]

Mac with Intel chips are doing fine, Apple Silicon with Rosetta emulation encounters the issue.

[adityapinglesf]

ah! the emulations do not work. I am working on adding an arm compatible image. no ETA, that should resolve this situation.

[pich4ya]

I have found the workaround for Apple Silicon.

Once you initialize the docker run, run the following commands very quickly, at light speed.

docker exec -it splunk_free /bin/sh -c "sudo microdnf install dnf -y;sudo dnf install https://rpmfind.net/linux/centos/7.9.2009/os/x86_64/Packages/prelink-0.5.0-9.el7.x86_64.rpm -y;sudo execstack -c /opt/splunk/lib/libcrypto.so.1.0.0"

We will pass the problematic enable_admin_auth step. Next, if you encounter the restart/starting splunkd freezing, then you need to unset volume.

Everything works now on Docker container with Apple Silicon.