elasticsearch fails to start when tmp directory defined #18406

bubo77 · 2016-05-17T13:41:29Z

Elasticsearch version: 2.3.2

JVM version: java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

OS version: Centos 7 SELinux enabled

Description of the problem including expected versus actual behavior:
We have hardened Centos 7 where /tmp is not writeable
We have elasticsearch homedir in custom location and we link that custom location into /var/lib/elasticsearch.
We define tmp dir path in /etc/sysconfig/elasticserach as

Additional Java OPTS

ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" (we also tried without "" and with java.io.tmpdir=)

Steps to reproduce:

change tmpdir path in sysconfig
restart application

Provide logs (if relevant):
[2016-05-17 12:31:30,875][WARN ][bootstrap ] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna306419785920339930.tmp: /tmp/jna--1985354563/jna306419785920339930.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(Unknown Source) at java.lang.ClassLoader.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736) at com.sun.jna.Native.<clinit>(Native.java:131) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Unknown Source) at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:89) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
elasticsearch_dump.txt

The text was updated successfully, but these errors were encountered:

clintongormley · 2016-05-17T18:19:36Z

I tried setting the tmpdir with ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp" and it worked just fine.

We have elasticsearch homedir in custom location and we link that custom location into /var/lib/elasticsearch.

Do you mean symlinked? That's probably your problem.

bubo77 · 2016-05-17T18:28:06Z

Yeah i mean we have a symlink.
But interestingly it can create read/write the indicies but can't write tmp

ls -alZ /var/lib/elasticsearch/
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       .
drwxr-xr-x. root          root          unconfined_u:object_r:usr_t:s0   ..
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       graylog2
drwxr-xr-x. elasticsearch elasticsearch system_u:object_r:usr_t:s0       tmp

I have tried without a symlink fails the same way

Have you checked the kernel dump i've attached to the original post?

clintongormley · 2016-05-20T09:40:47Z

Looks like a duplicate of #18406

Closing

FlorianHeigl · 2016-11-18T01:53:10Z

You closed this with reference to itself. :-)

clintongormley · 2016-11-19T14:09:37Z

Whoops. Reopening.

@jasontedor any thoughts?

tlrx · 2016-11-21T09:32:36Z

This looks similar to #18272

FlorianHeigl · 2016-11-21T10:24:37Z

Thanks! sounds like the right thing. Not sure why google liked the duplicate better.
I think you can re-close with that reference!

In summary that bug says:

you cannot run ElasticSearch on a system that follows standard and basic security practices.
The issue is not in ES, but in the JNA thing (but obviously testing does not include systems that are secure)
The issue is related to the combination of JNA, noexec mounts and SELinux
It will not help to set a different ElasticSearch tmpdir since the problem isn't in ES itself.
There's no mention of a JNA issue for that. I'll try to find out about that. Expecting to end up in 10 year old stale code ;-)

FlorianHeigl · 2016-11-21T10:55:38Z

I think my last comment was lost. Sounds you found the right reference.
(Sounds also like a hopeless case right now)

clintongormley · 2016-11-21T10:57:08Z

Reclosing in favour of #18272

jasontedor · 2016-11-22T05:15:52Z

you cannot run ElasticSearch on a system that follows standard and basic security practices.

This is false. It's not clear exactly what the poor interaction is because no one can provide a clear reproduction, but it does run fine on some systems with SELinux set to enforcing and /tmp mounted with noexec.

The issue is not in ES, but in the JNA thing (but obviously testing does not include systems that are secure)

This is false. We have CI machines, VMs for our packaging tests, and my workstation (and likely other developer's workstations) that all run SELinux in enforcing mode.

jasontedor · 2016-11-22T05:16:36Z

(Sounds also like a hopeless case right now)

Yes, until someone can provide a 100% reliable reproduction.

Fabian1976 · 2017-02-27T10:28:35Z

It does seem to be related to the noexec mount options. I've had the same issue on 1 of our servers. When the noexec mount option is active on the /tmp directory, I get this error:

[2017-02-27T11:27:17,588][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna3776152293556606246.tmp: /tmp/jna--1985354563/jna3776152293556606246.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_111]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_111]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_111]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_111]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_111]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_111]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_111]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:203) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) [elasticsearch-5.2.1.jar:5.2.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) [elasticsearch-5.2.1.jar:5.2.1]

After that I removed the noexec mount option, and it works fine.
Setting de -Djna.tmpdir of -Djava.io.tmpdir to something different also didn't work on my server.

jcmcken · 2017-03-13T17:13:47Z

Just chiming in here to say we have the same problem. We use -Djna.tmpdir with other applications just fine. Only Elasticsearch seems not to work, requiring us to set exec on /tmp.

ES: elasticsearch-5.2.0-1.noarch (RPM distribution)
OS: CentOS Linux release 7.3.1611 (Core)
JVM:

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Fabian1976 · 2017-03-16T07:00:26Z

@jasontedor Reproduction was quite simple for me. Place noexec mount options on /tmp, installations fails. Even if a seperate tmp dir is defined (it is not respected or used). Remove noexec mount options, installation works fine.

tlrx · 2017-03-16T11:36:01Z

@Fabian1976 @jcmcken Can you please share more about your OS, versions of elasticsearch and java, /proc/mounts, /etc/fstab, use of tmpfs etc please? That would help.

@jcmcken Do the other applications you mentioned also use JNA? Also, I can't reproduce on a fresh CentOS installation (see above) with noexec.

$ cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)

$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

$ cat /proc/mounts | grep /tmp
tmpfs /tmp tmpfs rw,noexec 0 0

$ sudo rpm -i elasticsearch-5.2.0.rpm 
warning: elasticsearch-5.2.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

$ sudo systemctl start elasticsearch

$ curl -XPOST 'http://localhost:9200/books/book' -d '{"title":"foo"}'
{"_index":"books","_type":"book","_id":"AVrW3iFUg6TNDnTr-o09","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XGET 'http://localhost:9200/_search?q=foo'
{"took":59,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"books","_type":"book","_id":"AVrW3iFUg6TNDnTr-o09","_score":0.2876821,"_source":{"title":"foo"}}]}}

jasontedor · 2017-03-16T14:25:46Z

@Fabian1976 I know that this thread is quite long, but the stack trace that you provided is different than the one that triggered this issue. I think that we are looking at separate problems. It might be best if you open a separate issue.

FlorianHeigl · 2017-03-16T14:47:19Z

@tlrx did you look in your elasticsearch logs too or just run the tests?

Fabian1976 · 2017-03-17T10:50:03Z

@tlrx
My OS:

[root@elasticsearch01 ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 
[root@elasticsearch01 ~]#

Elasticsearch version:

[root@elasticsearch01 ~]# yum list installed | grep elasticsearch
elasticsearch.noarch       5.2.1-1               @CMC_Elastic_co_5_x            
[root@elasticsearch01 ~]#

Java version:

[root@elasticsearch01 ~]# java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)
[root@elasticsearch01 ~]#

Proc mounts:

[root@elasticsearch01 ~]# cat /proc/mounts | grep /tmp
/dev/mapper/VolGroup01-tmp /tmp ext4 rw,seclabel,nosuid,nodev,relatime,data=ordered 0 0
/dev/mapper/VolGroup01-tmp /var/tmp ext4 rw,seclabel,nosuid,nodev,relatime,data=ordered 0 0
[root@elasticsearch01 ~]#

FStab:

[root@elasticsearch01 ~]# cat /etc/fstab | grep /tmp
/dev/mapper/VolGroup01-tmp	/tmp	ext4	nodev,nosuid	0	2
/tmp	/var/tmp	ext4	bind,nodev,nosuid,noexec	0	0
[root@elasticsearch01 ~]#

jcmcken · 2017-03-17T17:29:35Z

@tlrx

Just to be clear, because I'm not sure we're talking about the same thing: ES launches just fine when /tmp is mounted noexec (which you've demonstrated). That's not my issue.

The issue is that it's falling back to NOT using JNA in this configuration, which you can see by examining the logs. But I would like to actually use whatever native extensions are available. And so I'm setting jna.tmpdir system prop to an alternate location on a filesystem that's mounted exec (e.g. ES_JAVA_OPTS="-Djna.tmpdir=/some/other/path" in /etc/sysconfig/elasticsearch). In this configuration, ES launches, and then dies a few seconds later.

Jenkins CI is the example that comes to mind where we're setting jna.tmpdir to an alternate location.

Balajiswz · 2017-04-04T12:45:05Z

I am also hitting the same error when we try to define separate TMP folder for elasticsearch with similar configurations.

c4tom · 2017-05-09T12:28:42Z

Try this:

The problem can be solved download jars, and copy to elasticsearch lib folder

https://github.com/java-native-access/jna

Version 4.4.0
Maven Central jna.jar
Maven Central jna-platform.jar

jasontedor · 2017-05-09T13:06:56Z

@candido1212 Elasticsearch 5.4.0 ships with JNA 4.4.0 now. Dropping jars into the lib folder is fully unsupported.

FlorianHeigl · 2017-05-09T13:19:52Z

@candido1212 were you doing that on ES 5.4.0?
If dropping in the same version did in fact change the behaviour that is at least interesting information.

AverageJoe2 · 2017-05-27T02:33:45Z

@jasontedor
hi buddy how you make it with version Elasticsearch 5.4.0 now ?

Workaround for these 3 bugs: elastic/elasticsearch#18406 elastic/elasticsearch#18272 https://bugs.openjdk.java.net/browse/JDK-8135224

yami12376 · 2017-06-19T10:09:30Z

I have the same issue - tried what candido suggested but without success.
(CentOS 5.2 (Final) x64) - i think i have to update to CentOs 6
ES 5.4.1
java version "1.8.0_112"
Installed and started as posted in https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html

 cat /proc/mounts | grep /tmp
<empty>

cat /etc/fstab | grep /tmp
<empty>

sudo service elasticsearch start
Starting elasticsearch:                                    [  OK  ]

after that:

starts elasticsearch

Option                Description
------                -----------
-E <KeyValuePair>     Configure a setting
-V, --version         Prints elasticsearch version information and exits
-d, --daemonize       Starts Elasticsearch in the background
-h, --help            show help
-p, --pidfile <Path>  Creates a pid file in the specified path on start
-q, --quiet           Turns off standard ouput/error streams logging in console
-s, --silent          show minimal output
-v, --verbose         show verbose output
ERROR: D is not a recognized option


[2017-06-19T12:04:30,595][WARN ][o.e.b.Natives            ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna8670561425318125531.tmp: /lib/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/jna-3506402/jna8670561425318125531.tmp)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_112]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_112]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_112]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_112]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_112]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.4.0.jar:4.4.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_112]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_112]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.1.jar:5.4.1]
[2017-06-19T12:04:30,604][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2017-06-19T12:04:30,604][WARN ][o.e.b.Natives            ] cannot install system call filter because JNA is not available
[2017-06-19T12:04:30,605][WARN ][o.e.b.Natives            ] cannot register console handler because JNA is not available
[2017-06-19T12:04:30,607][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2017-06-19T12:04:30,608][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2017-06-19T12:04:30,802][INFO ][o.e.n.Node               ] [] initializing ...
[2017-06-19T12:04:30,924][INFO ][o.e.e.NodeEnvironment    ] [c51pvG3] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup00-LogVol00)]], net usable_space [12.4gb], net total_space [31.1gb], spins? [possibly], types [ext3]
[2017-06-19T12:04:30,924][INFO ][o.e.e.NodeEnvironment    ] [c51pvG3] heap size [1.9gb], compressed ordinary object pointers [unknown]
[2017-06-19T12:04:30,934][INFO ][o.e.n.Node               ] node name [c51pvG3] derived from node ID [c51pvG3UT62JdbAwGWOJuQ]; set [node.name] to override
[2017-06-19T12:04:30,935][INFO ][o.e.n.Node               ] version[5.4.1], pid[21917], build[2cfe0df/2017-05-29T16:05:51.443Z], OS[Linux/2.6.18-92.el5/i386], JVM[Oracle Corporation/Java HotSpot(TM) Server VM/1.8.0_112/25.112-b15]
[2017-06-19T12:04:30,935][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/install/elasticsearch-5.4.1]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [aggs-matrix-stats]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [ingest-common]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-expression]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-groovy]
[2017-06-19T12:04:32,221][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-mustache]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [lang-painless]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [percolator]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [reindex]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [transport-netty3]
[2017-06-19T12:04:32,222][INFO ][o.e.p.PluginsService     ] [c51pvG3] loaded module [transport-netty4]
[2017-06-19T12:04:32,223][INFO ][o.e.p.PluginsService     ] [c51pvG3] no plugins loaded
[2017-06-19T12:04:34,468][INFO ][o.e.d.DiscoveryModule    ] [c51pvG3] using discovery type [zen]
[2017-06-19T12:04:35,703][INFO ][o.e.n.Node               ] initialized
[2017-06-19T12:04:35,703][INFO ][o.e.n.Node               ] [c51pvG3] starting ...
[2017-06-19T12:04:37,077][INFO ][o.e.t.TransportService   ] [c51pvG3] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-06-19T12:04:37,140][WARN ][o.e.b.BootstrapChecks    ] [c51pvG3] max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2017-06-19T12:04:37,140][WARN ][o.e.b.BootstrapChecks    ] [c51pvG3] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-06-19T12:04:40,334][INFO ][o.e.c.s.ClusterService   ] [c51pvG3] new_master {c51pvG3}{c51pvG3UT62JdbAwGWOJuQ}{Xk4Yetg-Q3etoZ3YGo6Z5Q}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-06-19T12:04:40,486][INFO ][o.e.h.n.Netty4HttpServerTransport] [c51pvG3] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-06-19T12:04:40,490][INFO ][o.e.n.Node               ] [c51pvG3] started
[2017-06-19T12:04:40,614][INFO ][o.e.g.GatewayService     ] [c51pvG3] recovered [1] indices into cluster_state
[2017-06-19T12:04:40,931][INFO ][o.e.c.r.a.AllocationService] [c51pvG3] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

jasontedor · 2017-06-19T15:32:04Z

@yami12376 As I mentioned, dropping jars in the lib folder is fully unsupported. The problem that you're running to here is that the official JNA jar from upstream does not support older distributions that do not support newer versions of glibc. This is why we ship with our own build of JNA that does support some of the older distributions that we still support. The only difference between our build and upstream is that the Linux native library is recompiled to support OS that are using older versions of glibc, everything else is identical to upstream so dropping a different version of the jar in the lib folder is not going to help with anything to begin with.

Please, stop following the advice to drop a different build into the lib folder. It is not a solution to the problem here, and it is fully unsupported.

yami12376 · 2017-06-23T12:50:10Z

I updated to CentOs 7 - and it works fine

thosfelt · 2017-07-07T19:42:42Z

I'm currently running ES 2.4.0 running on CentOS 6.6 (PPC) and am haveing some of the same issues after our SA's have done some upgrades to the systems. I can't get past the "because JNA is not available" and ES won't use ES_JAVA_OPTS to point to a different jna.tmpdir. Also /tmp doesn't have the noexec option turned on and still won't start up after the system upgrade.

chunji08 · 2017-09-14T05:47:54Z

Do we have a solution to this issue. I am using ES ( version 5.6) now and I have hit the same issue. Same platform, if I was using my previous installed ES ( version 2.4), I don't have this problem. Thanks.

bschneiders · 2017-10-24T15:01:44Z

I encountered this issue on 5.3.3-1 on RHEL 7.4 when relocating multiple ES directories. I was modifying jvm.options to add -Djava.io.tmpdir=... with a custom directory. I eventually solved it when I realized that the system still uses /etc/elasticsearch/jvm.options on RHEL7 even if you relocate default.path.conf. I was modifying copies of jvm.options in my new configuration directories at first.

jasontedor · 2017-10-24T15:54:00Z

I eventually solved it when I realized that the system still uses /etc/elasticsearch/jvm.options on RHEL7 even if you relocate default.path.conf. I was modifying copies of jvm.options in my new configuration directories at first.

In 5.x you can set ES_JVM_OPTIONS to point to a custom jvm.options file. In 6.x we will do this automatically based on the value of ES_PATH_CONF (the only way to customize the configuration path location there).

beaudet · 2017-11-14T13:54:29Z

I got around this in a Spring Boot application running on RHEL6 by setting -Djava.io.tmpdir to an alternative temporary folder with exec permissions in the startup script for the spring boot application that bundles ES.

scais · 2018-04-21T08:35:53Z

Got the same problem on W10, running as admin solved that - https://stackoverflow.com/a/49953880/2116237

neilneyman · 2018-05-18T16:57:27Z

Wanted to note one thing on RHEL 7:

We use a secure configuration that mounts /tmp as noexec

With SELinux in "Permissive" mode, ES will crash unless I set bootstrap.system_call_filter: false, even if I change the ES_TMPDIR location to a directory with exec perms.

Looks like the only way I could get it running was to either completely disable SELinux (not permissive) and change the tmp directory. Or set bootstrap.system_call_filter: false.

0huah0 · 2018-06-03T18:19:25Z

-Djava.io.tmpdir= does not work on ubuntu16

beaudet · 2018-06-03T22:36:03Z

Is the Ubuntu tmpdir behavior with a specific Java distribution like openjdk or all of them?

0x0badc0de · 2019-02-07T20:05:55Z

Got the same issue. Simplest way to get the exception:

docker run -e "discovery.type=single-node" --tmpfs /tmp elasticsearch:6.5.4

This way everything is fine:

docker run -e "discovery.type=single-node" elasticsearch:6.5.4

beaudet · 2019-02-07T20:14:25Z

Yeah, so I would wager that this is all part of the same issue with permissions on the tmpfs folder being used. It would be nice for for ES to log this condition with much more verbosity and specificity to in order to cut down on the time spent by admins in diagnosing this problem.

jasontedor · 2019-02-10T14:56:44Z

@0x0badc0de That is not a reproduction of the issue at hand here. With --tmpfs /tmp, the resulting mount will be mounted with the noexec flag. That flag is not compatible with JNA as JNA needs to map the native libraries as executable. Note that the initial report here is about mounting /tmp with noexec yet specifying a custom jna.tmpdir for JNA (so that it doesn't use /tmp). In this case, if you do this, all is well:

09:52:17 [jason@totoro:~] 130 $ docker run --rm -it -p 9200:9200 -p 9300:9300 -e discovery.type=single-node -e ES_JAVA_OPTS=-Djna.tmpdir=/usr/share/elasticsearch/data --tmpfs /tmp elasticsearch:6.6.0 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-02-10T14:53:18,190][INFO ][o.e.e.NodeEnvironment    ] [rPuFwlQ] using [1] data paths, mounts [[/ (overlay)]], net usable_space [25.7gb], net total_space [48.9gb], types [overlay]
[2019-02-10T14:53:18,193][INFO ][o.e.e.NodeEnvironment    ] [rPuFwlQ] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-02-10T14:53:18,194][INFO ][o.e.n.Node               ] [rPuFwlQ] node name derived from node ID [rPuFwlQEQJSem6TeJ7sdDQ]; set [node.name] to override
[2019-02-10T14:53:18,195][INFO ][o.e.n.Node               ] [rPuFwlQ] version[6.6.0], pid[1], build[default/tar/a9861f4/2019-01-24T11:27:09.439740Z], OS[Linux/4.20.6-200.fc29.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-02-10T14:53:18,195][INFO ][o.e.n.Node               ] [rPuFwlQ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14411002066878058815, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Djna.tmpdir=/usr/share/elasticsearch/data, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [aggs-matrix-stats]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [analysis-common]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [ingest-common]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-expression]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-mustache]
[2019-02-10T14:53:19,288][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [lang-painless]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [mapper-extras]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [parent-join]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [percolator]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [rank-eval]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [reindex]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [repository-url]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [transport-netty4]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [tribe]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ccr]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-core]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-deprecation]
[2019-02-10T14:53:19,289][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-graph]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ilm]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-logstash]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-ml]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-monitoring]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-rollup]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-security]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-sql]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-upgrade]
[2019-02-10T14:53:19,290][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded module [x-pack-watcher]
[2019-02-10T14:53:19,291][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded plugin [ingest-geoip]
[2019-02-10T14:53:19,291][INFO ][o.e.p.PluginsService     ] [rPuFwlQ] loaded plugin [ingest-user-agent]
[2019-02-10T14:53:21,688][INFO ][o.e.x.s.a.s.FileRolesStore] [rPuFwlQ] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2019-02-10T14:53:22,026][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [rPuFwlQ] [controller/98] [Main.cc@109] controller (64 bit): Version 6.6.0 (Build bbb4919f4d17a5) Copyright (c) 2019 Elasticsearch BV
[2019-02-10T14:53:22,456][INFO ][o.e.d.DiscoveryModule    ] [rPuFwlQ] using discovery type [single-node] and host providers [settings]
[2019-02-10T14:53:22,984][INFO ][o.e.n.Node               ] [rPuFwlQ] initialized
[2019-02-10T14:53:22,984][INFO ][o.e.n.Node               ] [rPuFwlQ] starting ...
[2019-02-10T14:53:23,105][INFO ][o.e.t.TransportService   ] [rPuFwlQ] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2019-02-10T14:53:23,120][WARN ][o.e.b.BootstrapChecks    ] [rPuFwlQ] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-02-10T14:53:23,139][INFO ][o.e.h.n.Netty4HttpServerTransport] [rPuFwlQ] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2019-02-10T14:53:23,139][INFO ][o.e.n.Node               ] [rPuFwlQ] started
[2019-02-10T14:53:23,238][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [rPuFwlQ] Failed to clear cache for realms [[]]
[2019-02-10T14:53:23,277][INFO ][o.e.g.GatewayService     ] [rPuFwlQ] recovered [0] indices into cluster_state
[2019-02-10T14:53:23,369][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.watches] for index patterns [.watches*]
[2019-02-10T14:53:23,427][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-02-10T14:53:23,458][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-02-10T14:53:23,495][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2019-02-10T14:53:23,542][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2019-02-10T14:53:23,571][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2019-02-10T14:53:23,605][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2019-02-10T14:53:23,637][INFO ][o.e.c.m.MetaDataIndexTemplateService] [rPuFwlQ] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2019-02-10T14:53:23,724][INFO ][o.e.l.LicenseService     ] [rPuFwlQ] license [fba30e1b-d535-4eee-9b5d-2205bb1867c6] mode [basic] - valid

At the end of the day, for this issue, I don't think Elasticsearch was actually picking up the options the OP thinks they specified. This can be seen in the error message:

[2016-05-17 12:31:30,875][WARN ][bootstrap ] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna306419785920339930.tmp: /tmp/jna--1985354563/jna306419785920339930.tmp: failed to map segment from shared object: Operation not permitted

Note how it mentions /tmp even though the user thinks they specified a custom tmpdir in /var/lib/elasticsearch/tmp.

Today we log all JVM options used to start Elasticsearch, so today it would be easier to verify whether or not that is the case.

jasontedor · 2019-02-10T14:57:58Z

If someone has a reproduction of any issues related to JNA please open a new issue instead of continuing on this long and old discussion. It will be easier to start fresh with new reports.

FlorianHeigl · 2019-02-10T15:44:35Z

@jasontedor:
This discussion is just so long and old because noone before you actually made the effort to accept this issue existed - from there to a perfect explanation that everyone can understand and act on it was less than an hour.
Thank you for that.

amancipe · 2019-08-17T18:21:36Z

This actually does happens and it can be reproduced on RHEL6.10 running Elasticsearch7.1.1 with /tmp mounted with noexec and ES_TMPDIR=/var/lib/elasticsearch/tmp on /etc/sysconfig/elasticsearch. If I remont /tmp without noexec it all work as intended. However that is a STIG requirement that cannot be omitted.

jasontedor · 2019-08-18T17:23:16Z

@amancipe That does not reproduce for me.

centos-6:~$ mount | grep "/tmp"
tmpfs on /tmp type tmpfs (rw,noexec,nosuid,nodev)
centos-6:~$ curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.1-x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  330M  100  330M    0     0  73.5M      0  0:00:04  0:00:04 --:--:-- 95.7M
centos-6:~$ sudo rpm -i elasticsearch-7.1.1-x86_64.rpm 
warning: elasticsearch-7.1.1-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
 sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
 sudo service elasticsearch start
Created elasticsearch keystore in /etc/elasticsearch
centos-6:~$ sudo bash -c "echo ES_TMPDIR=/var/lib/elasticsearch/tmp >> /etc/sysconfig/elasticsearch"
centos-6:~$ sudo mkdir -p /var/lib/elasticsearch/tmp
centos-6:~$ sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/tmp
centos-6:~$ sudo service elasticsearch start
Starting elasticsearch: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
                                                           [  OK  ]
centos-6:~$ sudo cat /var/log/elasticsearch/elasticsearch.log
[2019-08-18T17:19:51,416][WARN ][o.e.b.JNANatives         ] [localhost.localdomain] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
	at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:107) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
[2019-08-18T17:19:51,635][INFO ][o.e.e.NodeEnvironment    ] [localhost.localdomain] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [13.7gb], net total_space [17.1gb], types [rootfs]
[2019-08-18T17:19:51,638][INFO ][o.e.e.NodeEnvironment    ] [localhost.localdomain] heap size [990.7mb], compressed ordinary object pointers [true]
[2019-08-18T17:19:51,640][INFO ][o.e.n.Node               ] [localhost.localdomain] node name [localhost.localdomain], node ID [AvNItga_RQmkqzSB5ahkcg], cluster name [elasticsearch]
[2019-08-18T17:19:51,640][INFO ][o.e.n.Node               ] [localhost.localdomain] version[7.1.1], pid[6485], build[default/rpm/7a013de/2019-05-23T14:04:00.380842Z], OS[Linux/2.6.32-754.17.1.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-08-18T17:19:51,641][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM home [/usr/share/elasticsearch/jdk]
[2019-08-18T17:19:51,641][INFO ][o.e.n.Node               ] [localhost.localdomain] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/lib/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [aggs-matrix-stats]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [analysis-common]
[2019-08-18T17:19:52,854][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-common]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-geoip]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [ingest-user-agent]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-expression]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-mustache]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [lang-painless]
[2019-08-18T17:19:52,855][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [mapper-extras]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [parent-join]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [percolator]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [rank-eval]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [reindex]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [repository-url]
[2019-08-18T17:19:52,856][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [transport-netty4]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ccr]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-core]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-deprecation]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-graph]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ilm]
[2019-08-18T17:19:52,857][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-logstash]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-ml]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-monitoring]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-rollup]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-security]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-sql]
[2019-08-18T17:19:52,858][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] loaded module [x-pack-watcher]
[2019-08-18T17:19:52,859][INFO ][o.e.p.PluginsService     ] [localhost.localdomain] no plugins loaded
[2019-08-18T17:19:56,113][INFO ][o.e.x.s.a.s.FileRolesStore] [localhost.localdomain] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2019-08-18T17:19:56,588][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [localhost.localdomain] [controller/6501] [Main.cc@109] controller (64 bit): Version 7.1.1 (Build fd619a36eb77df) Copyright (c) 2019 Elasticsearch BV
[2019-08-18T17:19:56,927][DEBUG][o.e.a.ActionModule       ] [localhost.localdomain] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-08-18T17:19:57,316][INFO ][o.e.d.DiscoveryModule    ] [localhost.localdomain] using discovery type [zen] and seed hosts providers [settings]
[2019-08-18T17:19:57,950][INFO ][o.e.n.Node               ] [localhost.localdomain] initialized
[2019-08-18T17:19:57,950][INFO ][o.e.n.Node               ] [localhost.localdomain] starting ...
[2019-08-18T17:19:58,069][INFO ][o.e.t.TransportService   ] [localhost.localdomain] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2019-08-18T17:19:58,110][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2019-08-18T17:19:58,110][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2019-08-18T17:19:58,111][WARN ][o.e.b.BootstrapChecks    ] [localhost.localdomain] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-08-18T17:19:58,115][INFO ][o.e.c.c.Coordinator      ] [localhost.localdomain] cluster UUID [UmAFChzKQdCn5TO9II8SBg]
[2019-08-18T17:19:58,122][INFO ][o.e.c.c.ClusterBootstrapService] [localhost.localdomain] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2019-08-18T17:19:58,294][INFO ][o.e.c.s.MasterService    ] [localhost.localdomain] elected-as-master ([1] nodes joined)[{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 16, reason: master node changed {previous [], current [{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-08-18T17:19:58,393][INFO ][o.e.c.s.ClusterApplierService] [localhost.localdomain] master node changed {previous [], current [{localhost.localdomain}{AvNItga_RQmkqzSB5ahkcg}{zMUBg8_1QvanFW-k4Qzbbg}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=8387620864, xpack.installed=true, ml.max_open_jobs=20}]}, term: 2, version: 16, reason: Publication{term=2, version=16}
[2019-08-18T17:19:58,479][INFO ][o.e.h.AbstractHttpServerTransport] [localhost.localdomain] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-08-18T17:19:58,480][INFO ][o.e.n.Node               ] [localhost.localdomain] started
[2019-08-18T17:19:58,587][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [localhost.localdomain] Failed to clear cache for realms [[]]
[2019-08-18T17:19:58,620][INFO ][o.e.l.LicenseService     ] [localhost.localdomain] license [db85f7c5-f3d4-414b-aecb-c87f965b8579] mode [basic] - valid
[2019-08-18T17:19:58,628][INFO ][o.e.g.GatewayService     ] [localhost.localdomain] recovered [0] indices into cluster_state
centos-6:~$

pro4soft1 · 2021-08-12T12:53:23Z

its Work For Me After add
-Djava.io.tmpdir=/var/lib/elasticsearch/tmp
in : /etc/elasticsearch/jvm.options

spricer · 2021-11-10T17:38:01Z

It seems that -Djava.io.tmpdir alone is not enough to allow elasticsearch startup.
I analyzed running tasks of elasticsearch when /tmp is mounted with noexec removed and there is additional path required for elasticsearch to work.
This variable is called TMPDIR.
So in order to make it work the best way is to append next in /etc/sysconfig/elasticsearch file:

ES_TMPDIR=/var/lib/elasticsearch/tmp
TMPDIR=$ES_TMPDIR

If you have issue with above path (e.g. also mounted with noexec flag) you can introduce new path that is not under
noexec flag:

mkdir /opt/tmp
chown elasticsearch:elasticsearch /opt/tmp
chmod 755 /opt/tmp

Then update /etc/sysconfig/elasticsearch file accordingly:

ES_TMPDIR=/opt/tmp
TMPDIR=$ES_TMPDIR

I think developers need to hardcode somewhere TMPDIR=$ES_TMPDIR

droberts195 · 2021-11-10T17:44:39Z

@spricer this issue is 5 years old and closed. A better issue to follow is #77014, which is also about noexec problems and is being actively worked on.

spricer · 2021-11-10T17:52:39Z

This is what google returned to me and by going through comments it did not looked resolved (people actively commented after was closed since 2016). Nevertheless thanks for update and I submitted my findings at latest issue #.

bubo77 mentioned this issue May 17, 2016

Elasticsearch 2.2 fails to start with jna tmp dir configured (core dump) #18272

Closed

clintongormley added feedback_needed :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts labels May 17, 2016

clintongormley closed this as completed May 20, 2016

clintongormley reopened this Nov 19, 2016

clintongormley closed this as completed Nov 21, 2016

jasontedor mentioned this issue Jan 10, 2017

Elasticsearch fails to start with selinux enforcing #22493

Closed

bcdonadio added a commit to stone-payments/ansible-elasticsearch that referenced this issue Jun 13, 2017

Use JNA temp folder on /var/lib

82b9948

Workaround for these 3 bugs: elastic/elasticsearch#18406 elastic/elasticsearch#18272 https://bugs.openjdk.java.net/browse/JDK-8135224

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

elasticsearch fails to start when tmp directory defined #18406

elasticsearch fails to start when tmp directory defined #18406

Comments

bubo77 commented May 17, 2016 • edited

Additional Java OPTS

clintongormley commented May 17, 2016

bubo77 commented May 17, 2016 • edited

clintongormley commented May 20, 2016

FlorianHeigl commented Nov 18, 2016

clintongormley commented Nov 19, 2016

tlrx commented Nov 21, 2016

FlorianHeigl commented Nov 21, 2016

FlorianHeigl commented Nov 21, 2016

clintongormley commented Nov 21, 2016

jasontedor commented Nov 22, 2016 • edited

jasontedor commented Nov 22, 2016

Fabian1976 commented Feb 27, 2017 • edited

jcmcken commented Mar 13, 2017

Fabian1976 commented Mar 16, 2017

tlrx commented Mar 16, 2017

jasontedor commented Mar 16, 2017 • edited

FlorianHeigl commented Mar 16, 2017

Fabian1976 commented Mar 17, 2017

jcmcken commented Mar 17, 2017 • edited

Balajiswz commented Apr 4, 2017

c4tom commented May 9, 2017

jasontedor commented May 9, 2017

FlorianHeigl commented May 9, 2017

AverageJoe2 commented May 27, 2017

yami12376 commented Jun 19, 2017 • edited

jasontedor commented Jun 19, 2017

yami12376 commented Jun 23, 2017

thosfelt commented Jul 7, 2017

chunji08 commented Sep 14, 2017

bschneiders commented Oct 24, 2017

jasontedor commented Oct 24, 2017

beaudet commented Nov 14, 2017

scais commented Apr 21, 2018

neilneyman commented May 18, 2018

0huah0 commented Jun 3, 2018

beaudet commented Jun 3, 2018

0x0badc0de commented Feb 7, 2019

beaudet commented Feb 7, 2019

jasontedor commented Feb 10, 2019

jasontedor commented Feb 10, 2019

FlorianHeigl commented Feb 10, 2019

amancipe commented Aug 17, 2019

jasontedor commented Aug 18, 2019

pro4soft1 commented Aug 12, 2021

spricer commented Nov 10, 2021

droberts195 commented Nov 10, 2021

spricer commented Nov 10, 2021 • edited

bubo77 commented May 17, 2016 •

edited

bubo77 commented May 17, 2016 •

edited

jasontedor commented Nov 22, 2016 •

edited

Fabian1976 commented Feb 27, 2017 •

edited

jasontedor commented Mar 16, 2017 •

edited

jcmcken commented Mar 17, 2017 •

edited

yami12376 commented Jun 19, 2017 •

edited

spricer commented Nov 10, 2021 •

edited