Log4j vulnerability #964

mattecasu · 2022-01-10T17:33:41Z

Hello dear b2i,
do you plan to release dockerised patched versions for the log4j "0-day" vulnerability [containing the latest log4j version 2.17.1, or alternatively none at all]? Or, is there a mitigation that you suggest, perhaps one of these? https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide .
Thank you

cmark · 2022-01-10T18:09:52Z

Hi @mattecasu,

Snow Owl is unaffected of this vulnerability (it uses a different logging library than log4j), but Elasticsearch as storage and search backend is somewhat vulnerable, RCE is not possible, but information leakage in certain cases yes, please refer to this site: https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/.
Whether you are running your Snow Owl instance in embedded mode or connecting it to an external Elasticsearch cluster, it is recommended to set the -Dlog4j2.formatMsgNoLookups=true command-line argument for both Snow Owl (just in case) and Elasticsearch and restart the service(s). This eliminates the possibility of information leakage and mitigates the issue while we are in the process of releasing an official patch.

All three dev streams (6.x, 7.x and 8.x) have received their respective patches already back in December before the holidays, but due to other issues (see #958) with the latest 7.16.2 Elasticsearch we couldn't release the 7.x and 8.x streams that time, only the 6.x patch. Also, it is very likely that another Elasticsearch patch (7.16.3 that contains log4j 2.17.1) will be released soon and we have decided to wait for it before we perform an actual release.

The currently planned release date for Snow Owl 7.18.1 and 8.1.0 is the 21st of January.

Cheers,
Mark

cmark · 2022-01-27T15:59:35Z

Hi @mattecasu,

Snow Owl 7.18.1 is out, which depends on Elasticsearch 7.16.3 and resolves any potential security vulnerability around log4j2.
Please find the docker image here.
If you have any further questions or issues feel free to contact us.

Cheers,
Mark

mattecasu · 2022-01-31T16:03:25Z

Dear Mark,

thanks for the patched tags - we are having an issue when using 7.18.1, I suspect it's not compatible with the latest AWS ElasticSearch clusters version (that is on 7.10). It seems like it cannot contact the cluster or find the indexes ("cluster not available" and also Couldn't check the existence of ES index 'locks-lock' ).
[Are you aware of this? Or maybe we are not considering something]

By the way: I confirmed 7.18.0 instead worked fine (which is on ES 7.10)

cmark · 2022-01-31T16:23:51Z

Hi @mattecasu,

Do you have the full stack trace?
It could be a compatibility issue between the Elasticsearch 7.16.3 client and the AWS Elasticsearch 7.10 service.

Cheers,
Mark

aelred · 2022-01-31T16:36:24Z

Hi @cmark, here's the full stacktrace we're getting:

[2022-01-31 16:28:06.733] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] INFO  elastic-snowowl - Connecting to Elasticsearch cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com', connect timeout: 2000 ms, socket timeout: 120000 ms.
[2022-01-31 16:28:07.222] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] INFO  index.locks - Preparing 'locks' indexes...
[2022-01-31 16:29:07.275] [Start Level: Equinox Container: 7e07ab0e-607f-4e6e-9464-35eeabc71548] ERROR snowowl - Couldn't check the existence of ES index 'locks-lock'.
com.b2international.index.IndexException: Couldn't check the existence of ES index 'locks-lock'.
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:150)
	at com.b2international.index.es.admin.EsIndexAdmin.create(EsIndexAdmin.java:171)
	at com.b2international.snowowl.core.locks.DefaultOperationLockManager.<init>(DefaultOperationLockManager.java:85)
	at com.b2international.snowowl.core.internal.locks.LockPlugin.preRun(LockPlugin.java:43)
	at com.b2international.snowowl.core.setup.Plugins.preRun(Plugins.java:88)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:291)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:272)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:260)
	at com.b2international.snowowl.server.product.SnowOwlServerActivator.start(SnowOwlServerActivator.java:52)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)
	at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
	at org.eclipse.osgi.container.Module.doStart(Module.java:605)
	at org.eclipse.osgi.container.Module.start(Module.java:468)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1845)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1838)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1781)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1743)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1665)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: com.b2international.commons.exceptions.BadRequestException: Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available.
	at com.b2international.index.es.client.EsClientBase.checkAvailable(EsClientBase.java:101)
	at com.b2international.index.es.client.http.IndicesHttpClient.exists(IndicesHttpClient.java:60)
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:148)
	... 26 common frames omitted

(I slightly redacted our Elasticsearch URL)

cmark · 2022-01-31T17:18:09Z

Hi @aelred,

Could you please try the latest 7.x image from Dockerhub (it should be available at any minute now)? That has a change that will print out the cluster diagnostic message next to the Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available. error, so we can see what is going on under the hood and hopefully get a fix for it quickly.

Thank you,
Mark

aelred · 2022-01-31T17:33:09Z

Thanks Mark, we'll give it a try when it's pushed up!

It looks like the build failed though:

->  Pinging Codecov
https://codecov.io/upload/v4?package=github-action-1.0.6&token=<hidden>&package=github-action-1.0.6&token=&branch=8.x&commit=f1bc20cf61320f7162c7bb20853e5915e33fc699&build=1773785924&build_url=https%3A%2F%2Fgithub.com%2Fb2ihealthcare%2Fsnow-owl%2Factions%2Fruns%2F1773785924&name=&tag=&slug=b2ihealthcare%2Fsnow-owl&service=github-actions&flags=&pr=&job=Java%20CI&cmd_args=n,F,Q,Z
{'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
404
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  575k  100   171  100  575k   1195  4025k --:--:-- --:--:-- --:--:-- 4027k
    {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}
Error: Codecov failed with the following error: The process '/usr/bin/bash' failed with exit code 1

https://github.com/b2ihealthcare/snow-owl/runs/5009331342?check_suite_focus=true

cmark · 2022-01-31T17:40:59Z

Yeah, that's the 8.x build (Codecov sometimes misbehaves, could not figure out why so far). The 7.x builds are running on our internal CI servers and they have been completed 30 minutes now. Dockerhub refreshes itself quite slowly, I recommend pulling the image anyway via docker pull b2ihealthcare/snow-owl-oss:7.x and checking its image ID. It should be f0a33069ff08.

I hope this helps.

Cheers,
Mark

aelred · 2022-01-31T18:21:21Z

Oh you're right! Sorry, I was looking at the wrong thing.

With the new 7.x we get: Invalid or missing build flavor [oss].

Here's the full stack trace:

[2022-01-31 18:18:52.724] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] INFO  elastic-snowowl - Connecting to Elasticsearch cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com', connect timeout: 2000 ms, socket timeout: 120000 ms.
[2022-01-31 18:18:53.030] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] INFO  index.locks - Preparing 'locks' indexes...
[2022-01-31 18:19:53.041] [Start Level: Equinox Container: 70218979-5f75-4858-810c-033d1cf7d27d] ERROR snowowl - Couldn't check the existence of ES index 'locks-lock'.
com.b2international.index.IndexException: Couldn't check the existence of ES index 'locks-lock'.
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:150)
	at com.b2international.index.es.admin.EsIndexAdmin.create(EsIndexAdmin.java:171)
	at com.b2international.snowowl.core.locks.DefaultOperationLockManager.<init>(DefaultOperationLockManager.java:85)
	at com.b2international.snowowl.core.internal.locks.LockPlugin.preRun(LockPlugin.java:43)
	at com.b2international.snowowl.core.setup.Plugins.preRun(Plugins.java:88)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:291)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:272)
	at com.b2international.snowowl.core.SnowOwl.run(SnowOwl.java:260)
	at com.b2international.snowowl.server.product.SnowOwlServerActivator.start(SnowOwlServerActivator.java:52)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)
	at org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)
	at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
	at org.eclipse.osgi.container.Module.doStart(Module.java:605)
	at org.eclipse.osgi.container.Module.start(Module.java:468)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1845)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1838)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1781)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1743)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1665)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: com.b2international.commons.exceptions.BadRequestException: Cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' is not available. Diagnosis: 'The cluster at 'https://our-es-url.eu-west-1.es.amazonaws.com' reported an error: 'Invalid or missing build flavor [oss]''
	at com.b2international.index.es.client.EsClientBase.checkAvailable(EsClientBase.java:102)
	at com.b2international.index.es.client.http.IndicesHttpClient.exists(IndicesHttpClient.java:60)
	at com.b2international.index.es.admin.EsIndexAdmin.exists(EsIndexAdmin.java:148)
	... 26 common frames omitted

mattecasu · 2022-02-01T10:50:11Z

Hello Mark,
we think it's probably this? elastic/elasticsearch#76091
The thing is that the AWS-provided ES is probably used by many (including us 😅), so it would be best for Snowowl to make sure the ES client adopted can query it (hopefully AWS will soon bump their version..).

As for log4j, one way is to:

exclude any old log4j group
include log4j-1.2-api (translates versione 1 to version 2)
optionally include a bridge to something else, like log4j-to-slf4j

cmark · 2022-02-01T17:28:55Z

Hi @mattecasu,

Interesting, according to the official Elasticsearch docs, it should be backward compatible with the older 7.x versions, but apparently that's not true.
Elastic has changed Elasticsearch's licensing from Apache license starting from 7.11.0, none of the other Elasticsarch hosting providers will ever going to upgrade their version above 7.10.2 due to that, because the new license forbids hosting like that. AWS also forked the project and created OpenSearch as an alternative, which is interesting and sad at the same time.

Anyway, we are going to debug this and hopefully get a fix for it sometime soon, in the meantime I recommend a rollback to 7.18.0 version which is able to communicate with your ES cluster without an issue and configure the mentioned -Dlog4j2.formatMsgNoLookups=true to mitigate the security vulnerability around log4j.

I hope this helps, if you have any questions please let us know.

Cheers,
Mark

aelred · 2022-02-02T10:54:59Z

Thanks Mark, we'll apply the configuration for now!

mattecasu · 2022-02-14T15:52:51Z

Hello again Mark,

did you find a way to make the patched versions compatible with AWS's ElasticSearch? Just asking to understand if you have it on the roadmap.

cmark · 2022-02-15T14:56:34Z

Hi @mattecasu,

Not yet, we could fix it by altering the Elasticsearch source somehow but that could easily turn into a licensing issue which we would like to avoid. Feel free to open another ticket that tracks the compatibility of Snow Owl with the AWS Elasticsearch service, which only fixes the client-side part of the Log4j2 vulnerability, the AWS Elasticsearch service will still offer Elasticsearch 7.10.2, which is somewhat vulnerable to information leakage attacks (but not to RCE attacks).

I still recommend either switching to Elastic Cloud as Elasticsearch hosting solution or hosting the Elasticsearch service on your own. Please refer to the AWS documentation to see the supported Elasticsearch versions (up to Elasticsearch OSS 7.10).

I hope this helps. Feel free to reach out to us if you have any further questions.

Cheers,
Mark

mattecasu added the question label Jan 10, 2022

cmark self-assigned this Jan 10, 2022

cmark pinned this issue Jan 10, 2022

apeteri unpinned this issue Jan 12, 2022

apeteri pinned this issue Jan 12, 2022

This was referenced Jan 14, 2022

chore(deps): upgrade elasticsearch to 7.16.3... #956

Merged

[7.x] chore(deps): upgrade elasticsearch to 7.16.3 and log4j to 2.17.1 #957

Merged

cmark closed this as completed Jan 27, 2022

aelred mentioned this issue Feb 2, 2022

[MTS-1006] Disable Log4j exploit babylonhealth/snow-owl#96

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log4j vulnerability #964

Log4j vulnerability #964

mattecasu commented Jan 10, 2022

cmark commented Jan 10, 2022

cmark commented Jan 27, 2022

mattecasu commented Jan 31, 2022 •

edited

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

mattecasu commented Feb 1, 2022

cmark commented Feb 1, 2022

aelred commented Feb 2, 2022

mattecasu commented Feb 14, 2022

cmark commented Feb 15, 2022

Log4j vulnerability #964

Log4j vulnerability #964

Comments

mattecasu commented Jan 10, 2022

cmark commented Jan 10, 2022

cmark commented Jan 27, 2022

mattecasu commented Jan 31, 2022 • edited

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

cmark commented Jan 31, 2022

aelred commented Jan 31, 2022

mattecasu commented Feb 1, 2022

cmark commented Feb 1, 2022

aelred commented Feb 2, 2022

mattecasu commented Feb 14, 2022

cmark commented Feb 15, 2022

mattecasu commented Jan 31, 2022 •

edited