CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes #693

vbalani002 · 2024-05-13T09:10:44Z

CVE Fixes

CC-26341, CC-26346: Fix CVEs in jackson-databind and netty-codec-http by bumping kafka-connect-storage-common.
CC-27071: Pin avatica-core and bump calcite-core & calcite-druid to highest version compatible with hive
CC-26343: Pin apache-hbase to exclude common-httpclients
CC-26342: Pin nimbus-jose-jwt with non-vulnerable version
CC-26345: Pin okio with non-vulnerable version

Docker Playground Tests

Test 1: #693 (comment)
Test 2: #693 (comment)

Does this solution apply anywhere else?

yes
no

If yes, where?

Test Strategy

Testing done:

Unit tests
Integration tests
System tests
Manual tests

Release Plan

…ng calcite-core to latest supported version (#698)

sonarqube-confluent · 2024-05-18T09:36:16Z

Analysis Details

0 Issues

0 Bugs
0 Vulnerabilities
0 Code Smells

Coverage and Duplications

No coverage information (72.70% Estimated after merge)
No duplication information (1.50% Estimated after merge)

Project ID: kafka-connect-hdfs

View in SonarQube

vbalani002 · 2024-05-18T09:37:31Z

> playground run -f hdfs2-sink.sh --connector-zip ~/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
14:45:58 ℹ️ 🚀 Running example with flags
14:45:58 ℹ️ ⛳ Flags used are --connector-zip=/Users/vbalani/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
14:45:58 ℹ️ 💀 Kill all docker containers
14:46:01 ℹ️ 📋 command to run again example has been copied to the clipboard (disable with 'playground config set clipboard false')
14:46:02 ℹ️ 🚀 Number of examples ran so far: 32
14:46:02 ℹ️ ####################################################
14:46:02 ℹ️ 🚀 Executing hdfs2-sink.sh in dir .
14:46:02 ℹ️ ####################################################
14:46:02 ℹ️ 💫 Using default CP version 7.6.1
14:46:02 ℹ️ 🎓 Use --tag option to specify different version, see https://kafka-docker-playground.io/#/how-to-use?id=🎯-for-confluent-platform-cp
14:46:02 ℹ️ 🎯🤐 CONNECTOR_ZIP (--connector-zip option) is set with /Users/vbalani/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
14:46:03 ℹ️ 🧰 Checking if Docker image confluentinc/cp-server-connect-base:7.6.1 contains additional tools
14:46:03 ℹ️ ⏳ it can take a while if image is downloaded for the first time
14:47:46 ℹ️ 👷📦 Re-building Docker image confluentinc/cp-server-connect-base:7.6.1 to include additional tools
[+] Building 19.3s (6/6) FINISHED                                                        docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                     0.0s
 => => transferring dockerfile: 575B                                                                     0.0s
 => [internal] load .dockerignore                                                                        0.0s
 => => transferring context: 2B                                                                          0.0s
 => [internal] load metadata for docker.io/confluentinc/cp-server-connect-base:7.6.1                     0.0s
 => [1/2] FROM docker.io/confluentinc/cp-server-connect-base:7.6.1                                       0.1s
 => [2/2] RUN if [ ! -f /tmp/done ]; then curl http://mirror.centos.org/centos/8-stream/AppStream/x86_  18.5s
 => exporting to image                                                                                   0.7s
 => => exporting layers                                                                                  0.7s
 => => writing image sha256:c576a6d7da20dce4023ba821435f2615a935b2716ce646a8231c07a4b8112fb1             0.0s
 => => naming to docker.io/confluentinc/cp-server-connect-base:7.6.1                                     0.0s

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
14:48:06 ℹ️ 🎱 Installing connector from zip confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
Installing a component Kafka Connect HDFS 10.2.8-SNAPSHOT, provided by Confluent, Inc. from the local file: /tmp/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip into directory: /usr/share/confluent-hub-components
14:53:50 ℹ️ 💀 Kill all docker containers
14:53:56 ℹ️ 🛑 control-center is disabled
14:53:57 ℹ️ 🛑 ksqldb is disabled
14:53:57 ℹ️ 🛑 REST Proxy is disabled
14:53:57 ℹ️ 🛑 Grafana is disabled
14:53:58 ℹ️ 🛑 kcat is disabled
14:53:58 ℹ️ 🛑 conduktor is disabled
[+] Building 0.0s (0/0)                                                       docker-container:buildx-builder
[+] Running 3/0
 ✔ Volume plaintext_datanode  Removed                                                                    0.0s
 ✔ Volume plaintext_namenode  Removed                                                                    0.0s
 ✔ Network plaintext_default  Removed                                                                    0.1s
[+] Running 9/9
 ✔ datanode Pulled                                                                                      92.6s
 ✔ schema-registry Pulled                                                                              224.8s
 ✔ namenode Pulled                                                                                      92.6s
 ✔ broker Pulled                                                                                         7.2s
 ✔ hive-metastore-postgresql Pulled                                                                     77.4s
 ✔ hive-server Pulled                                                                                   99.7s
 ✔ zookeeper Pulled                                                                                    105.4s
 ✔ hive-metastore Pulled                                                                                99.7s
 ✔ presto-coordinator Pulled                                                                           273.9s
[+] Building 0.0s (0/0)                                                       docker-container:buildx-builder
[+] Running 13/13
 ✔ Network plaintext_default            Created                                                          0.2s
 ✔ Volume "plaintext_namenode"          Created                                                          0.0s
 ✔ Volume "plaintext_datanode"          Created                                                          0.0s
 ✔ Container presto-coordinator         Started                                                          1.8s
 ✔ Container datanode                   Started                                                          1.7s
 ✔ Container namenode                   Started                                                          1.7s
 ✔ Container broker                     Started                                                          1.8s
 ✔ Container zookeeper                  Started                                                          1.8s
 ✔ Container hive-metastore-postgresql  Started                                                          1.7s
 ✔ Container hive-metastore             Started                                                          1.7s
 ✔ Container hive-server                Started                                                          1.7s
 ✔ Container schema-registry            Started                                                          0.1s
 ✔ Container connect                    Started                                                          0.1s
14:58:45 ℹ️ 📝 To see the actual properties file, use cli command playground container get-properties -c <container>
14:58:48 ℹ️ ✨ If you modify a docker-compose file and want to re-create the container(s), run cli command playground container recreate
14:58:48 ℹ️ ⌛ Waiting up to 300 seconds for connect to start
[2024-05-18 09:31:22,713] INFO [Worker clientId=connect-adminclient-producer, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1873)
15:01:25 ℹ️ 🚦 containers have started!
15:01:25 ℹ️ 📊 JMX metrics are available locally on those ports:
15:01:25 ℹ️     - zookeeper       : 9999
15:01:25 ℹ️     - broker          : 10000
15:01:25 ℹ️     - schema-registry : 10001
15:01:25 ℹ️     - connect         : 10002
15:01:44 ℹ️ Creating HDFS Sink connector
15:01:55 ℹ️ 🛠️ Creating 🌎onprem connector hdfs-sink
15:01:56 ℹ️ 📋 🌎onprem connector config has been copied to the clipboard (disable with 'playground config set clipboard false')
15:01:58 ℹ️ ✅ 🌎onprem connector hdfs-sink was successfully created
15:02:01 ℹ️ 🧰 Current config for 🌎onprem connector hdfs-sink (using REST API /config endpoint)
playground connector create-or-update --connector hdfs-sink --no-clipboard << EOF
{
  "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
  "flush.size": "3",
  "hadoop.conf.dir": "/etc/hadoop/",
  "hive.database": "testhive",
  "hive.integration": "true",
  "hive.metastore.uris": "thrift://hive-metastore:9083",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "logs.dir": "/tmp",
  "name": "hdfs-sink",
  "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
  "rotate.interval.ms": "120000",
  "schema.compatibility": "BACKWARD",
  "store.url": "hdfs://namenode:8020",
  "tasks.max": "1",
  "topics": "test_hdfs",
  "value.converter": "io.confluent.connect.avro.AvroConverter",
  "value.converter.schema.registry.url": "http://schema-registry:8081"
}
EOF
15:02:13 ℹ️ 🔩 list of all available parameters for 🌎onprem connector hdfs-sink (org.apache.kafka.connect.mirror.MirrorSourceConnector) and version 7.6.1-ce (with default value when applicable)
    "allow.optional.map.keys": "false",
    "avro.codec": "",
    "connect.hdfs.keytab": "STRING",
    "connect.hdfs.principal": "STRING",
    "connect.meta.data": "true",
    "directory.delim": "/",
    "enhanced.avro.schema.support": "true",
    "file.delim": "+",
    "filename.offset.zero.pad.width": "10",
    "flush.size": "",
    "format.class": "io.confluent.connect.hdfs.avro.AvroFormat",
    "hadoop.conf.dir": "STRING",
    "hadoop.home": "STRING",
    "hdfs.authentication.kerberos": "false",
    "hdfs.namenode.principal": "STRING",
    "hdfs.url": "",
    "hive.conf.dir": "STRING",
    "hive.database": "default",
    "hive.home": "STRING",
    "hive.integration": "false",
    "hive.metastore.uris": "STRING",
    "hive.table.name": "${topic}",
    "kerberos.ticket.renew.period.ms": "3600000",
    "locale": "STRING",
    "logs.dir": "logs",
    "partition.duration.ms": "-1",
    "partition.field.name": "LIST",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "path.format": "STRING",
    "retry.backoff.ms": "5000",
    "rotate.interval.ms": "-1",
    "rotate.schedule.interval.ms": "-1",
    "schema.compatibility": "NONE",
    "schemas.cache.config": "1000",
    "shutdown.timeout.ms": "3000",
    "storage.class": "io.confluent.connect.hdfs.storage.HdfsStorage",
    "store.url": "",
    "timestamp.extractor": "Wallclock",
    "timestamp.field": "timestamp",
    "timezone": "STRING",
    "topic.capture.groups.regex": "",
    "topics.dir": "topics",
15:02:24 ℹ️ 🥁 Waiting a few seconds to get new status
15:02:32 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink                      ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
15:02:37 ℹ️ 🌐 documentation for 🌎onprem connector kafka-connect-hdfs is available at:
https://docs.confluent.io/kafka-connect-hdfs/current/index.html
15:02:50 ℹ️ Sending messages to topic test_hdfs
15:02:59 ℹ️ 🔮 value schema was identified as avro
15:02:59 ℹ️ ✨ generating value data...
15:02:59 ℹ️ ☢️ --forced-value is set
15:02:59 ℹ️ ✨ 10 records were generated based on --forced-value  (only showing first 10), took: 0min 0sec
{"f1":"value1"}
{"f1":"value2"}
{"f1":"value3"}
{"f1":"value4"}
{"f1":"value5"}
{"f1":"value6"}
{"f1":"value7"}
{"f1":"value8"}
{"f1":"value9"}
{"f1":"value10"}
15:03:12 ℹ️ 📤 producing 10 records to topic test_hdfs
15:03:20 ℹ️ 📤 produced 10 records to topic test_hdfs, took: 0min 8sec
15:03:30 ℹ️ Listing content of /topics/test_hdfs/partition=0 in HDFS
Found 3 items
-rwxrwxrwx   3 appuser supergroup        213 2024-05-18 09:33 /topics/test_hdfs/partition=0/test_hdfs+0+0000000000+0000000002.avro
-rw-r--r--   3 appuser supergroup        213 2024-05-18 09:33 /topics/test_hdfs/partition=0/test_hdfs+0+0000000003+0000000005.avro
-rw-r--r--   3 appuser supergroup        213 2024-05-18 09:33 /topics/test_hdfs/partition=0/test_hdfs+0+0000000006+0000000008.avro
15:03:33 ℹ️ Getting one of the avro files locally and displaying content with avro-tools
Successfully copied 2.05kB to /tmp/
{"f1":"value1"}
{"f1":"value2"}
{"f1":"value3"}
15:04:13 ℹ️ Check data with beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 2.3.2 by Apache Hive
beeline> !connect jdbc:hive2://hive-server:10000/testhive
Enter username for jdbc:hive2://hive-server:10000/testhive: Connecting to jdbc:hive2://hive-server:10000/testhive
hive
Enter password for jdbc:hive2://hive-server:10000/testhive: ****
Connected to: Apache Hive (version 2.3.2)
Driver: Hive JDBC (version 2.3.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hive-server:10000/testhive> show create table test_hdfs;
+----------------------------------------------------+
|                   createtab_stmt                   |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE `test_hdfs`(                 |
|   `f1` string COMMENT '')                          |
| PARTITIONED BY (                                   |
|   `partition` string COMMENT '')                   |
| ROW FORMAT SERDE                                   |
|   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'   |
| STORED AS INPUTFORMAT                              |
|   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  |
| OUTPUTFORMAT                                       |
|   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' |
| LOCATION                                           |
|   'hdfs://namenode:8020/topics/test_hdfs'          |
| TBLPROPERTIES (                                    |
|   'avro.schema.literal'='{"type":"record","name":"ConnectDefault","namespace":"io.confluent.connect.avro","fields":[{"name":"f1","type":"string"}]}',  |
|   'transient_lastDdlTime'='1716024802')            |
+----------------------------------------------------+
15 rows selected (2.379 seconds)
0: jdbc:hive2://hive-server:10000/testhive> select * from test_hdfs;
+---------------+----------------------+
| test_hdfs.f1  | test_hdfs.partition  |
+---------------+----------------------+
| value1        | 0                    |
| value2        | 0                    |
| value3        | 0                    |
| value4        | 0                    |
| value5        | 0                    |
| value6        | 0                    |
| value7        | 0                    |
| value8        | 0                    |
| value9        | 0                    |
+---------------+----------------------+
9 rows selected (4.753 seconds)
0: jdbc:hive2://hive-server:10000/testhive> Closing: 0: jdbc:hive2://hive-server:10000/testhive
| value1        | 0                    |
15:04:26 ℹ️ ####################################################
15:04:26 ℹ️ ✅ RESULT: SUCCESS for hdfs2-sink.sh (took: 18min 24sec - )
15:04:26 ℹ️ ####################################################

15:04:36 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink                      ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
15:04:38 ℹ️ 🌐 documentation is available at:
https://docs.confluent.io/current/connect/kafka-connect-hdfs/index.html

vbalani002 · 2024-05-18T09:51:41Z

> playground run -f hdfs2-sink-kerberos.sh --connector-zip ~/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
15:08:14 ℹ️ 🚀 Running example with flags
15:08:14 ℹ️ ⛳ Flags used are --connector-zip=/Users/vbalani/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
15:08:15 ℹ️ 💀 Kill all docker containers
15:08:17 ℹ️ 📋 command to run again example has been copied to the clipboard (disable with 'playground config set clipboard false')
15:08:17 ℹ️ 🚀 Number of examples ran so far: 33
15:08:17 ℹ️ ####################################################
15:08:17 ℹ️ 🚀 Executing hdfs2-sink-kerberos.sh in dir .
15:08:17 ℹ️ ####################################################
15:08:18 ℹ️ 💫 Using default CP version 7.6.1
15:08:18 ℹ️ 🎓 Use --tag option to specify different version, see https://kafka-docker-playground.io/#/how-to-use?id=🎯-for-confluent-platform-cp
15:08:18 ℹ️ 🎯🤐 CONNECTOR_ZIP (--connector-zip option) is set with /Users/vbalani/gitrepos/kafka-connect-hdfs/target/components/packages/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
15:08:18 ℹ️ 🧰 Checking if Docker image confluentinc/cp-server-connect-base:7.6.1 contains additional tools
15:08:18 ℹ️ ⏳ it can take a while if image is downloaded for the first time
15:08:18 ℹ️ 🎱 Installing connector from zip confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip
Installing a component Kafka Connect HDFS 10.2.8-SNAPSHOT, provided by Confluent, Inc. from the local file: /tmp/confluentinc-kafka-connect-hdfs-10.2.8-SNAPSHOT.zip into directory: /usr/share/confluent-hub-components
15:12:48 ℹ️ 💀 Kill all docker containers
15:12:56 ℹ️ 🛑 control-center is disabled
15:12:57 ℹ️ 🛑 ksqldb is disabled
15:12:57 ℹ️ 🛑 REST Proxy is disabled
15:12:57 ℹ️ 🛑 Grafana is disabled
15:12:58 ℹ️ 🛑 kcat is disabled
15:12:58 ℹ️ 🛑 conduktor is disabled
[+] Building 0.0s (0/0)                                                       docker-container:buildx-builder
[+] Running 1/0
 ! Network kerberos-demo.local  No resource found to remove                                              0.0s
[+] Running 2/2
 ✔ kdc Pulled                                                                                          126.9s
 ✔ hadoop Pulled                                                                                       188.2s
[+] Building 0.0s (0/0)                                                       docker-container:buildx-builder
[+] Running 7/7
 ✔ Network kerberos-demo.local  Created                                                                  0.1s
 ✔ Container broker             Started                                                                  2.6s
 ✔ Container kdc                Started                                                                  2.6s
 ✔ Container zookeeper          Started                                                                  2.6s
 ✔ Container hadoop             Started                                                                  0.1s
 ✔ Container schema-registry    Started                                                                  0.1s
 ✔ Container connect            Started                                                                  0.1s
15:16:13 ℹ️ 📝 To see the actual properties file, use cli command playground container get-properties -c <container>
15:16:16 ℹ️ ✨ If you modify a docker-compose file and want to re-create the container(s), run cli command playground container recreate
15:16:16 ℹ️ ⌛ Waiting up to 300 seconds for connect to start
[2024-05-18 09:47:28,241] INFO [Worker clientId=connect-adminclient-producer, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1873)
15:17:28 ℹ️ 🚦 containers have started!
15:17:28 ℹ️ 📊 JMX metrics are available locally on those ports:
15:17:28 ℹ️     - zookeeper       : 9999
15:17:28 ℹ️     - broker          : 10000
15:17:29 ℹ️     - schema-registry : 10001
15:17:29 ℹ️     - connect         : 10002
kinit: Client not found in Kerberos database while getting initial credentials
15:17:49 ℹ️ Restarting docker container hadoop
hadoop
Password for root@EXAMPLE.COM:
24/05/18 09:48:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15:18:31 ℹ️ Add connect kerberos principal
Authenticating as principal root/admin@EXAMPLE.COM with password.
<ndkey connect/connect.kerberos-demo.local@EXAMPLE.COM
WARNING: no policy specified for connect/connect.kerberos-demo.local@EXAMPLE.COM; defaulting to no policy
Principal "connect/connect.kerberos-demo.local@EXAMPLE.COM" created.
<enewable connect/connect.kerberos-demo.local@EXAMPLE.COM
Principal "connect/connect.kerberos-demo.local@EXAMPLE.COM" modified.
kadmin.local:  modprinc -maxrenewlife 11days krbtgt/EXAMPLE.COM
Principal "krbtgt/EXAMPLE.COM@EXAMPLE.COM" modified.
<xlife 11days connect/connect.kerberos-demo.local@EXAMPLE.COM
Principal "connect/connect.kerberos-demo.local@EXAMPLE.COM" modified.
<nnect.keytab connect/connect.kerberos-demo.local@EXAMPLE.COM
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/connect.keytab.
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/connect.keytab.
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/connect.keytab.
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/connect.keytab.
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/connect.keytab.
Entry for principal connect/connect.kerberos-demo.local@EXAMPLE.COM with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/connect.keytab.
kadmin.local:  listprincs
HTTP/hadoop.kerberos-demo.local@EXAMPLE.COM
K/M@EXAMPLE.COM
admin/admin@EXAMPLE.COM
connect/connect.kerberos-demo.local@EXAMPLE.COM
dn/hadoop.kerberos-demo.local@EXAMPLE.COM
jhs/hadoop.kerberos-demo.local@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/kdc.kerberos-demo.local@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
nm/hadoop.kerberos-demo.local@EXAMPLE.COM
nn/hadoop.kerberos-demo.local@EXAMPLE.COM
rm/hadoop.kerberos-demo.local@EXAMPLE.COM
root@EXAMPLE.COM
yarn/hadoop.kerberos-demo.local@EXAMPLE.COM
kadmin.local:  15:18:32 ℹ️ Copy connect.keytab to connect container /tmp/sshuser.keytab
Successfully copied 2.56kB to /Users/vbalani/gitrepos/kafka-docker-playground/connect/connect-hdfs2-sink/.
Successfully copied 2.56kB to connect:/tmp/connect.keytab
15:18:32 ℹ️ Creating HDFS Sink connector
15:18:35 ℹ️ 🛠️ Creating 🌎onprem connector hdfs-sink-kerberos
15:18:35 ℹ️ 📋 🌎onprem connector config has been copied to the clipboard (disable with 'playground config set clipboard false')
15:18:35 ℹ️ ✅ 🌎onprem connector hdfs-sink-kerberos was successfully created
15:18:37 ℹ️ 🧰 Current config for 🌎onprem connector hdfs-sink-kerberos (using REST API /config endpoint)
playground connector create-or-update --connector hdfs-sink-kerberos --no-clipboard << EOF
{
  "connect.hdfs.keytab": "/tmp/connect.keytab",
  "connect.hdfs.principal": "connect/connect.kerberos-demo.local@EXAMPLE.COM",
  "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
  "flush.size": "3",
  "hadoop.conf.dir": "/etc/hadoop/",
  "hdfs.authentication.kerberos": "true",
  "hdfs.namenode.principal": "nn/hadoop.kerberos-demo.local@EXAMPLE.COM",
  "kerberos.ticket.renew.period.ms": "60000",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "logs.dir": "/logs",
  "name": "hdfs-sink-kerberos",
  "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
  "rotate.interval.ms": "120000",
  "schema.compatibility": "BACKWARD",
  "store.url": "hdfs://hadoop.kerberos-demo.local:9000",
  "tasks.max": "1",
  "topics": "test_hdfs",
  "value.converter": "io.confluent.connect.avro.AvroConverter",
  "value.converter.schema.registry.url": "http://schema-registry:8081"
}
EOF
15:18:40 ℹ️ 🔩 list of all available parameters for 🌎onprem connector hdfs-sink-kerberos (org.apache.kafka.connect.mirror.MirrorSourceConnector) and version 7.6.1-ce (with default value when applicable)
    "allow.optional.map.keys": "false",
    "avro.codec": "",
    "connect.hdfs.keytab": "STRING",
    "connect.hdfs.principal": "STRING",
    "connect.meta.data": "true",
    "directory.delim": "/",
    "enhanced.avro.schema.support": "true",
    "file.delim": "+",
    "filename.offset.zero.pad.width": "10",
    "flush.size": "",
    "format.class": "io.confluent.connect.hdfs.avro.AvroFormat",
    "hadoop.conf.dir": "STRING",
    "hadoop.home": "STRING",
    "hdfs.authentication.kerberos": "false",
    "hdfs.namenode.principal": "STRING",
    "hdfs.url": "",
    "hive.conf.dir": "STRING",
    "hive.database": "default",
    "hive.home": "STRING",
    "hive.integration": "false",
    "hive.metastore.uris": "STRING",
    "hive.table.name": "${topic}",
    "kerberos.ticket.renew.period.ms": "3600000",
    "locale": "STRING",
    "logs.dir": "logs",
    "partition.duration.ms": "-1",
    "partition.field.name": "LIST",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
    "path.format": "STRING",
    "retry.backoff.ms": "5000",
    "rotate.interval.ms": "-1",
    "rotate.schedule.interval.ms": "-1",
    "schema.compatibility": "NONE",
    "schemas.cache.config": "1000",
    "shutdown.timeout.ms": "3000",
    "storage.class": "io.confluent.connect.hdfs.storage.HdfsStorage",
    "store.url": "",
    "timestamp.extractor": "Wallclock",
    "timestamp.field": "timestamp",
    "timezone": "STRING",
    "topic.capture.groups.regex": "",
    "topics.dir": "topics",
15:18:40 ℹ️ 🥁 Waiting a few seconds to get new status
15:18:46 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink-kerberos
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink-kerberos             ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
15:18:48 ℹ️ 🌐 documentation for 🌎onprem connector kafka-connect-hdfs is available at:
https://docs.confluent.io/kafka-connect-hdfs/current/index.html
15:18:49 ℹ️ Sending messages to topic test_hdfs
15:18:50 ℹ️ 🔮 value schema was identified as avro
15:18:50 ℹ️ ✨ generating value data...
15:18:50 ℹ️ ☢️ --forced-value is set
15:18:51 ℹ️ ✨ 10 records were generated based on --forced-value  (only showing first 10), took: 0min 1sec
{"f1":"value1"}
{"f1":"value2"}
{"f1":"value3"}
{"f1":"value4"}
{"f1":"value5"}
{"f1":"value6"}
{"f1":"value7"}
{"f1":"value8"}
{"f1":"value9"}
{"f1":"value10"}
15:18:55 ℹ️ 📤 producing 10 records to topic test_hdfs
15:18:58 ℹ️ 📤 produced 10 records to topic test_hdfs, took: 0min 3sec
15:19:08 ℹ️ Listing content of /topics/test_hdfs/partition=0 in HDFS
24/05/18 09:49:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `/topics/test_hdfs/partition=0': No such file or directory

15:19:15 ℹ️ 🧩 Displaying status for 🌎onprem connector hdfs-sink-kerberos
Name                           Status       Tasks                                                        Stack Trace
-------------------------------------------------------------------------------------------------------------
hdfs-sink-kerberos             ✅ RUNNING  0:🟢 RUNNING[connect]        -
-------------------------------------------------------------------------------------------------------------
15:19:16 ℹ️ 🌐 documentation is available at:
https://docs.confluent.io/current/connect/kafka-connect-hdfs/index.html

tarunjain-confluent

LGTM

Upgrade kafka-connect-storage-common version

9d8c708

vbalani002 requested a review from a team as a code owner May 13, 2024 09:10

vbalani002 changed the title ~~CC-26341: Upgrade jackson-databind to fix CVE~~ CC-26341: Bump kafka-connect-storage-common version to fix CVE May 13, 2024

This comment has been minimized.

Sign in to view

Bump storage common version

121cd00

This comment has been minimized.

Sign in to view

vbalani002 added 5 commits May 18, 2024 13:19

CC-27071: Fix CVEs in avatica-core by pinning the dependency + Updati…

516c859

…ng calcite-core to latest supported version (#698)

CC-26343: Exclude commons-httpclient to fix CVE (#697)

3b90afe

CC-26342: Pin numbus-jose-jwt to fix CVE (#696)

dbd43bf

CC-26345: Pin okio version to fix CVE (#694)

d31fdb3

Fix non-parsable pom

27cdb5d

vbalani002 changed the title ~~CC-26341: Bump kafka-connect-storage-common version to fix CVE~~ CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes May 18, 2024

This comment has been minimized.

Sign in to view

Pin nimbus-jose-jwt

84037c5

tarunjain-confluent approved these changes May 19, 2024

View reviewed changes

vbalani002 merged commit 16aa5d9 into 10.2.x May 19, 2024
2 checks passed

vbalani002 deleted the CC-26341 branch May 19, 2024 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes #693

CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes #693

vbalani002 commented May 13, 2024 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented May 18, 2024

vbalani002 commented May 18, 2024

vbalani002 commented May 18, 2024

tarunjain-confluent left a comment

CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes #693

CC-26341, CC-26342, CC-26343, CC-26345, CC-26346, CC-27071: CVE Fixes #693

Conversation

vbalani002 commented May 13, 2024 • edited

CVE Fixes

Docker Playground Tests

Does this solution apply anywhere else?

If yes, where?

Test Strategy

Testing done:

Release Plan

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented May 18, 2024

Analysis Details

0 Issues

Coverage and Duplications

vbalani002 commented May 18, 2024

vbalani002 commented May 18, 2024

tarunjain-confluent left a comment

Choose a reason for hiding this comment

vbalani002 commented May 13, 2024 •

edited