connect-jdbc

Kafka Connect JDBC test environment

This Docker Compose can be used to spin up an environment in which to explore and test the Kafka Connect JDBC source connector.

The associated blog article can be found at https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector

Table 1. Current environment

Confluent Platform	5.4.0
MySQL	8.0.13
Postgres	11.1
MS SQL Server	2017 (RTM-CU13) (KB4466404) - 14.0.3048.4 (X64)
Oracle	12.2.0.1.0

Getting a CLI session

• KSQL

  docker exec --tty --interactive ksql-cli ksql http://ksql-server:8088

• MySQL

  docker exec --tty --interactive mysql bash -c 'mysql -u root -p$MYSQL_ROOT_PASSWORD'

• Postgres

  docker exec --tty --interactive postgres bash -c 'psql -U $POSTGRES_USER $POSTGRES_DB'

• Oracle

  docker exec --tty --interactive oracle bash -c 'sqlplus sys/$ORACLE_PWD@localhost:1521/ORCLCDB as sysdba'

• MS SQL Server

  docker exec --tty --interactive mssql bash -c '/opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -U sa -P $SA_PASSWORD'

Example Connector Configuration

• MySQL

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
          "name": "jdbc_source_mysql_01",
          "config": {
                  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                  "connection.url": "jdbc:mysql://mysql:3306/demo",
                  "connection.user": "connect_user",
                  "connection.password": "asgard",
                  "topic.prefix": "mysql-01-",
                  "mode":"bulk",
                  "poll.interval.ms" : 3600000
                  }
          }'

• Postgres

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
          "name": "jdbc_source_postgres_01",
          "config": {
                  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                    "connection.url": "jdbc:postgresql://postgres:5432/postgres",
                  "connection.user": "connect_user",
                  "connection.password": "asgard",
                  "topic.prefix": "postgres-01-",
                  "mode":"bulk",
                  "poll.interval.ms" : 3600000
                  }
          }'

• Oracle

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
                  "name": "jdbc_source_oracle_01",
                  "config": {
                          "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                          "connection.url": "jdbc:oracle:thin:@oracle:1521/ORCLPDB1",
                          "connection.user": "connect_user",
                          "connection.password": "asgard",
                          "topic.prefix": "oracle-01-",
                          "table.whitelist" : "NUM_TEST",
                          "mode":"bulk",
                          "poll.interval.ms" : 3600000
                          }
                  }'

• MS SQL Server

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
                  "name": "jdbc_source_mssql_01",
                  "config": {
                          "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                          "connection.url": "jdbc:sqlserver://mssql:1433;databaseName=demo",
                          "connection.user": "connect_user",
                          "connection.password": "Asgard123",
                          "topic.prefix": "mssql-01-",
                          "table.whitelist" : "demo..num_test",
                          "mode":"bulk",
                          "poll.interval.ms" : 3600000
                          }
                  }'