Skip to content

Commit

Permalink
Merge pull request #15 from ibm-messaging/1.0.1
Browse files Browse the repository at this point in the history 
Version 1.0.1
  • Loading branch information
@AndrewJSchofield
AndrewJSchofield committed Nov 30, 2018
2 parents 222a723 + 0a48642 commit 09d967c
Show file tree
Hide file tree
Showing 8 changed files with 256 additions and 260 deletions.
40 changes: 16 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ kafka-connect-mq-source is a [Kafka Connect](http://kafka.apache.org/documentati

The connector is supplied as source code which you can easily build into a JAR file.

**Note**: A sink connector for IBM MQ is also available on [GitHub](https://github.com/ibm-messaging/kafka-connect-mq-sink).

## Contents

- [Building the connector](#building-the-connector)
Expand All @@ -20,7 +22,7 @@ The connector is supplied as source code which you can easily build into a JAR f
## Building the connector
To build the connector, you must have the following installed:
* [git](https://git-scm.com/)
* [Maven](https://maven.apache.org)
* [Maven 3.0 or later](https://maven.apache.org)
* Java 8 or later

Clone the repository with the following command:
Expand All @@ -38,24 +40,26 @@ Build the connector using Maven:
mvn clean package
```

Once built, the output is a single JAR called `target/kafka-connect-mq-source-1.0-SNAPSHOT-jar-with-dependencies.jar` which contains all of the required dependencies.
Once built, the output is a single JAR called `target/kafka-connect-mq-source-<version>-jar-with-dependencies.jar` which contains all of the required dependencies.


## Running the connector

**NOTE:** For a more detailed guide to running the connector see the [IBM Event Streams documentation](https://ibm.github.io/event-streams/connecting/mq/).
For step-by-step instructions, see the following guides for running the connector:
- connecting to Apache Kafka [running locally](UsingMQWithKafkaConnect.md)
- connecting to an installation of [IBM Event Streams](https://ibm.github.io/event-streams/connecting/mq/source)

To run the connector, you must have:
* The JAR from building the connector
* A properties file containing the configuration for the connector
* Apache Kafka 1.0 or later, either standalone or included as part of an offering such as IBM Event Streams
* Apache Kafka 2.0.0 or later, either standalone or included as part of an offering such as IBM Event Streams
* IBM MQ v8 or later, or the IBM MQ on Cloud service

The connector can be run in a Kafka Connect worker in either standalone (single process) or distributed mode. It's a good idea to start in standalone mode.

You need two configuration files, one for the configuration that applies to all of the connectors such as the Kafka bootstrap servers, and another for the configuration specific to the MQ source connector such as the connection information for your queue manager. For the former, the Kafka distribution includes a file called `connect-standalone.properties` that you can use as a starting point. For the latter, you can use `config/mq-source.properties` in this repository.

The connector connects to MQ using a client connection. You must provide the name of the queue manager, the connection name (one or more host/port pairs) and the channel name. In addition, you can provide a user name and password if the queue manager is configured to require them for client connections. If you look at the supplied `config/mq-source.properties`, you'll see how to specify the configuration required.
The connector connects to MQ using either a client or a bindings connection. For a client connection, you must provide the name of the queue manager, the connection name (one or more host/port pairs) and the channel name. In addition, you can provide a user name and password if the queue manager is configured to require them for client connections. If you look at the supplied `config/mq-source.properties`, you'll see how to specify the configuration required. For a bindings connection, you must provide provide the name of the queue manager and also run the Kafka Connect worker on the same system as the queue manager.

To run the connector in standalone mode from the directory into which you installed Apache Kafka, you use a command like this:

Expand Down Expand Up @@ -140,7 +144,7 @@ In MQ, the message ID and correlation ID are both 24-byte arrays. As strings, th
The connector supports authentication with user name and password and also connections secured with TLS using a server-side certificate and mutual authentication with client-side certificates.

### Setting up MQ connectivity using TLS with a server-side certificate
To enable use of TLS, set the configuration `mq.ssl.cipher.suite` to the name of the cipher suite which matches the CipherSpec in the SSLCIPH attribute of the MQ server-connection channel. Use the table of supported cipher suites for MQ 9.1 [here] ((https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.dev.doc/q113220_.htm) as a reference. Note that the names of the CipherSpecs as used in the MQ configuration are not necessarily the same as the cipher suite names that the connector uses. The connector uses the JMS interface so it follows the Java conventions.
To enable use of TLS, set the configuration `mq.ssl.cipher.suite` to the name of the cipher suite which matches the CipherSpec in the SSLCIPH attribute of the MQ server-connection channel. Use the table of supported cipher suites for MQ 9.1 [here](https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.dev.doc/q113220_.htm) as a reference. Note that the names of the CipherSpecs as used in the MQ configuration are not necessarily the same as the cipher suite names that the connector uses. The connector uses the JMS interface so it follows the Java conventions.

You will need to put the public part of the queue manager's certificate in the JSSE truststore used by the Kafka Connect worker that you're using to run the connector. If you need to specify extra arguments to the worker's JVM, you can use the EXTRA_ARGS environment variable.

Expand All @@ -165,6 +169,7 @@ The configuration options for the Kafka Connect source connector for IBM MQ are
| Name | Description | Type | Default | Valid values |
| ---------------------------- | ----------------------------------------------------------- | ------- | ------------- | ------------------------------------------------------- |
| mq.queue.manager | The name of the MQ queue manager | string | | MQ queue manager name |
| mq.connection.mode | The connection mode - bindings or client | string | client | client, bindings |
| mq.connection.name.list | List of connection names for queue manager | string | | host(port)[,host(port),...] |
| mq.channel.name | The name of the server-connection channel | string | | MQ channel name |
| mq.queue | The name of the source MQ queue | string | | MQ queue name |
Expand All @@ -179,14 +184,14 @@ The configuration options for the Kafka Connect source connector for IBM MQ are
| topic | The name of the target Kafka topic | string | | Topic name |

### Using a CCDT file
Some of the connection details for MQ can be provided in a [CCDT file](https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.con.doc/q016730_.htm) by setting `mq.ccdt.url` in the Kafka Connect source connector configuration file. If using a CCDT file the `mq.connection.name.list` and `mq.channel.name` configuration options are not required.
Some of the connection details for MQ can be provided in a [CCDT file](https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.con.doc/q016730_.htm) by setting `mq.ccdt.url` in the Kafka Connect source connector configuration file. If using a CCDT file the `mq.connection.name.list` and `mq.channel.name` configuration options are not required.

### Externalizing secrets
[KIP 297](https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations) introduced a mechanism to externalize secrets to be used as configuration for Kafka connectors.

#### Example: externalizing secrets with FileConfigProvider

Given a file `secrets.properties` with the contents:
Given a file `mq-secrets.properties` with the contents:
```
secret-key=password
```
Expand All @@ -206,28 +211,15 @@ Update the connector configuration file to reference `secret-key` in the file:
mq.password=${file:mq-secret.properties:secret-key}
```

#### Using custom Config Providers

Custom config providers can also be enabled in the worker configuration file:
```
# Additional properties for the worker configuration to enable use of ConfigProviders
# multiple comma-separated provider types can be specified here
config.providers=file,other-provider
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
# Other ConfigProvider implementations might require parameters passed in to configure() as follows:
config.providers.other-provider.param.foo=value1
config.providers.other-provider.param.bar=value2
```

## Troubleshooting

### Unable to connect to Kafka

You may receive an `org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed` error when trying to run the MQ Source Connector using SSL to connect to your Kafka cluster. In the case that the error is caused by the following exception: `Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching XXXXX found.`, Java may be replacing the IP address of your cluster with the corresponding hostname in your `/etc/hosts` file. For example, to push Docker images to your ICP cluster, you may add an entry in this file which corresponds to the IP of your cluster e.g. `123.456.78.90 mycluster.icp`. To fix this, you can comment out this line in your `/etc/hosts` file.
You may receive an `org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed` error when trying to run the MQ Source Connector using SSL to connect to your Kafka cluster. In the case that the error is caused by the following exception: `Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching XXXXX found.`, Java may be replacing the IP address of your cluster with the corresponding hostname in your `/etc/hosts` file. For example, to push Docker images to a custom Docker repository, you may add an entry in this file which corresponds to the IP of your repository e.g. `123.456.78.90 mycluster.icp`. To fix this, you can comment out this line in your `/etc/hosts` file.


## Support
A commercially supported version of this connector is available for customers with a support entitlement for [IBM Event Streams](https://developer.ibm.com/messaging/event-streams/).
A commercially supported version of this connector is available for customers with a support entitlement for [IBM Event Streams](https://www.ibm.com/cloud/event-streams).


## Issues and contributions
Expand All @@ -247,4 +239,4 @@ Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.The project is licensed under the Apache 2 license.
limitations under the License.The project is licensed under the Apache 2 license.

0 comments on commit 09d967c

Please sign in to comment.