Subscribes to a Kafka topic and deposits messages to the Situation Data Warehouse (SDW).
This is a submodule of the jpo-ode repository. It subscribes to a Kafka topic and listens for incoming messages. Upon message arrival, this application deposits it over REST to the SDX.
The current version and release history of the Jpo-sdw-depositor: Jpo-sdw-depositor Release Notes
- Docker
Use this option when you want to run the depositor by itself. This will listen to any Kafka topic you specify and deposit messages to the Situation Data Exchange.
- Configure your desired properties. See Configuration Reference at the bottom of this README.
- Rename your
sample.envfile to.envif you haven't already done so - Execute the
run.shscript OR execute these commands:
docker build -t jpo-sdw-depositor .
docker run --rm --env-file .env jpo-sdw-depositor:latest
** IN PROGRESS! Further instructions pending ODE compatibility. **
Use this option when you want to run this module in conjuction with the jpo-ode. The only action you must take here is to set the configuration properties in the env file. See the bottom of this README for a reference.
SOME OF THESE PROPERTIES ARE SENSITIVE. DO NOT PUBLISH THEM TO VERSION CONTROL
You may configure these values in jpo-sdw-depositor/src/main/resources/application.properties or by editing them in the sample.env file.
IMPORTANT When using the env file method, you must You must rename or duplicate the sample.env file to .env
Value in application.properties |
Value as env var (in sample.env) | Description | Example Value |
|---|---|---|---|
| sdw.kafkaBrokers | DOCKER_HOST_IP | Host IP (instructions) | 10.1.2.3 |
| sdw.kafkaPort | SDW_KAFKA_PORT | Port of the Kafka instance | 9092 |
| sdw.subscriptionTopic | SDW_SUBSCRIPTION_TOPIC | Kafka topic to listen to | topic.J2735TimBroadcastJson |
| sdw.destinationUrl | SDW_DESTINATION_URL | Full path of the SDX server address | 127.0.0.1 |
| sdw.apikey | SDW_API_KEY | SDX API Key (generated by SDX) | (n/a) |
| sdw.emailList | SDW_EMAIL_LIST | Comma-delimited email list to send error emails to | error@email.com,test@test.com |
| sdw.emailFrom | SDW_EMAIL_FROM | Support email to send from | error@email.com |
Rather than using a local kafka instance, this project can utilize an instance of kafka hosted by Confluent Cloud via SASL.
- The DOCKER_HOST_IP environment variable is used to communicate with the bootstrap server that the instance of Kafka is running on.
- The KAFKA_TYPE environment variable specifies what type of kafka connection will be attempted and is used to check if Confluent should be utilized.
- The CONFLUENT_KEY and CONFLUENT_SECRET environment variables are used to authenticate with the bootstrap server.
- DOCKER_HOST_IP must be set to the bootstrap server address (excluding the port)
- KAFKA_TYPE must be set to "CONFLUENT"
- CONFLUENT_KEY must be set to the API key being utilized for CC
- CONFLUENT_SECRET must be set to the API secret being utilized for CC
There is a provided docker-compose file (docker-compose-confluent-cloud.yml) that passes the above environment variables into the container that gets created. Further, this file doesn't spin up a local kafka instance since it is not required.
This has only been tested with Confluent Cloud but technically all SASL authenticated Kafka brokers can be reached using this method.
The unit tests can be run by executing the following command from the root directory of the project:
mvn test
It should be noted that Maven & Java are required to run the unit tests. If you do not have Maven or Java installed, you can reopen the project in the provided dev container and run the tests from there.
The KafkaConsumerRestDepositor will accept any string as input to be passed into the SDW. If provided a JSON object, the tokens of "encodedMsg" and "estimatedRemovalDate" will be passed through directly to the SDW in the form of the following: {depositRequests:[{"encodeType": STRING ,"encodedMsg": STRING, "estimatedRemovalDate": STRING}]}
If provided a string of non-json form, the value of "encodedMsg" will inherit the passed value and information will be passed to the SDW in the form of the following: {depositRequests:[{"encodeType": STRING ,"encodedMsg": STRING}]}