Demo: Self-managed REST Proxy with Confluent Cloud

Let's try to host REST Proxy in localhost and make it work with CC.
## DISCLAIMER: THIS IS A SUPER SIMPLE EXAMPLE, not really useful for production etc.

If you want a kubernetes example by Confluent
https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/hybrid/ccloud-integration

Scenario:

• hosting Confluent Platform in my local machine, with Docker.
  
• Taking into advtange REST PROXY for sending messages to a topic in Confluent Cloud.

Good inspo from these links!

https://www.confluent.io/es-es/blog/a-comprehensive-rest-proxy-for-kafka/
https://github.com/confluentinc/kafka-rest
https://docs.confluent.io/platform/current/kafka-rest/api.html#rest-api-usage-examples

First clone the git repo, and enter in the folder:

git clone https://github.com/angietd94/confluent-cloud-rest-proxy.git
cd confluent-cloud-hybrid-demo

Second part: Setting up CP on localhost with Docker! (REST PROXY IS INCLUDED)

NOTE: first change the values in config.sh depending on your own account.

You can find the description on how to find all values in this repo:
https://github.com/angietd94/confluent-cloud-hybrid-demo
Where also it is explained how to launch the script in the "Second part: Launching the script! " but basically this is just to have CP running (AND THEN REST PROXY included).

## Detailed video of every step here:
---->

https://drive.google.com/file/d/1uh-MYKVAOFPCmMYWZnNLR3vbl6dsZeyX/view?usp=sharing

That basically is summaryzing:

chmod +x executable.sh 
chmod +x config_files.sh
chmod +x config.sh
chmod +x ccloud-generate-cp-configs.sh
./executable.sh

Create a topic and fill it with data!

- Let's create the topic (here rest_proxy_example, and put a simple schema AVRO, for example having a name and age of someone:

{
  "doc": "Sample schema to help you get started.",
  "fields": [
    {
      "name": "age",
      "type": "int"
    },
    {
      "doc": "The string is a unicode character sequence.",
      "name": "name",
      "type": "string"
    }
  ],
  "name": "sampleRecord",
  "namespace": "com.mycorp.mynamespace",
  "type": "record"
}

Is it all up and running in Docker? GOOD!

Let's try some curl commands...

For example:

curl http://localhost:8082/v3/clusters
curl http://localhost:8082/topics
curl http://localhost:8082/topics/rest_proxy_example | jq

A sample message that will work here:

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"name":"angelica","age":29}}]}' 

If good it will give 200 OK

And in Confluent Cloud:

Errors examples

Comprehensive guide can be found in https://docs.confluent.io/platform/current/kafka-rest/api.html and other documentations. Summizing you will get errors like:

In this guide, check https://github.com/angietd94/confluent-cloud-rest-proxy/blob/main/restproxy_commands.sh .

Error 422 Example wrong data: name field wrong in the schema

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"nombre\",\"type\":\"string\"}]}","records":[{"value":{"name":"angelica","age":29}}]}' 

## Error 422 Example wrong data: name field wrong in the schema AND in the data

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"nombre\",\"type\":\"string\"}]}","records":[{"value":{"nombre":"angelica","age":29}}]}' 
<br>

Error 422 Example: SampleRecord missing!

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"age":29}}]}' 

Error 400 Example: Name is an int and not a string

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"nombre":30,"age":29}}]}' 

Error 400: OR if missing data...

curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json"  --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"age":29}}]}' 

Examples with mock data

In the other parts of the video I basically just made ChatGPT make the work by creating some mock data similar to this.
The first time is working fine and in the second you see some errors. Not really relevant just an example.