Varadhi is the open source version of a similar FK internal project, a message bus implementation with a REST interface. Varadhi supports both P2P (point-to-point) and Publish-Subscribe messaging models. The idea behind RESTBus is to take your HTTP API stack and automatically transform them into service bus driven, queue/pub-sub, message oriented endpoints. The communication between Varadhi and endpoints will be on HTTP.
Varadhi is in active development and has been in production since last 10 years inside Flipkart as the backbone of async REST communication between various microservices sending billions of messages daily.
With Varadhi we hope others would be able to benefit from our learnings of a decade.
See the Wiki for concepts, use cases, architecture details, API spec and other detailed information.
./gradlew build
To provide the required functionality Varadhi takes a dependencies on various tech stack. Current development environment supports below option for these.
Varadhi needs a message broker for message persistence and delivery. Apache Pulsar is used as underlying message broker. For the development environment users can use containerised Pulsar in a standalone mode. Details can be found here.
docker run -it -p 6650:6650 -p 8081:8080 --mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf apachepulsar/pulsar:3.0.0 bin/pulsar standalone
For storing metadata about various Varadhi entities, a datastore is needed. Zookeeper is used as global datastore to persist json formatted entities. For the development environment, containerised Zookeeper can be used in a standalone mode. Details can be found here.
docker run --name some-zookeeper --restart always -d -p 2181:2181 -p 2888:2888 -p 3888:3888 -p 8082:8080 zookeeper
To run the Varadhi server execute below from repo root.
./gradlew run --args="src/main/resources/configuration.yml"
- entities: It has all the entities used by the spi module and varadhi apis.
- spi: It has all the interfaces related to persistence and messaging.
- pulsar: It contains messaging-spi implementation using Apache Pulsar.
- core: It contains all the core logic of
Varadhiand is relied upon by various other sub-modules. - server: It is the entry point of
Varadhi. It houses all server related logic and binds together all other modules.
[ TBD ]
Status: Initial Review Completed.
See the Wiki page Roadmap for details.
[ TBD ]
Reachout to sahil.chachan@flipkart.com or k.dhruv@flipkart.com incase you want to contribute here :)
[ TBD ]
Effective Failure Handling in Flipkart’s Message Bus
[ TBD ]