Home
Welcome to the wiki for the Azure Persistor!
Here, you will find more in-depth explanations on the Persistor's functionality, intended usage, limitations and overall performance. If you've had any issues or questions regarding the Persistor, make sure to take a peek here first!
The goal of the Persistor project is quite simple: achieving an easily-attachable component to existing pipelines that use Azure messaging services, and store messages to blob storage. The need for such a feature has become a common occurence during development. Trivail mistakes and miscommunication can break even the most well-though-out workflows, leading to major problems; especially if the messages that were supposed to be processed by said workflows failed to be and -- more critically -- cannot be re-sent.
Azure offers three messaging services: Event Grid, Event Hub and Service Bus. Of those three, Event Hub does offer message retention in the form of the Capture functionality. However, it comes with a few caveats. The first is that it's charged by the hour, regardless of the actual number of events being passed through. For high-activity services, this arrangement is, perhaps, more than welcome. However, there are undoubtedly cases where it may look to be an unnecessary extra charge. If the throughput is not high, why not simply be charged for the time it would take to store the messages? In addition, the captured events are all stored in the Avro format, which may or may not be the preferred option to work with, depending on the circumstances.
That's where the idea for the Persistor ultimately comes from. Using Azure's serverless Azure Functions, you can store messages when you need and as cleanly as possible: JSON text files containing the payload and any optional metadata that could've been passed, depending on the used messaging service.