Difference Between Apache Nifi and Apache Streampipes

[Praveenstein]

Just curious, i've used apache nifi a lot, just wondering how does streampipes differ?? I think streampipes architecture follows microservices pattern, apart from this any other difference, One thing that i noticed is, in apache nifi, everything is time based scheduled, so if in case i want to consume a mqtt message, i have to put a time (say every 1 min) to check for new messages, not like as soon as a message is published it will be consumed by nifi (the so called event driven strategy), can someone explain in detail, the difference, in both the underlying design principle, features, etc?? so that it could be for anybody looking for the answer in future also

[dominikriemer]

Hi @Praveenstein, good question! I totally agree with your findings. In general and from a use case perspective , I'd say Nifi is a generic dataflow processing system, while StreamPipes targets the Industrial IoT space in form of a toolbox with the goal to support non-technical users in analyzing IIoT data. E.g., the pipeline editor which is closest to the graphical modeling approach of Nifi (or also NodeRED) is supplemented with other tools such as the live dashboard, data explorer, or very new additions such as standalone functions and Python support targeted at data scientists.
From an architectural point of view, StreamPipes follows an event-driven way (pull-based sources can be connected, but internally everything is a stream). The exception is the data explorer, which relies on a time-series database for offline analytics. While Nifi and StreamPipes are both extensible in terms of data processors and sinks, I'm not 100% sure how the Nifi guys do it, but StreamPipes consists of microservices which contain one or more pipeline elements and which can be deployed in a geo-distributed way, so that also edge processing is supported (we are also working on further improving support for such use cases). Nifi has MiNifi which also aims for such cases. In terms of supported pipeline elements, we try to add more and more analytics-oriented processors to StreamPipes, e.g., for easy integration of AI models or working with PLC data while we are not so much focusing on data transformation and processing, which is an area Nifi is great at.

In terms of architecture, there are probably quite a lot of differences between both tools (you have mentioned a few), e.g., the underlying data model allows to define requirements on input streams of pipeline elements, which aims at reducing modeling errors and hiding technical details of connected event streams.

Hope this explains it a little bit! I'll be happy to dive deeper in case you have any questions and maybe other community members have other views ;-)

[dominikriemer]

It would also be great to hear your opinion on the differences since you've worked with Nifi and also your opinion where you would see StreamPipes and which features you are missing! (I'm already working on the OPC UA tree improvement ;-))

[Praveenstein]

Thanks for the reply, one thing which i feel is... Apache nifi is more matured (given its long history)... But it lacks some processors for industrial scenario, sometimes i feel like instead of creating something from the scratch (such as streampipes).... We could add more processors to nifi to support industrial use case. Is there any specific reason you had to built something from the scratch??? Something related to performance??? I know streampipes has integrated support for using ml models, i still do the same with nifi, but in an indirect way, i could have an http api endpoint just to run a model and return the results, this could be integrated into nifi using http processor, or in best scenario we (or your team) could create own processor just for running ml model ( or any analytics for that matter) and yes streampipes does have inbuilt dashboard and everything, but the same could be done using grafana also .....so any specific reason why your team have opted for building something from the scratch instead of creating nifi processor for industrial use case??