This project collaborates with the capstone design class at Kyungpook National University(KNU) and CSP MOBILE LAB.
We develop a Real-time Data Distribution Pipeline System with log data from KIOSK.
- OS : Ubuntu server 20.04 LTS
- Log generator: python 3.8.10
- File Beats 8.1.0
- Kafka 3.1.0
- Logstash 8.1.0
- Elasticsearch 8.1.0
- Kibana 8.1.0
Elasticsearch consists of three nodes.
- c1-elastic (Data Node)
- c2-elastic (Master node)
- c3-elastic (Data Node)
If a node acting as a master is disconnected from the network or goes down, one of the other master candidate nodes is elected as the master node and acts as the master node instead. The master candidate nodes share the information of the master node from the beginning, so the master role can be performed immediately.
Kafka conssists of three nodes.
- c1-kafka
- c2-kafka
- c3-kafka
We have built an on-premise server, and the following is a list of the components of the server.
- Raspberry Pi 4B - 4GB(1ea), 8GB(6ea)
- Disk - Sandisk Extreme 32GB MLC
- Router - iptime A3004T
- Switching hub - iptime H6008
- Raspberry Pi Power - 5V 3A C-type charger
- Etc - cat 6 LAN cable 30cm, Holder box case
More detailed information is written here.
We create a blog and post how to overcome the challenges, especially security settings, faced by developing the system. We think this can contribute a little bit to the open-source ecosystem.