Undergraduate Capstone Project: Moving Object Detection System on a Trainer Conveyor with Raspberry Pi and Schneider M221 PLC
This repository contains my final project as a requirement to obtain my bachelor's degree. I developed a device to detect objects on a trainer conveyor in this project. This device is expected to be used by students to understand how similar devices work in industry. The device is connected to a programmable logic controller (PLC) through the Modbus protocol to support this goal. Through this protocol, detection data will be received by the controller so that it can be used for further processing.
In detecting objects, two types of computer vision methods have been implemented. The first method is traditional methods such as color thresholding and contour detection. The second method is detection with an artificial neural network model (deep learning), namely the EfficientDet model. Since the main hardware has limited computational capabilities, the model used is a lightweight variant called EfficientDet-Lite.
The main hardware used in building this tool is Raspberry Pi 4. In addition to Raspberry Pi, IR sensors are also used to detect the presence of objects and calculate the detection delay time. A camera module was also installed to acquire images from the top side of the conveyor. Finally, an LED is added as an indicator of the detected class.
A demonstration of how the device works can be observed in the following YouTube video. Please click on the thumbnail image to view the video.
The scope of the capstone design project can be observed in the following Figure
In preparing the EfficientDet-Lite model, the steps taken consist of image data collection (which is stored in Google Drive), data augmentation process, labeling with Label Studio, and training process. Most of these processes were carried out at Google Collaboratory. The code will then call the resulting model on the Raspberry Pi. Both the dataset and the model can be found in the folder of the same name.
The dependencies of each code in this repository can be observed in the following figure. The main codes to be called are run.py and capture.py. The first code will run the detection while the second code will record the detected object.
The first detection scenario is to detect objects based on color differences. For this purpose, three rubber duck objects of the same size but with different colors were prepared. When detecting objects based on color, the program will return the predicted class index and a tuple containing the RGB value of the object. These two values will be extracted by the PLC.
The second detection scenario is to detect objects based on shape differences. Therefore, three toy objects of the same color but different sizes have been prepared. In shape detection, the predicted class index will also be collected. However, in this scenario, the RGB value will be replaced by the size or dimension parameter of the object. For example, the program will return the area of the object in pixels and the approximate corner points of the contour.
The difference in the detection process for the two scenarios mentioned can be observed below. When an object is detected, the user will be able to observe the object class and the collected features.
Besides the detection video, other outputs of the program consist of a message on the terminal, detection performance graphs, and a CSV file. The terminal message and the resulting performance graphs can be observed below. The performance visualized here consists of the delay time between camera and IR sensor detection, frame rate, and detected/undetected object ratio.
These performance values can also be observed in greater detail within the CSV file. The CSV file allows the performance data to be further analyzed. Examples of CSV files generated by the program can be observed in the csv_data folder. The explanation for each column in the CSV data can be observed in the Table below.
| Column | Description | Unit |
|---|---|---|
| Delay | Delay time between camera and IR | Seconds |
| FPS | Detection frame rate | FPS |
| Latency (Regs) | Latency between Raspberry Pi and PLC registers | Seconds |
| Latency (Coils) | Latency between Raspberry Pi and PLC coils | Seconds |
| Detected | Boolean value whether the object is detected or not | - |
| Probability | (Deep Learning) Prediction probability for the object | - |
| Blue, Green, Red | (Traditional) BGR color obtained from object detection | - |
| Area, Points | (Traditional) Contour parameters obtained from object detection | (Area) Pixels |
| Prediction | (Traditional) Predicted object class | - |
| Label | (Traditional) True object class | - |
| Feature (Pred) | (Deep Learning) Model-predicted features | (Height, Width) centimeters |
| Feature (True) | (Deep Learning) True feature value | (Height, Width) centimeters |
During detection, the resulting data can be accessed by the PLC. For illustration, in this project, the PLC used is PLC M221 from Schneider Electric. This PLC can be programmed with Ecostruxure Machine Expert - Basic software to receive data. Through this software, the detection data will be able to be observed and further processed.
Simple processing can be performed. For example, with predicted class index data, the PLC can be programmed to turn on I/O indicator lights based on class. Programming is done with Ladder Diagram. The changes to the PLC can be observed below. More complex processing can be performed as needed.
This data transmission is possible because the Raspberry Pi acts as a Modbus Server or Modbus Client. By default, the Raspberry Pi will run as a server. Nonetheless, the Modbus communication mode of the Raspberry Pi can be changed through program arguments.
Each dataset contains three classes. The first dataset is Color Ducky. The classes in this dataset are "yellow_duck", "pink_duck", and "blue_duck". This dataset contains 181 images. The second dataset is Ducky Chicken. This dataset has classes consisting of "duck", "chick", and "cock". This dataset contains 233 images. Meanwhile, the Ducky Frog dataset is only used in testing so it can be ignored. Samples for the two datasets mentioned earlier can be observed below.
All datasets can be obtained from the dataset folder. The datasets have been separated for the sake of training the deep learning model (train) and validating its performance (validation).
EfficientDet-Lite is a scalable model. By modifying hyperparameters such as input resolution, layer depth, and layer width, EfficientDet-Lite will experience a shift in performance and computational cost. From the model development process, five variants of EfficientDet-Lite have been produced. The variants and their performance comparison can be observed in the following table.
| Model | Parameters | Size (KB) | mAP (color) | mAP (shape) |
|---|---|---|---|---|
| EfficientDet-Lite0 | 3.239.711 | 4.342 | 0,315 | 0,320 |
| EfficientDet-Lite1 | 4.241.703 | 5.799 | 0,476 | 0,371 |
| EfficientDet-Lite2 | 5.236.415 | 7.220 | 0,400 | 0,348 |
| EfficientDet-Lite3 | 8.328.991 | 11.456 | 0,420 | 0,438 |
| EfficientDet-Lite4 | 15.110.223 | 20.068 | 0,502 | 0,401 |
The trained model can be retrieved from the model folder. The model name is based on the usage scenario and its variants. For example, the model color_detector2.tflite is the EfficientDet-Lite2 model trained for the color detection scenario.
In running this project, the Python libraries used can be observed in the requirements.txt file. However, when installing the tflite-support library, errors may occur. Therefore, the library has been provided in the lib folder. To access the Raspberry Pi GPIO, the RPi.GPIO library is required.
Before running the code in the repository, users need to set up a virtual environment with the required libraries. This code needs to be run on Raspberry OS.
First, check the version of Raspberry Pi OS.
cat /etc/os-releaseSecond, update the packages on Raspberry Pi OS.
sudo apt-get updateThird, check the Python version. Use Python 3.7 and above.
python3 --versionFourth, install virtualenv and upgrade pip.
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenvFifth, create a virtual environment to run the code.
python3 -m venv ~/tflite-pymodbusWhen the virtual environment is ready, activate the environment with the following command (this command needs to be called EVERY time you open a new Terminal).
source ~/tflite-pymodbus/bin/activateClone the repository from the project and change the active directory to the project directory.
git clone https://github.com/javanese-programmer/conveyor-object-detection.git
cd conveyor-object-detectionInstall the dependencies of the project. When everything is installed, the code can be run immediately.
python3 -m pip install pip --upgrade
python3 -m pip install -r requirements.txtBefore running the code, access to port 502 (Modbus port) needs to be granted to the program. This is because the port can only be accessed by the root user. Therefore, run the command to redirect port 502 to a higher port, such as 5020. Run this command every time you want to use the device as a Modbus server.
sudo iptables -t nat -A PREROUTING -p tcp --dport 502 -j REDIRECT --to-port 5020NOTE:
Before running the code, copy all the libraries in the lib folder and then paste them at the address ~/tflite-pymodbus/lib/python3.7/site-packages on the Raspberry Pi OS. This is because the process of installing these libraries with pip can cause errors.
The two main codes that can be used are run.py and capture.py. The first code will run the detection and establish the Raspberry Pi as a Modbus server that can send object detection data to the PLC. Meanwhile, the second will record the detection process and return both a video file and an image file.
python run.pypython capture.pyTo check the arguments that can be added to the code, add -h after each command.
python run.py -hpython capture.py -hThe code below will collect images from the conveyor WITHOUT performing object detection.
python collect.py- Raspberry Pi 4B
- Camera Module OV5647
- IR sensor
- Jumper cable
- Ethernet cable
- Breadboard
- LED
- Resistor 220 Ohm
- SunFounder 7-inch Monitor
- Keyboard
- Mouse
- HDMI to Micro-HDMI Cable (To connect the monitor)
- USB to Micro-USB Cable (To connect the monitor)
In a paper by O'Mahony, N et.al., traditional computer vision methods, such as color thresholding, can be used for problems that deep learning methods are not suitable for solving. One of the problems mentioned is the classification of products on an assembly line conveyor belt. Therefore, in this project, traditional and deep learning methods have been implemented and compared.
When the time taken to detect objects is compared, EfficientDet-Lite0 (the lightest variant of the EfficientDet model) costs 4 times as much as the traditional method. This cost will increase for other EfficientDet variants until it reaches 45 times for EfficientDet-Lite4. As a result, detection with the Deep Learning model will be slower as shown below.
Detection with Deep Learning is generally probabilistic so objects can be mislabeled or not detected at all. This is a shortcoming of Deep Learning methods compared to traditional methods. However, this program has been developed to handle these conditions.
This project shows that I have successfully developed a system to recognize objects on a conveyor for student training purposes. The developed system combines Raspberry Pi 4 and PLC with Modbus protocol. To recognize the objects, traditional computer vision methods and deep learning have been implemented.
In this case, traditional methods are indeed superior to Deep Learning. The reason is that the environment for object detection is constrained and fixed. However, in a more dynamic environment, the traditional methods used here would not be able to match the capabilities of Deep Learning.
