Infinite-ISP

Infinite-ISP is a full-stack ISP development platform - from algorithm development to RTL design, FPGA/ASIC implementation, and associated firmware, tools, etc. It offers a unified platform that empowers ISP developers to accelerate ISP innovation. It includes a complete collection of camera pipeline modules written in Python, an FPGA bitstream & the associated Firmware for the implementation of the pipeline on the Kria KV260 development board, and lastly, a stand-alone Python-based Tuning Tool application for the pipeline. The main components of the Infinite-ISP project are listed below:

Repository name	Description
Infinite-ISP_AlgorithmDesign	Python based model of the Infinite-ISP pipeline for algorithm development
Infinite-ISP_ReferenceModel ⚓	Python based fixed-point model of the Infinite-ISP pipeline for hardware implementation
Infinite-ISP_FPGABinaries	FPGA binaries (bitstream + firmware executable) for the Xilinx® Kria KV260’s XCK26 Zynq UltraScale+ MPSoCA
Infinite-ISP_Firmware	Firmware for the Kria kV260’s embedded Arm® Cortex®A53 processor
Infinite-ISP_TuningTool	Collection of calibration and analysis tools for the Infinite-ISP

Infinite-ISP Reference Model: A Python-based Model for RTL Implementation of Camera Pipeline Modules

Overview

Infinite-ISP Reference Model is a Python based fixed-point implementation of Infinite-ISP pipeline. It is a comprehensive collection of camera pipeline modules, designed to convert input RAW images from sensors to output RGB images. The model provides a reference implementation that aids in generating RTL code enabling rigorous testing, verification, and validation of the pipeline's functionality and behavior.

This model uses lookup tables for complex functions like Gaussian and Sigmoid, applies fixed-point numbers or custom approximations for divisions and square roots, optimizing for minimum loss in image quality.

In its current state, the model implements simple algorithm per module, with plans to incorporate RTL-friendly complex algorithms in future versions.

ISP pipeline for InfiniteISP_ReferenceModel v1.0

Contents

1. RTL-friendly Code: The model provides a Python implementation of camera pipeline modules that can be directly translated to RTL, with optimizations such as lookup tables, custom approximations, and integer calculations.

2. Test Vector Generation: The project includes automated scripts for generating test vectors for individual as well as multiple modules, simplifying the testing process.

3. Dataset Processing: The model facilitates execution for multiple images with different or same configuration files.

4. Video Processing: The model also features a video processing script that allows for sequential frame processing, with operational 3A Statistics data flowing between frames.

Objectives

The primary objective of Infinite-ISP_ReferenceModel is to create a open source Python-based Model that streamlines the process of implementing camera pipeline modules into RTL-friendly implementations. This enables seamless integration with hardware designs and simplifies the development of efficient image processing systems. By providing optimized algorithms, test vector generation, and video processing capabilities, the model aims to be an invaluable tool for developers working on image processing projects and RTL implementations.

Features List

The table below provides a feature list of the model. The version 1.0 of the model implements hardware-friendly and simple algorithms for each module.

Modules	Infinite-ISP_ReferenceModel
Crop	Crops image with Bayer pattern safe cropping
Dead Pixel Correction	Modified Yongji's et al, Dynamic Defective Pixel Correction for Image Sensor
Black Level Correction	Calibration / sensor dependent - Applies parameters from the config file tunable using the tuning tool
Optical Electronic Transfer Function (OECF)	Calibration / sensor dependent - Implements a LUT from config
Digital Gain	Gains from config file - In auto mode AE feedback is incorporated for digital gain selection
Bayer Noise Reduction	Green Channel Guiding Denoising by Tan et al - Implements Chroma and Spatial filters through LUTs
Auto White Balance	Enhanced Gray World - AWB stats calculations within an optimal threshold
White Balance	WB gains multiplication - Parameters from the config file tunable using the tuning tool
Demosaic	Malwar He Cutler’s demosaicing algorithm
Color Correction Matrix	Calibration / sensor dependent - 3x3 CCM from the config file tunable using the tuning tool
Gamma Correction	Implements a LUT from config
Auto Exposure	Auto Exposure - AE stats calculations based on skewness
Color Space Conversion	YCbCr digital - BT 601 - Bt 709
Noise Reduction	Non-local means filter - Implements intensity level difference through a LUT
RGB Conversion	Converts YCbCr digital image to RGB
Invalid Region Crop	Crops image to a fixed size
Scale	Nearest Neighbor - Integer Scaling
YUV Format	YUV - 444 - 422

Dependencies

The project is compatible with Python_3.9.12

The dependencies are listed in the requirements.txt file.

The project assumes pip package manager as a pre-requisite.

How to Run

Follow the following steps to run the pipeline

1. Clone the repo using

git clone https://github.com/10xEngineersTech/Infinite-ISP_ReferenceModel

2. Install all the requirements from the requirements file by running

pip install -r requirements.txt

3. Run isp_pipeline.py

python isp_pipeline.py

Example

There are a few sample images with tuned configurations already added to the project at in_frames/normal folder. In order to run any of these, just replace the config file name with any one of the sample configurations provided. For example to run the pipeline on Indoor1_2592x1536_12bit_RGGB.raw simply replace the config file name and data path in isp_pipeline.py

CONFIG_PATH = './config/Indoor1_2592x1536_12bit_RGGB-configs.yml'
RAW_DATA = './in_frames/normal/data'

How to Run on Pipeline on Multiple Images/Dataset

There are two scripts that run Infinite-ISP on multiple images:

1. isp_pipeline_dataset.py
   Execute multiple images. Raw image should have its own config file with name <filename>-configs.yml where <filename> is raw filename otherwise the default configuration file configs.yml is used.

2. video_processing.py
   Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from configs.yml and 3A Stats calculated on a frame are applied to the next frame.

After cloning the repository and installing all the dependencies follow the following steps:

1. Set DATASET_PATH to dataset folder. For example if images are in in in_frames/normal/data folder

DATASET_PATH = './in_frames/normal/data'

2. If your dataset is present on another git repository you can use it as a submodule by using the following commands in the root directory. In the command, <url> is the address of git repository such as https://github.com/<user>/<repository_name and <path> is the location in your repository where you want to add the submodule and for Infinite ISP <path> should be ./in_frames/normal/<dataset_name>. Please keep in mind that your <dataset_name> should not be data because directory in_frames/normal/data already exists.

git submodule add <url> <path>
git submodule update –-init –-recursive

4. After adding git repository as a submodule update DATASET_PATH variable in isp_pipeline_dataset.py to ./in_frames/normal/<dataset_name>. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify the DATASET_PATH variable in isp_pipeline_dataset.py or video_processing.py accordingly.

DATASET_PATH = './in_frames/normal/<dataset_name>'

5. Run isp_pipeline_dataset.py or video_processing.py
6. The processed images are saved in out_frames folder.

Test Vector Generation

Please refer to the provided instructions for generating test vectors for multiple images, considering individual or multiple modules as the Device Under Test (DUT).

Contributing

Please read the Contribution Guidelines before making a Pull Request

Results

Here are the results of this pipeline compared with a market competitive ISP. The outputs of our ISP are displayed on the right, with the underlying ground truths on the left.

           ground truths                            infiniteISP_RM

A comparison of the above results based on PSNR and SSIM image quality metrics

Images	PSNR	SSIM
Indoor1	22.5788	0.8923
Outdoor1	19.1544	0.9048
Outdoor2	18.8681	0.8071
Outdoor3	17.2825	0.7304
Outdoor4	19.9814	0.8198

User Guide

For more comprehensive algorithms documentation and to understand how to use the python model, please visit the User Guide.

License

This project is licensed under Apache 2.0 (see LICENSE file).

Contact

For any inquiries or feedback, feel free to reach out.

Email: isp@10xengineers.ai

Website: http://www.10xengineers.ai

LinkedIn: https://www.linkedin.com/company/10x-engineers/