With the increasing adoption of structured-light and ToF sensing devices in smartphones, tablets, and other devices, the hardware foundation for 3D perception has gradually matured. The 3D market is expected to shift from an emphasis on imaging to an emphasis on perception. Perception, in turn, imposes higher requirements on the reliability and accuracy of imaging. As an important bridge connecting sensors and 3D perception applications, 3D ISP, namely Image Signal Processing, can effectively eliminate interference and improve real-time performance. Therefore, how to construct an efficient ISP architecture and functional components is a key issue in the development of ISP technology.
Figure 1. Functional Block Diagram of 2D ISP and 3D ISP Technologies
2D ISP technology has become relatively mature. However, due to the differences in 3D imaging, the core computational operations of 3D ISP differ significantly from those of 2D ISP, making it impossible to directly adopt 2D ISP technology. Compared with ordinary 2D image sensors, 3D image sensors involve far more control parameters, imaging computation, calibration complexity, and application difficulty. Based on 3D ToF and ISP technologies, this project investigates high-performance and low-power hardware processing algorithms and architectures in the RAW domain, depth domain, and point-cloud domain, which is of critical importance for improving 3D image quality and promoting the development of 3D applications.
Some hardware IP modules have been preliminarily implemented, including depth imaging, pseudo-color conversion, depth-map filtering and hole filling, and depth-to-point-cloud conversion, and verification has been completed on a development board. The next step will focus on memory access optimization and architectural design for point-cloud processing. Meanwhile, by integrating RAW-domain and depth-map processing, the ISP system will be gradually improved, further research on ISP will be carried out, and a 3D ISP chip will ultimately be developed to advance the development of 3D imaging.
Depth（左）Pseudo-color（右）
Depth Map (Left) and Point Cloud (Right)
Before (Left) and After (Right) Defective Pixel or Micropore Restoration
Before (Left) and After (Right) Depth Map Filtering
