MIPI CSI-2 for Machine Vision Applications

The use of embedded camera modules has enabled a wealth of imaging applications across market sectors globally. Mobile remains the largest of those, but huge embedded-camera market growth is also evident across numerous other sectors, including consumer electronics, computing, automotive, industrial, medical, security and defense/aerospace. The MIPI Alliance’s engagement in the mobile market since the arrival of the first camera phones has resulted in wide adoption of MIPI Camera Serial Interface 2 (MIPI CSI-2) within the broader embedded-camera market. MIPI members have invested in the CSI-2 camera manufacturing infrastructures for numerous sensor classes with highly optimized and evolved engineering processes. Consequently, the core of the CSI-2 specification – the frame format for transfer of pixel array data from image sensors – has become nearly ubiquitous within embedded camera modules. Today it is regarded as the world’s most widely embedded camera and imaging interface.

Future machine vision (MV) applications will become smarter, leveraging the latest sensor technology in combination with rapid advancements of AI-assisted inferencing targeting multiple platforms. To keep CSI-2 at the center of future applications, MIPI is exploring key emerging MV use cases and considerations:

• AI-assisted cinematography, utilizing seamless multi-image sensor transition on flagship mobile phones
• Enhanced motion vector analysis to enable real-time depth perception and trajectories of multiple objects
• Harmonized packet header provisions to help alleviate MV system integration challenges
• Wire-efficient and emissions-resilient, long-reach physical transport options to enable MV applications in beyond-mobile platforms (for example, robotic vision, client video conferencing, surveillance applications, automotive driver assistance and factory automation)
• Support for multiple sensors on one interface to help optimize the number of high-speed input/output (HSIO) pins on MV application processors
• Definition of an efficient and scalable CSI-2 controller backend interface for image signal processors (ISPs) with support for linear transport and high dynamic range (HDR) processing on MV systems.

With image sensors continually evolving to support greater pixel resolution, color depth, light sensitivity and spectrum, the volume of data generated by images sensors is always growing, necessitating ever-more powerful vision compute capabilities. The MIPI Camera Working Group’s years of expertise in developing sophisticated camera solutions for premium mobile phones positions CSI-2 as the de facto protocol for emerging MV applications demanding high performance, low power and provisions for infer-encing.

Supporting MV Applications

Over the past 20 years, the MIPI Camera Working Group has consistently refined and updated CSI-2 to deliver the new features paving the way for systems designers‘ next required capabilities in MV across a wide range of scenarios – from applications demanding top-end image quality and highest speeds, to those prioritizing lowest power and processing overhead. Key features have been added since CSI-2’s introduction in 2005 and are reshaping MV’s horizons:

• Differential Pulse Code Modulation to enable compression at reduced bandwidth and deliver superior image quality and pristine edge detection
• Latency Reduction and Transport Efficiency (LRTE) Packet Delimiter Quick (PDQ) and Alternate Low Power (ALP) capabilities to help aggregate sensors and reduce system power
• Power Spectral Density (PSD) reduction based on Galois Field 2^16 polynomial scrambling to help reduce emissions and interference
• Smart Region of Interest (SROI) to analyze images using inference algorithms to execute better deductions in multiple application spaces
• Unified Serial Link (USL) for wire reduction by alleviating the need for a physically separate I2C/I3C/GPIO side channel to offer encapsulated data and control transport over MIPI D-PHY, C-PHY, and A-PHY.

The most recent major update of CSI-2, version 4.0 released in 2022, marked a milestone in that it was the first version of the specification to support CSI-2 image transmission over the low-cost, low-pin-count MIPI I3C/I3C Basic two-wire interface.

Environmental Monitoring

CSI-2 v4.0’s Always-On Sentinel Conduit (AOSC) feature enables a combination of ultra-low-power image sensors and a vision digital signal processor (VDSP) to constantly monitor surroundings and wake a higher-power, host central processing unit (CPU) when AO events demand. This capability, requiring as few as two I3C wires, cuts the cost and complexity of ultra-low-power awareness/inferencing solutions and MV applications, with event detection among emerging use cases. CSI-2 image frames can be inexpensively and efficiently streamed from image sensors to VDSPs over the low-power I3C bus in applications that require low-to-medium image resolutions. I3C lanes and bandwidth can be added for scaling options, as defined within the I3C/I3C Basic specifications.

Multi Pixel Compression

Image sensor resolutions are ever growing in resolution and, therefore, require more data to be sent from the sensor to the SoC. For example, leading-edge tetracell and nonacell image sensors with multi-pixel color filter arrays (CFAs) enable video capture on mainstream mobile devices at a cinematographic quality that not long ago was limited to commercial movie making. This growth was the driver behind CSI-2 v4.0’s multi pixel compression (MPC) feature. By enabling compression of multi pixel and standard Bayer CFA images at greater efficiency and quality than otherwise possible, MPC is designed to address the bandwidth demand of state-of-the-art sensors. This is partially accomplished by factoring the statistical probability that neighboring pixels within a given image will share similar characteristics, while meeting Subjective Image Quality vector requirements and reducing overall system power. This renders CSI-2 v4.0 more valuable to designers of emerging premium imaging applications with the stiffest requirements around power-signal-to-noise ratio (PSNR) and dynamic range (DR).

RAW-28 Color Depth

Given that the stakes in automotive applications include human safety, requirements around image quality and superior PSNR and DR in this space are very high. An advanced driver-assistance system (ADAS) must discern and appropriately react to the surroundings of a vehicle in motion and in all environmental conditions. The RAW-28 color depth feature introduced in CSI-2 v4.0 is a method for gathering more and better information for such a process. RAW-20 and then RAW-24 were featured in previous CSI-2 versions. With introduction of RAW-28 pixel encoding, the MIPI Camera Working Group leverages the high-DR extraction techniques of next-generation MV image sensors.

Shaping ongoing Innovation

MIPI CSI-2 v4.0.1 was published in 2022, delivering minor editorial clarifications, including revisions for inclusive terminology that more accurately reflect the functions of technical devices. The MIPI Security Framework, a suite of four specifications initially targeted at automotive applications requiring end-to-end protection of CSI-2 data streams, is scheduled for release in late 2023 and early 2024. The MIPI Camera Working Group continues to monitor MV innovation in planning updates to CSI-2. Multi-sensor support and extended virtual channels, for example, are under exploration. Furthermore, MIPI has launched a Camera Imaging and Vision Interest Group, open to all MIPI Alliance members, to explore emerging MV applications.

CSI-2 for MV Ecosystems