Image 2 | The result of the exposure measurement is asynchronously output off the sensor together with the pixel’s coordinates in the sensor array. (Image: Prophesee)
Autonomously Operating Pixels
Following this paradigm, Prophesee has developed an image sensor containing an array of autonomously operating pixels that combine an asynchronous level-crossing detector with a separate exposure measurement circuit. Each exposure measurement by an individual pixel is triggered by a level-crossing event. Inspired by biology, every pixel in these sensors optimizes its own sampling depending on the visual information it sees. In case of rapid changes, the pixel samples at a high rate. On the contrary, if nothing happens, the pixel stops acquiring redundant data and goes idle until things start to happen again in its field of view. Hence each pixel independently samples its illuminance upon detection of a change of a certain magnitude in this same luminance, thus re-measuring its new light level after it has changed. The result of the exposure measurement (i.e. the new gray level) is asynchronously output off the sensor together with the pixel’s coordinates in the sensor array. As a result, image information is not acquired and transmitted frame-wise but continuously, and conditionally only from parts of the scene where there is new visual information. Or in other words, only information that is relevant – because unknown – is acquired, transmitted, stored and processed by machine vision algorithms. This way, both the acquisition of highly redundant and useless data by over-sampling static or slow parts of the scene, and the under-sampling of fast scene dynamics due to fixed frame rates, can be eliminated.
Image 3 | The results are no longer a sequence of images but a time-continuous stream of individual pixel data, generated and transmitted conditionally, based on what is happening in the scene. (Image: Prophesee)
Pixel acquisition and readout times of milliseconds to microseconds are achieved, resulting in temporal resolutions equivalent to conventional sensors running at tens to hundreds of thousands of frames per second. Now, for the first time, the strict temporal resolution vs. data rate tradeoff that limits all frame-based vision acquisition can be overcome. As the temporal resolution of the image data sampling process is no longer governed by a fixed clock driving all pixels, the data volume of the sensor output, independently of the temporal resolution available for the acquisition at the single pixel, is only depending on the dynamic contents of the visual scene. Visual data acquisition simultaneously becomes fast and sparse, leading to ultra-high-speed acquisition combined with reduced power consumption, transmission bandwidth and memory requirements.
