Day & Night

Controllable Extendable Depth of Field with Fully Open Aperture

A surveillance system with large depth of field in day and night conditions is now reality through ultra high speed volumetric sweeping.

Image 1 | One significant advantage of instantaneous 2D projection with the Tag lens is that no data processing is involved to create the 2D composed image: everything is performed at the hardware level and as such, each camera pixel perceives the summation of the volumetric sweep. (Image: TAG Optics Inc.)

Traffic monitoring and surveillance systems or other vision technologies based on camera lenses must all find a compromise between depth of field and the amount of light that reaches the sensor (aperture setting). For many vision systems however, particularly those equipped with telephoto lenses, shallow images are problematic as objects outside of the depth of field will be out of focus and blurry. Consequently, many of these systems operate with high F#, where the aperture is largely closed, in order to generate an acceptable depth of field, but significantly reducing the amount of light that reaches the sensor. For systems that need to operate in bright and low-light conditions, this can be particularly problematic as the user must make a choice between an acceptable depth of field and sufficient photon collection. As a result, monitoring systems often use multiple cameras, one of which has a high sensitivity sensor dedicated for low light conditions. However, having a sensors with high quantum efficiency does not alleviate the need for large depth of fields, which once again curtails the amount of light that reaches the sensors. As such, the added sensitivity simply acts as a compromise: while it may facilitate operating in low light conditions with high F#, additional photons would still be highly beneficial.

Change of Focus in Nanoseconds

The TAG Lens technology provides a unique value proposition to such dilemmas encountered by technologists who requires their vision systems to operate in both day and night conditions. The TAG Lens has the ability to change focus in nanoseconds, taking at its slowest a mere 7µs to sweep through an entire volume. In fact, the resulting (liquid) lens continuously changes from a converging to a diverging wavefront, going through infinity. When paired with a camera operating at 10s to 100s of fps, such a variable focus element sends photons to the sensor that were gathered by performing 1,000s of volumetric scans per exposure. This results in a complex point spread function where all objects present in the volume are both in focus and out of focus at effectively the same time. Consequently, each frame 2D projections of the 3D volume effectively instantaneously, thanks to the temporal difference between the TAG Lens‘ focusing speed the sensor’s exposure time. The integration of the Lens can enable the use of a camera lens with fully open aperture in any vision systems relying on the new technology’s capability to control the system’s depth of field; thereby, maximizing photon collection while not being limited by shallow images. A similar embodiment was first presented in 2017 as the TAG ZIP, a 3D volumetric technology where the camera lens was synchronized to obtain depth information through high speed focus stacking. This present embodiment of the technology represents a much simpler setup where the TAG Lens is simply added in the optical path between the sensor and the camera lens with no synchronization or other complex techniques involved.

Image 2 | Because of a point spread function all objects present in the volume are both in focus and out of focus at effectively the same time. (Image: TAG Optics Inc.)

No Data Processing

One significant advantage of this type of instantaneous 2D projection is that no data processing is involved to create the 2D composed image. Everything is performed at the hardware level and as such, each camera pixel perceives the summation of the volumetric sweep. However, this methodology does create a drawback: since the image is the summation of an entire volume, when objects are in focus they contribute high contrast information which is then added to low contrast information contributed when the objects is out of focus during the volumetric sweep. The resulting image is one where a small defocus blur can be observed around all elements present in the field of view, similar to a Gaussian blur. This slightly reduces the signal to noise ratio due to the systems elongated point spread functi

on which is generated by the summation of a shallow image that is continuously swept through the volume. However, thanks to the properties of the TAG Lens which uses sound to shape light, compensation through image filters is possible. The TAG Lens is a variable gradient index of refraction lens with a parabolic wavefront that is operated in a closed loop system enabling high stability and fidelity. This enables automatic real-time compensation for any change in temperatures or any other environmental conditions. Moreover, as it operates at frequencies ranging from 70 to over 300kHz, it is highly resistant to vibrations and physical shocks. This stability of operation yields a high degree of predictability and therefore it is possible to create custom image filters enabling specific machine vision capabilities.

Specifically, the continuous knowledge of the system’s specific temporal point spread function can be used to create custom image filters. Such algorithms have the ability to reduce much of the blur by compensating for the defocus effect created by volumetric sweeping, but they can also be used in tandem with other standard machine vision algorithms or even serve as pre-processing steps. Thanks to recent advancement in mobile graphic computing , it is possible to operate these filters in quasi real-time, with internal prototypes reaching refresh rates near 20Hz.

Conclusion

The TAG lens technology results in a simple one camera system which combines the benefits of having a large aperture camera lens (F# <2.0), such as high resolution and large photon collection abilities, while answering the need for large, controllable depth of field. By operating this particular embodiment of the TAG technology along with the latest advancement in graphical processing technologies, it is possible to enable surveillance systems such as traffic monitoring to perform shape, object, or text recognition across both day and night conditions with one optical hardware.