Speed up Real-Time Image Processing With GPU Software
With the Fastvideo SDK data processing is done on Nvidia GPUs to speedup the performance. From tests with Nvidia Quadro RTX 6000 or GeForce RTX 2080ti it can be seen that GPU-based raw image processing is very fast and it could offer high image quality at the same time. The total performance could reach 4GPix/s for color cameras.
Image 1 | At the Menzi Muck remote-controlled walking excavator the Fastvideo SDK solution allowed up to 60fps of synchronized image acquisition from each of the two 3.1MP cameras. (Image: ETH Zurick Inst. f. Robotik u. Intell. Syst.)
Machine vision cameras are widely used in industry, science, and robotics. However, when working with them, the same question invariably arises: „How to process the data received?“ But why does it arise at all? The point is that cameras usually transmit raw data (RAW) at high frame rate, which takes up a lot of memory and needs to be converted to the required image format in real time. Image processing algorithms must provide the quality and speed necessary for the task at hand. Unfortunately, it is sometimes not easy to ensure both quality and speed at the same time. That’s why, whenever there’s a task which requires processing a lot of images in real time, experts put a high priority on optimizing computer vision-related algorithms. It’s even more important when there’s a limited budget, or the physical size or power consumption of the device is constrained for practical reasons. Generally, high-quality algorithms that perform computations on Intel/AMD processors do well with this task. However, there are special cases: the processing of images from high data rate machine vision cameras, which is the case for high image resolution or a high frame rate or multi-camera system with real-time image processing. For such situations, the capabilities of a CPU are not enough. CPU just can’t handle the huge data stream quickly enough (for example, when it’s dealing with gigapixels per second), and this leads to the unavoidable loss of some data. Unfortunately, it’s difficult to speed things up further on a CPU without a trade-off for quality. So, how can we speed up image processing without losing quality?
Image 2 | The SDK allows real-time processing with H.264/H.265 encoding and streaming and glass-to-glass video latency over 4G/5G network with nearly 50ms at the Menzi Muck. (Bild: Fastvideo LLC)
From CPU to GPU
The main idea for the solution was to transfer most of the computations from the central processor (CPU) to the graphics processor (GPU). To solve that task, Fastvideo developed the Fastvideo SDK, which works on Nvidia GPU. This approach has accelerated the necessary algorithms and simplified the software architecture, because computations in this case no longer interfere with system functions based on the CPU. Let’s look at the advantages of image processing on a GPU instead of a CPU:
- • A graphics card is a more specialized device than a CPU, and due to its architecture, it can perform many tasks much faster.
- • A graphics processor can offer significantly faster memory access than a central processor.
- • The algorithms can be run in parallel at graphics cards.
Transferring computations to the graphics card does not mean that the CPU is completely free. The CPU is responsible for I/O and system control. The proposed solution is heterogeneous, since it uses all the available resources of both the CPU and GPU for image processing, which in turn leads to high performance. In addition to increasing the speed of image processing, using a graphics processor has allowed Fastvideo to implement more complicated algorithms to increase the image quality and color reproduction. The workflow is similar to that used in filmmaking, where the colors in the frame are given special attention.
Image 3 | The wind turbine inspection drone from Alerion is intended to fully automate the inspection of wind turbines for damage. The time spent on inspection of one turbine was reduced from 2h to 10min. (Image: Alerion)
Walking Excavator And Drones
One of the best examples where the Fastvideo SDK this solution can be applied is image processing for Ximea cameras. The cameras are used, for example, in the Menzi Muck remote-controlled walking excavator. For this particular project, the Fastvideo SDK solution allowed up to 60fps of synchronized image acquisition from each of the two 3.1MP cameras; real-time processing with H.264/H.265 encoding and streaming (including black level, white balance, demosaicing, auto exposure, etc.) and glass-to-glass video latency over 4G/5G network with nearly 50ms. Another project using Ximea cameras is the wind turbine inspection drone from Alerion. This drone is intended to fully automate the inspection of wind turbines for damage. For this task, it is important to ensure good quality of images, based on which a 3D model is subsequently built. Using the cameras in conjunction with the GPU image processing solution made it possible to achieve the required image quality and high processing speed, which in turn made it possible to automate the inspection process. As a result, the time spent on inspection of one turbine was reduced from 2 to 3h to 10min. Here, of course, process automation played a big role. However, this would not have been possible without high processing speed and excellent image quality that allows even very small damage to be noticed. The Fastvideo SDK for GPU image processing can work on both stationary and mobile graphics cards.
