Vision Programming Interface (VPI)

Kajanan mentioned this to me. It’s like a better CUDA accelerated OpenCV. However, there is also CV-CUDA.

Installation

sudo apt install nvidia-vpi

• Will be installed at /opt/nvidia/vpi2

For some reason, it didn’t fully install. Do

sudo apt install libnvvpi2 vpi2-dev vpi2-samples

NOTE: Always reference VPI’s official documentation. My stuff is easily outdated.

Resources

• https://docs.nvidia.com/vpi/index.html
• https://docs.nvidia.com/vpi/bench_comparison.html

https://www.youtube.com/watch?v=6-FhpfwBpXk&ab_channel=NVIDIADeveloper

Questions:

• I have some confusion about streams. Why do we have to specify the backend for the stream when we do vpiCreateStreamWrapper, but not vpiStreamCreate. Why the difference?
• In cudaMalloc, you can specify the stream. However, vpiImageCreate doesn’t require you to define the stream?

Basics

Allocating a VPIImage

CHECK_STATUS(vpiImageCreate(cols, rows, VPI_IMAGE_FORMAT_U8, 0, &image));

Free a VPIImage

vpiImageDestroy(image);

Few ways to create images

• vpiImageCreate
• vpiImageCreateWrapper (from wrapping VPIImage around some other data)
• vpiImageLockData / vpiImageUnlock (for getting the raw pointer from VPI, wrap this around another data structure)

VPIImageFormats

I realize that an image format is tightly coupled with how it is stored in memory. So what are the different formats?

The higher level thing is this

/** Represents the available methods to access image contents.
 * The correct method depends on \ref VPIImageData::bufferType. */
typedef union VPIImageBufferRec
{
    /** Image stored in pitch-linear layout.
     * To be used when \ref VPIImageData::bufferType is:
     * - \ref VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR
     * - \ref VPI_IMAGE_BUFFER_CUDA_PITCH_LINEAR
     */
    VPIImageBufferPitchLinear pitch;
 
    /** Image stored in a `cudaArray_t`.
     * To be used when \ref VPIImageData::bufferType is:
     * - \ref VPI_IMAGE_BUFFER_CUDA_ARRAY
     */
    cudaArray_t cudaarray;
 
    /** Image stored as an EGLImageKHR.
     * To be used when \ref VPIImageData::bufferType is:
     * - \ref VPI_IMAGE_BUFFER_EGLIMAGE
     */
    EGLImageKHR egl;
 
    /** Image stored as an NvBuffer file descriptor.
     * To be used when \ref VPIImageData::bufferType is:
     * - \ref VPI_IMAGE_BUFFER_NVBUFFER
     */
    int fd;
 
} VPIImageBuffer;

https://docs.nvidia.com/vpi/group__VPI__Image.html#unionVPIImageBuffer https://docs.nvidia.com/vpi/group__VPI__ImageFormat.html

So for grayscale, they only have 1 channel

 #define VPI_IMAGE_FORMAT_U8 VPI_DETAIL_MAKE_NONCOLOR_FMT1(PL, UNSIGNED, X000, X8)
 
#define VPI_IMAGE_FORMAT_U8_BL VPI_DETAIL_MAKE_NONCOLOR_FMT1(BL, UNSIGNED, X000, X8)

They have something like NV12

 #define VPI_IMAGE_FORMAT_NV12 VPI_DETAIL_MAKE_YCbCr_FMT2(BT601, 420, PL, UNSIGNED, XYZ0, X8, X8_Y8)
 
 #define VPI_IMAGE_FORMAT_NV12_ER_BL VPI_DETAIL_MAKE_YCbCr_FMT2(BT601_ER, 420, BL, UNSIGNED, XYZ0, X8, X8_Y8)

VPI Backends

The backends: https://docs.nvidia.com/vpi/architecture.html#arch_backend

1. VPI_BACKEND_CPU: Utilizes the CPU for processing
2. VPI_BACKEND_CUDA: Leverages NVIDIA’s CUDA technology for GPU processing
3. VPI_BACKEND_PVA: PVA (Parallel Vector Architecture) backend, optimized for vectorized operations
4. VPI_BACKEND_VIC: VIC (Video Image Compositor) backend, used for image compositing tasks
5. VPI_BACKEND_NVENC: NVENC backend, focuses on video encoding using NVIDIA’s hardware acceleration
6. VPI_BACKEND_OFA: OFA backend, specific purpose not clearly defined
7. VPI_BACKEND_INVALID: Represents an invalid or unspecified backend

Read this, really good example of how you can split into different streams:

• https://docs.nvidia.com/vpi/architecture.html#arch_backend

Related

• CV-CUDA