Control Divergence

This is a concept that I was introduced through PMPP.

When threads in the same warp follow different execution paths, we say that these threads diverge in their execution.

If-Else

In the if-else example, divergence arises if some threads in a warp take the if path and some the else path.

What is the cost of divergence?

The cost of divergence is the extra pass the hardware needs to take in order to allow the threads in a warp to make their own decisions.

Why is this important? Well oftentimes, you are not working with imperfect size vectors, in which case you need to handle boundary conditions.

For a vector length of 100, one of the four warps will have control divergence (since you need 32 threads/warp * 4 warp = 128 threads > 100), which can have significant impact on performance. For a vector size of 1000, only one out of the 32 warps will have control divergence (you need ) The performance impact of control divergence decreases with the size of the vectors being processed.