Asynchronous execution about accelerate HOT 4 OPEN

Asynchronous execution entails using non-default stream(s) and event waiting for dependencies.

With support for streams and events, we should also (correctly) support asynchronous memory transfer, which additionally requires:

• The host memory is pinned, so the CUDA driver can do a DMA. Currently Accelerate (base) allocates in pageable memory that is pinned only with respect to the Haskell RTS's GC. Internally, the CUDA driver must copy the data to a pinned region, before performing the DMA.
• If data transfers and kernels operate in distinct non-default streams these will also overlap on all devices which support the feature (almost all 1.1 and later devices).

from accelerate.

See also:

• https://developer.nvidia.com/content/how-optimize-data-transfers-cuda-cc
• https://developer.nvidia.com/content/how-overlap-data-transfers-cuda-cc

from accelerate.

Note: this issue is further discussed in June/July 2014 on the accelerate mailing list here.

from accelerate.

This is all possible now, just not exposed very nicely yet. See this profiler output, where compute and data transfer overlaps nicely with full-speed DMA to pinned memory:

Also note this example however, where the CUDA pinned memory allocator is (a) not concurrent, and (b) can be teeeerribly slow:

So we may want to do a nursery-style caching allocator. These screenshots are from different machines, and the latter is a 2-GPU box, so may have further strangeness going on...

from accelerate.