eth-cscs / tiled-mm Goto Github PK

• Overview
• Performance
• Features
• Building and Installing
• Minimal Working Example
• Running the Benchmarks
• Testing
• Author

Tiled-MM is a very fast and easy-to-use library for multiplying matrices on GPU. As opposed to NVIDIA's cublas, this library takes pointer from the host side (CPU), splits the matrices into tiles, pipelines them efficiently to the GPU and copies the result back to the CPU. It can serve as almost a drop-in replacement for cublasXt, and is ported to both NVIDIA and AMD gpus.

It offers more features than the standard cublas API. For example, the user can specify the number of gpu streams to be used, as well as the tile size for each dimension separately, which is not possible with the standard cublas API.

Tiled-MM is used in production as a backend of the COSMA algorithm and is thus well-tested.

The benchmarks were performed on a single node of Piz Daint Supercomputer (Cray XC50), equipped with a P100 NVIDIA GPU. We compared the performance of our library Tiled-MM with the vanilla version of cublasXt and also with the manually tuned version of cublasXt, where we manually set the tile size to 4000 and enabled the pinned memory mode. Tiled-MM was substantially faster than the vanilla version of cublasXt, and achieved similar performance as the manually tuned version of cublasXt, as can be seen from the results below.

In the benchmark, we used double precision, square matrices given in column-major ordering, and alpha = beta = 1.0.

• The user can specify the tile size of each dimension separately.
• The user can specify the number of streams to be used.
• The user can reuse the same context (and thus the same device memory) for many multiplications which can lead to significant performance improvements.
• Fully templatized, supporting arbitrary data types.
• Ported to both NVIDIA and AMD GPUs.

Assuming that you want to use the gcc 8 compiler, you can build the project as follows:

# clone the repo
git clone https://github.com/eth-cscs/Tiled-MM
cd Tiled-MM
mkdir build
cd build

# build
CC=gcc-8 CXX=g++-8 cmake -DTILEDMM_GPU_BACKEND=CUDA ..

# compile
make -j 4

When building the examples cxxopts is required. It is available in most package manager, apt-get install libcxxopts-dev (ubuntu) or brew install cxxopts (macos).

The option -DTILEDMM_GPU_BACKEND can have the following values:

• CUDA: for NVIDIA GPUs
• ROCM: for AMD GPUs

Using the library is very simple, just include #include <tiled_mm.hpp> and use it as follows:

// A dimensions: m x k
auto a_host = gpu::malloc_pinned<double>(m * k, 1);
// B dimensions: k x n
auto b_host = gpu::malloc_pinned<double>(k * n, 1);
// C dimensions: m x n
auto c_host = gpu::malloc_pinned<double>(m * n, 0);

double alpha = 1.0;
double beta = 0.0;

// preallocates device buffers and other CUDA stuff
// the context does not have to be created explicitly
// so the user can omit this part
auto ctx = gpu::make_context();

// compute c = alpha * a * b + beta * c
// There is also a version without ctx, in case the user
// does not want to create the context explicitly
gpu::gemm(*ctx,
          trans_a, trans_b,
          m, n, k,
          alpha,
          a_host, ld_a,
          b_host, ld_b,
          beta,
          c_host, ld_c);

// optionally, we can set the following two boolean flags
bool pin_buffers = false; // since a_host, b_host and c_host are already pinned, gpu::dgemm should not pin them
bool copy_c_back = true;  // if we want to copy the result back to the host or leave it on the gpu
gpu::gemm(*ctx,
          trans_a, trans_b,
          m, n, k,
          alpha,
          a_host, ld_a,
          b_host, ld_b,
          beta,
          c_host, ld_c,
          pin_buffers, copy_c_back);

// if copy_c_back == false, the result is stored on the device with the following pointer:
double* c_device = ctx->get_full_device_buffer_c().data()

When creating the context, the user can specify tile dimensions and the number of streams to be used as:

int tile_size_m = 5000;
int tile_size_n = 5000;
int tile_size_k = 5000;
int n_streams = 2;

auto ctx = gpu::make_context(n_streams, tile_size_m, tile_size_n, tile_size_k);

For detailed benchmarking, there is a miniapp that takes the host pointers for A, B and C and computes C = beta * C + alpha * A * B outputing the time-to-solution, as well as the throughput.

The miniapp consists of the executable ./build/examples/multiply which can be run with the following command line (assuming we are in the root folder of the project):

./build/examples/multiply -m 10000 -n 10000 -k 10000 -r 1

The overview of all supported options is given below:

For example, running with the following flags:

./build/examples/multiply -m 1000 -n 1000 -k 1000 --transpose=TN -r 1

should produce the following output:

==================================================
                Benchmarking Tiled-MM
==================================================
         MATRIX SIZES
=============================
 A = (1000, 1000)
 B = (1000, 1000)
 C = (1000, 1000)
=============================
         LEADING DIMS
=============================
 LD_A = 1000
 LD_B = 1000
 LD_C = 1000
=============================
      SCALING CONSTANTS
=============================
 alpha = 1
 beta  = 1
=============================
      TRANSPOSE FLAGS
=============================
 trans_a = T
 trans_b = N
=============================
         TILE SIZES
=============================
 tile_m = 5000
 tile_n = 5000
 tile_k = 5000
=============================
      ADDITIONAL OPTIONS
=============================
 num. of gpu streams = 2
 num. of repetitions = 1
=============================

==================================================
         Results of benchmarking Tiled-MM
==================================================
 1) The version with copying C to back to host:
    -> Avg Time [ms] = 11
    -> Throughput [Gflops] = 181.818
==================================================
 2) The version without copying C to back to host:
    -> Avg Time [ms] = 10
    -> Throughput [Gflops] = 200
==================================================

For testing purposes, there is a testing miniapp that generates random matrices A, B and C, computes C = beta * C + alpha * A * B with Tiled-MM as well as with blas and outputs whether the results are correct.

The miniapp consists of the executable ./build/tests/test-multiply supports the same parameters as the benchmarking miniapp (see above). It can be run e.g. with the following command line (assuming we are in the root folder of the project):

./build/tests/test-multiply -m 1000 -n 1000 -k 1000 --transpose=TN

which should produce the following output:

==================================================
                Benchmarking Tiled-MM
==================================================
         MATRIX SIZES
=============================
 A = (1000, 1000)
 B = (1000, 1000)
 C = (1000, 1000)
=============================
         LEADING DIMS
=============================
 LD_A = 1000
 LD_B = 1000
 LD_C = 1000
=============================
      SCALING CONSTANTS
=============================
 alpha = 1
 beta  = 1
=============================
      TRANSPOSE FLAGS
=============================
 trans_a = T
 trans_b = N
=============================
         TILE SIZES
=============================
 tile_m = 5000
 tile_n = 5000
 tile_k = 5000
=============================
      ADDITIONAL OPTIONS
=============================
 num. of gpu streams = 2
 num. of repetitions = 1
=============================
Time [ms] with copying C back: 11
Time [ms] without copying C back: 10
The result is CORRECT

Running make test will few default tests.

Marko Kabic ([email protected])