CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`
File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/src/matsci_opt_benchmarks/crabnet_hyperparameter/core.py", line 219, in matbench_metric_calculator
cb.fit(train_df=train_df)
File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/src/matsci_opt_benchmarks/crabnet_hyperparameter/core.py", line 104, in evaluate
results = matbench_metric_calculator(parameters, dummy=dummy)
File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/scripts/crabnet_hyperparameter/crabnet_hyperparameter_submitit.py", line 147, in <module>
evaluate(parameter_batch_sets[0][0], dummy=True)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`