Comments (13)
This all makes sense! The idea is that I ended up choosing top-5 to increase the prediction accuracy. But if you benchmark multiple kernels that modify one of their inputs, then you need to be careful to copy back the result. Can you confirm that setting N_TOP=1 resolves the issue?
from triton.
Hi, it still failed when setting N_TOP=1
from triton.
@ptillet could you reproduce this issue at your BDW machine?
from triton.
Yes, I will try this tomorrow. I suspect it should come from copying back the output's result. Perhaps a problem with queues. In the meantime, could you try N_TOP=1 and set modify_output = false ?
from triton.
Sorry, I still failed when setting N_TOP=1 and modify_output = false on my side
from triton.
Haha. That sounds pretty bad! I'll definitely take a look at it.
from triton.
Some updates: I've found a bunch of other bugs when trying to reproduce the issue with caffe. I've fixed some things, and I'm tuning ISAAC for intel's latest driver and double-precision on broadwell. I hope to have everything up and running by the end of the week-end.
from triton.
Could you try the latest master? It should not only fix bugs for clCaffe, but also add double-precision support and performance improvements with the latest Intel OpenCL 2.0 driver.
Caffe seems to work fine with ISAAC on my BROADWELL machine.
from triton.
@ptillet I tried the latest master with clcaffe. The test suite will not crash now, but still has some failures:
[ FAILED ] SGDSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithWeightDecayMultiIter, where TypeParam = caffe::GPUDevice
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaGradSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] NesterovSolverTest/2.TestNesterovLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaDeltaSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice
[ FAILED ] AdaDeltaSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] AdamSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] AdamSolverTest/2.TestAdamLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] RMSPropSolverTest/2.TestRMSPropLeastSquaresUpdateWithEverything, where TypeParam = caffe::GPUDevice
[ FAILED ] RMSPropSolverTest/2.TestSnapshot, where TypeParam = caffe::GPUDevice
[ FAILED ] InnerProductLayerTest/2.TestGradientTranspose, where TypeParam = caffe::GPUDevice
And if I ran some of the test cases directly, such as :
test/test.testbin --gtest_filter=SGDSolverTest/2.TestLeastSquaresUpdateWithEverything
It may fail or success at random manner. But If I use the old version:
commit 6ac5e1f55b1cae59394758f823d5c58f57ca561d
Author: Philippe Tillet <[email protected]>
Date: Fri Jan 1 05:44:28 2016 -0500
Templates/Reduce1D: now properly loading 2D scalars, it always passes all of the float gpu test cases.
It always passes all GPU float type testing.
from triton.
I see, thanks. I'm also having issues with some tests passing when called individually but failing when running the entire test suite. I'll solve this ASAP.
from triton.
Sometimes, ISAAC's GEMM uses two kernels. The event returned by clBlasSgemm always corresponded to the first one, which led to some synchronization issues. Setting label=0 solved the problem because it forced the library to use only one kernel. Fixed in f226837.
Could you retry now?
from triton.
@ptillet Nice catch, and it works great with clcaffe now. Thanks for your quick fix!
from triton.
That's good to hear :)
from triton.
Related Issues (20)
- Trivial outputs on MacOS using TRITON_INTERPRET=1 HOT 7
- 导入default_dump_dir报错
- What is the use of calling `propagateIfChanged` in `ConstantAnalysis::visit`
- Flash Attention benchmarks fail on AMD Instinct MI210 (`gfx90a`) HOT 2
- [OPS] AttributeError: module 'triton.language.math' has no attribute 'round' HOT 3
- Why is the documentation not versioned like other Read The Docs sites?
- Core dumped without "os.environ["TRITON_INTERPRET"] = "1"
- torch 2.4.0 requires triton==3.0.0; HOT 7
- Triton trying to download llvm even if self built, and build failed
- Seems mistake in setup.py HOT 1
- bfloat16 of fused attention seems have bug HOT 2
- Possible to use WMMA `.b1` instructions? HOT 1
- AttributeError("'AnnAssign' object has no attribute 'targets'") HOT 1
- [Bug] Compiler error in 06-fused-attention.py
- Fused Attention FP8 correctness on L20(Ada GPU) HOT 1
- What impact does changing the kernel shape's divisibility value have?
- Does Triton have a roadmap plan? HOT 1
- FP8 Casting Tensor Size Alignment Error on AMD MI210 HOT 1
- Output dtype of 09-persistent-matmul.py
- Worse performance on `H100` than `RTX3090`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from triton.