Comments (15)
Thanks for this @Herobring . We'll investigate and come back to you.
from tensorflow-directml.
I also ran into this problem with ai_benchmark. My GPU is Rx 580.
from tensorflow-directml.
I had the very same problem at the very same section yesterday with a vega56. If it helps to upload some logs, please let me know what you'd need.
from tensorflow-directml.
I also ran into this problem with ai_benchmark. My GPU is Rx 580.
Same problem with integrated GPU on ryzen 3500u
@PatriceVignola
from tensorflow-directml.
Ran into the same issue using
CPU: Ryzen 5 3600
GPU: Radeon RX 5700 XT
OS: Windows 10 (Version 10.0.18363 Build 18363)
Python: Python 3.7.4
Also Ran into the same issue when use_CPU was set to True
from tensorflow-directml.
Hi everyone,
We just released a new version of our pypi package with changes that should reduce the occurrences of device removals on AMD hardware. The package also has many bug fixes, some performance improvements and better operator coverage. Please update and let us know how it goes!
from tensorflow-directml.
Hi everyone,
We just released a new version of our pypi package with changes that should reduce the occurrences of device removals on AMD hardware. The package also has many bug fixes, some performance improvements and better operator coverage. Please update and let us know how it goes!
Still did not work. I tried running ai-benchmark and, on bench 7/19, got:
2020-12-20 15:10:19.371787: F tensorflow/core/common_runtime/dml/dml_command_queue.cc:35] Check failed: (((HRESULT)((queue_->Signal(fence_.Get(), last_fence_value_)))) >= 0) == true (0 vs. 1)
from tensorflow-directml.
@ffleader1 Are you sure that you upgraded tensorflow-directml to 1.15.4.dev201216? The error message here looks like a message from the previous version. In 1.15.4.dev201216, we changed the message of the DML_CHECK_SUCCEEDED
macro and it should now output a more helpful message along the lines of HRESULT failed with <error_code>
dml_command_queue.cc
DML_CHECK_SUCCEEDED definition
HandledFailedHr definition
from tensorflow-directml.
@ffleader1 Are you sure that you upgraded tensorflow-directml to 1.15.4.dev201216? The error message here looks like a message from the previous version. In 1.15.4.dev201216, we changed the message of the
DML_CHECK_SUCCEEDED
macro and it should now output a more helpful message along the lines ofHRESULT failed with <error_code>
dml_command_queue.cc
DML_CHECK_SUCCEEDED definition
HandledFailedHr definition
Oh my bad. Pycharm did update the plugin properly for me. Anyway, I tried again and I got to 14/19, with this error:
2020-12-20 21:33:35.204926: F tensorflow/core/common_runtime/dml/dml_command_recorder.cc:366] HRESULT failed with 0x887a0006: dml_device_->GetDeviceRemovedReason()
from tensorflow-directml.
Thank you, that makes more sense :) The change that we pushed in this release doesn't apply to all situations or all hardware equally, so getting more data on which hardware it helps and which hardware still hit device removals will help our ongoing efforts to minimize them. Can I ask which driver version you have installed?
In the meantime, you can also follow the instructions over here to temporarily disable TDRs and see if it helps. This is the same manipulation that CUDA users are asked to do to avoid TDRs on Windows, which is probably the reason you're getting a device removal in the first place.
from tensorflow-directml.
Thank you, that makes more sense :) The change that we pushed in this release doesn't apply to all situations or all hardware equally, so getting more data on which hardware it helps and which hardware still hit device removals will help our ongoing efforts to minimize them. Can I ask which driver version you have installed?
In the meantime, you can also follow the instructions over here to temporarily disable TDRs and see if it helps. This is the same manipulation that CUDA users are asked to do to avoid TDRs on Windows, which is probably the reason you're getting a device removal in the first place.
I do not update my GPU driver regularly. I use a Rx 580 with 20.9.1 driver.
I will try to both update driver and increase the TDR and head back.
from tensorflow-directml.
Thank you, that makes more sense :) The change that we pushed in this release doesn't apply to all situations or all hardware equally, so getting more data on which hardware it helps and which hardware still hit device removals will help our ongoing efforts to minimize them. Can I ask which driver version you have installed?
In the meantime, you can also follow the instructions over here to temporarily disable TDRs and see if it helps. This is the same manipulation that CUDA users are asked to do to avoid TDRs on Windows, which is probably the reason you're getting a device removal in the first place.
Oh I think I got it working. Changing the TDRs does seem to work. I am currently setting it to 10. I wonder if I should invert it back to 2.
from tensorflow-directml.
Good, it confirms that the device removal is due to a TDR and not a driver issue. In general you should keep the TDR as the default value since it protects you from having an unresponsive system, but since training models can take a long time, it's ok to disable it temporarily while you're executing the workload. Don't forget to set it back to the default value when you're done though!
from tensorflow-directml.
Awesome! I'll rerun test on a new version in the nearest weekends.
from tensorflow-directml.
Hi @PatriceVignola! Thank you for the fix, it works! 🎉
For those who interested results for non-overcloked AMD Radeon VII + directML setup are 7935, which is 40% of AMD Radeon VII + tf 2.1.0 + ROCm(OpenCL) + Debian10 =19367 (probably overcloked)
Anyway this is still a great start for making AMD on Windows builds possible, cheers! 🍾 🎉
- TF Version: 1.15.4
- Platform: Windows-10-10.0.19041-SP0
- CPU: N/A
- CPU RAM: 64 GB
- GPU/0:
- GPU RAM: 31.7 GB
- CUDA Version: N/A
- CUDA Build: N/A
Device Inference Score: 4486
Device Training Score: 3449
Device AI Score: 7935
from tensorflow-directml.
Related Issues (20)
- Not able to use my own callbacks HOT 3
- Tensorflow-directml is not making any difference in processing times in GPU vs CPU HOT 1
- AMD APU support? HOT 2
- Cannot assign a device for operation embedding/embeddings/Initializer/random_uniform/ HOT 5
- directml on custom tensoflow build ? HOT 1
- Windows Camera post process(DMFT) with DirectML(Tensorflow) HOT 10
- It's not working on Intel Graphics 5500 HOT 1
- Use c api to load pb models HOT 1
- how to set the default device using cAPI?
- session run crashed when runing on nvidia gpu HOT 1
- Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. HOT 2
- unbox expects Dml at::Tensor as inputs HOT 1
- Is it not supports the amd gpus without rocm?
- Does directml support multi-GPU training
- TensorFlow-DirectML Does Not Exist HOT 7
- AttributeError: module 'tensorflow' has no attribute 'float32' HOT 1
- CPU instructions notification after installing
- python newer versions support
- tensorflow GPU on WSL2 produces a segmentationfault
- RC astro tool operations not supported in DirectML HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorflow-directml.