Comments (4)
I got this error with your repro and TOT:
Traceback (most recent call last):
File "/home/nmaruyama/pytorch/debug3/nvfuser/repro.py", line 1, in <module>
from nvfuser import FusionDefinition, DataType
File "/home/nmaruyama/pytorch/debug3/nvfuser/nvfuser/__init__.py", line 15, in <module>
from . import _C
ImportError: cannot import name '_C' from partially initialized module 'nvfuser' (most likely due to a circular import) (/home/nmaruyama/pytorch/debug3/nvfuser/nvfuser/__init__.py)
I don't have any problem with the python frontend tests:
(/opt/conda/pytorch) docker-pyt-cuda-117_debug3:nvfuser$ python python_tests/test_python_frontend.py /home/nmaruyama/pytorch/debug3/pytorch/torch/backends/cudnn/__init__.py:93: UserWarning: PyTorch was compiled without cuDNN/MIOpen support. To use cuDNN/MIOpen, rebuild PyTorch making sure the library is visible to the build system.
warnings.warn(
......................................................
----------------------------------------------------------------------
Ran 54 tests in 174.014s
OK
Does it need some other packages?
from fuser.
The issue code be running the repro from the root directory of the nvFuser repository. It seems to cause an issue.
from fuser.
I did try running it from the PyTorch directory but had the same error. The repro was placed in the nvfuser directory. I copied the repro to the PyTorch directory and ran it from there, and now it worked. No idea why.
Anyway, I was able to reproduce the failure. What the symptom indicates is there may be some logic error in the segmenter. More specifically, the segmenter completed the segmentation of the fusion and the heuristics of each segment was found, and then it tried to schedule each segment with the chosen heuristic, in this case a reduction heuristic, but it didn't find any reduction in the given segment, which shouldn't happen as the reduction heuristic was chosen. I've seen similar errors before, and for this one I'm guessing there must be some logic error. I don't know if it has anything to do with the full
op and the reduction.
from fuser.
It was a much simpler bug. Fixed in #143
This fix isn't ideal as it segments at the full op and the following reduction. The reduction scheduler assumes there's an input, but that's not the case with the repro, so it failed. I'm sure we could support this pattern without segmentation, but I haven't looked into it yet.
How important to fuse patterns like this?
from fuser.
Related Issues (20)
- Hopper-only crash: Unsupported loop structure. Two loops are mapped together. HOT 26
- Sharding propagation
- ExpressionEvaluator errors out when running `logical_and` HOT 1
- Add runtime check for circular buffer correctness
- Codegen error: `have dynamic allocations but are placed in local memory.` coming from qkv_split_rope backwards HOT 9
- benchmarking thunder.jit
- MarkAliasPrepare does not preserve shardings
- Performance of innerOuterPersistent scheduler is very sensitive to register usage and spills HOT 3
- Generalize pattern matching of matmul patterns
- SDPAOp and ComputeAtMap HOT 2
- makeInitialSchedulerEntry requires more than the current segment's inputs. HOT 6
- nvFuser confuses sharded dimension size and unsharded. HOT 3
- crash in transformOutputFromAllocationToLogical: frontier.size() != logical.size() HOT 2
- Indexing failure due to self-mapped broadcast domains
- Indexing error in matmul with certain parameters HOT 1
- Allocs must be based on constant integers HOT 7
- Promote reuse over specialization for small reductions
- INTERNAL ASSERT FAILED: Allocations must be based on constant integers for local memory HOT 2
- performance issue on dynamic shaped tensor HOT 7
- Evaluation for rFactored reductions is not supported. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fuser.