Comments (7)
Please try using the latests rapids-4-spark_2.12-23.12.0 release from artifactory.
Also, were there any errors in the power-run output? You should capture the driver/executor output and see if there were any errors.
from spark-rapids-benchmarks.
To attempt to repro this, I ran the full nds power run at scale 100 on my desktop for orc/snappy. Output from the cpu and gpu matched for all of the queries except for this one known difference:
=== Comparing Query: query79 ===
Collecting rows from DataFrame
Collected 100 rows in 0.11637043952941895 seconds
Collecting rows from DataFrame
Collected 100 rows in 0.10641264915466309 seconds
Row 99:
[None, None, 'Bethel', 12213922, None, None]
[None, None, 'Bethel', 22767968, None, None]
The difference is in the ss_ticket_number
field, which is in the group by key, but not in the sort order for the output, so this is likely just ambiguous ordering in the output. I have verified that I get the same result on GPU with different compressors. This is a difference you may need to account for when doing comparisons.
Note that I built with the cuda11 jars.
from spark-rapids-benchmarks.
@yinqingh can you upload the query file and maybe point us to the input orc data path? if you still facing such diff after using the latest snapshot jar.
from spark-rapids-benchmarks.
The issue is found on the released nvcomp version is 3.0.3
, we will check if the latest snapshot version of nvcomp still has the issue. If yes, well update it here.
from spark-rapids-benchmarks.
from spark-rapids-benchmarks.
@jbrennan333 @wjxiz1992 Thanks for the information! We will keep testing with the latest nvcomp and update here if the same issue can be repro in 3.0.4.
from spark-rapids-benchmarks.
Since we did not observed this issue with the daily rapids-plugin jar(containing the 3.0.4 nvcomp libs), I suppose we can close this issue first.
If it is repro again, we can re-open later this later.
from spark-rapids-benchmarks.
Related Issues (20)
- Transcode from other formats besides CSV
- Query filtering for nds_power HOT 2
- [Bug] Failed to build twice in nds/tpcds-gen HOT 2
- [FEA] Support Iceberg and DeltaLake as input data format for data conversion
- [FEA] Allow property files for nds_transcode and other nds_* scripts HOT 1
- [BUG] Got error "cannot resolve 'd_date' given input columns" when run nds_maintenance.py HOT 1
- [BUG] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project tpcds-gen
- Update README to latest version of spark-rapids HOT 1
- [FEA] nds_validate.py can compare some specified queries
- [FEA]Define a pre-commit hook to update copyright year automatically
- [BUG] nds_transcode.py is not handling international characters correctly HOT 1
- [BUG] No need for patches/code.patch HOT 1
- [BUG] Throughput run should use a new template file instead of using the Power test one
- [BUG] Table meta information are missing when running Data Maintenance over Delta
- Implement a pre-commit / premerge check for licence headers HOT 1
- [FEA] Create NDS-H benchmark for performance analysis HOT 2
- [QST] Cannot run on GPU because GpuCSVScan only supports UTF8 encoded data HOT 14
- [BUG] Iceberg related jobs failed due to Spark version incompatibility
- [BUG] Delta related jobs failed due to Spark version incompatibility
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-rapids-benchmarks.