Comments (4)
you just need to grep the json files
IMO this should not be the default answer. Users don't expect jobs that produce the incorrect output to silently "succeed." Having to specify an extra option and then manually grep afterwards is not user friendly at all. We control the benchmark script, so we should be able to track when queries fail and have the overall Spark application return an error if any queries failed. Ideally it should have these running modes:
- Default mode: queries are run as they are today. If any query failed (not task attempt failed that succeeded on retry), the overall Spark application returns an error. If the driver needs to write to a json summary file and grep through it itself, then that's what it should do. Queries that fail should never lead to a successful Spark application by default.
- Hardcore benchmark run mode: max task attempts are configured to one. If any task attempt fails, benchmark performance run is essentially invalid and Spark application fails immediately.
from spark-rapids-benchmarks.
By using --json_summary_folder, the power run will save the status into a json file for each query.
Like our CI job does, you just need to grep the json files and check the query.status[Completed, CompletedWithTaskFailures, Failed].
Close this issue.
from spark-rapids-benchmarks.
I see.
Seems I misunderstood the requirement here. I thought it's just for our benchmark running, which we already added the check step.
For our benchmark, we're not setting spark.task.maxFailures be 1. It should be default as 4.
We still need to report the result when some queries failed or CompletedWithTaskFailures.
We can update the script default to fail the spark job when any query failed but continue to finish all the queries.
And for benchmark running, we'll add one special parameter to disable this feature to make our CI job unblock.
from spark-rapids-benchmarks.
The benchmark scripts are used by more than just CI. We point users to the benchmark scripts in many of our public presentations, encouraging them to run the benchmarks themselves. Therefore the benchmark scripts need to be as user-friendly as possible.
The script definitely needs to fail if any queries fail, because the benchmark run is clearly invalid at that point. I'm OK if it doesn't fail when there are task failures. Ideally it should complain very loudly when that happens, because that's not a clean benchmark run and will result in lower performance numbers being reported than theoretically should be attainable. Task attempt failures could also indicate OOM situations or other errors that shouldn't be there but where masked by an executor relaunch, and thus we would want to investigate.
from spark-rapids-benchmarks.
Related Issues (20)
- Transcode from other formats besides CSV
- Query filtering for nds_power HOT 2
- [Bug] Failed to build twice in nds/tpcds-gen HOT 2
- [FEA] Support Iceberg and DeltaLake as input data format for data conversion
- [FEA] Allow property files for nds_transcode and other nds_* scripts HOT 1
- [BUG] Got error "cannot resolve 'd_date' given input columns" when run nds_maintenance.py HOT 1
- [BUG] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile (default-compile) on project tpcds-gen
- Update README to latest version of spark-rapids HOT 1
- [FEA] nds_validate.py can compare some specified queries
- [FEA]Define a pre-commit hook to update copyright year automatically
- [BUG] nds_transcode.py is not handling international characters correctly HOT 1
- [BUG] validation failure of power run results on gpu with nvcomp and power run results on cpu HOT 7
- [BUG] No need for patches/code.patch HOT 1
- [BUG] Throughput run should use a new template file instead of using the Power test one
- [BUG] Table meta information are missing when running Data Maintenance over Delta
- Implement a pre-commit / premerge check for licence headers HOT 1
- [FEA] Create NDS-H benchmark for performance analysis HOT 2
- [QST] Cannot run on GPU because GpuCSVScan only supports UTF8 encoded data HOT 14
- [BUG] Iceberg related jobs failed due to Spark version incompatibility
- [BUG] Delta related jobs failed due to Spark version incompatibility
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-rapids-benchmarks.