Comments (10)
hoodie.simple.index.parallelism
can be modified to adjust the parallelism of stage 121, but it may cause the parallelism of stage 120 to decrease. You can give it a try.
from hudi.
Hi @KnightChess Thanks for commenting. But my major doubt is why shuffle write is nearly doubled in hudi 0.14? And that is leading to issues in step 121
from hudi.
@bibhu107 if next parallelism is too bigger, shuffle data will grow. And the mayjor reason is task is too much because high, spark need to scheduler too mush task.
from hudi.
@bibhu107 and why shuffle data grow, I haven't looked at the code in detail; the following is just my guess. you have too much reducer, so the shuffle data may be need more meta. And on the other head, 0.7 -> 0.14, may be the shuffle's java object attr has change, this alse can cause diff. But I think parallelism is the major problem.
from hudi.
@bibhu107 does it can work for you? I misread the stack trace, so this parameter hasn't taken effect in the question stage. The parameter that needs to be set is a different one. If you are insert
, try to set hoodie.insert.shuffle.parallelism
, if upsert
, set hoodie.upsert.shuffle.parallelism
from hudi.
Slack Thread https://apache-hudi.slack.com/archives/C4D716NPQ/p1718122404452279
from hudi.
Hello @KnightChess, thank you for your suggestions.
Initially, the Adaptive Query Execution (AQE) feature was disabled for the jobs because we were explicitly setting spark.sql.shuffle.partitions
. Later, we enabled it using the following configuration:
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.skewJoin.enabled=true
Additionally, we removed the spark.sql.shuffle.partitions
configuration. This change resulted in better job performance. However, we have not yet conducted any load/pressure testing.
We will share the results once we perform load testing. For now, we have moved back to Hudi 0.8 and using Spark3.3.1.
Thank you for raising the PR.
from hudi.
I have one query: Why do we need this PR? I expected Hudi to automatically take the deduced parallelism from Hudi 0.13.
As mentioned in the documentation for hoodie.upsert.shuffle.parallelism, it states:
From version 0.13.0 onwards, Hudi by default automatically uses the parallelism deduced by Spark based on the source data.
from hudi.
@bibhu107 hi, this pr is target to imporve the deduced parallelism and more user friendly, in hudi side, AQE can not effort because use rdd directly
from hudi.
I am closing this Issue. Thanks for support @KnightChess
from hudi.
Related Issues (20)
- [SUPPORT] Schema Evolution in MR Reading of Hudi Causes Metadata List Request for Each Split HOT 5
- [SUPPORT] hudi-0.15.0 spark sql query mro's ro table error: Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop/apps/hive/warehouse/hudi.db/hudi_qwe_cdc/partition_001/00000096-b062-4667-9c66-37fb8e3f26ef_1-8-2_20240802095544277.parquet HOT 1
- [SUPPORT] Error when selecting RT table on AWS Athena (0.14.1) - with custom Payload class HOT 1
- [SUPPORT] PartialUpdateAvroPayload still overwriting undefined columns as NULL HOT 8
- [SUPPORT] Failed to deserialize HoodieAvroIndexRecord HOT 5
- [SUPPORT] Upgrade from 0.6.0 to 0.15.0 HOT 5
- [SUPPORT] How to achieve parallelism for Offline compaction job utilising utility HoodieCompactor.java running on aws GLUE HOT 4
- [SUPPORT]BaseDatasetBulkInsertCommitActionExecutor execute WriteStatus donot persist cause data write hudi 4 times HOT 7
- [SUPPORT] s3 list cost increases exponentially when using COW table HOT 19
- org.apache.avro.AvroRuntimeException: Not a valid schema field: IncidentId,EntryTime,ModifiedTime HOT 11
- [SUPPORT] Disable only Async Compaction in hudi deltastreamer is not working HOT 7
- [SUPPORT] The rollback failed because the file could not be created because the marker file already existed. HOT 3
- [SUPPORT] how to start schema evolution HOT 3
- [SUPPORT] Hudi table created with dataframe API becomes unwritable to INSERT queries due to config conflict HOT 10
- [SUPPORT] HoodieException: Failed to apply clean commit to metadata - HUDI 0.15 HOT 4
- [SUPPORT] Hudi primary key config is case-sensitive HOT 10
- [SUPPORT]create hive table on top of existing hudi table HOT 2
- [SUPPORT] Failed to use Bloom filter Indexing HOT 7
- [SUPPORT] Duplicate fileId exception occurs in Flink Bucket MOR table HOT 10
- Unable to merge cdc records to Hudi snapshot. HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hudi.