Comments (4)
That is because you are using the append mode, the append mode relies on the async clustering for small files merging. You can switch to upsert
operation for the tests too, the upsert will merge the files during write.
from hudi.
That is because you are using the append mode, the append mode relies on the async clustering for small files merging. You can switch to
upsert
operation for the tests too, the upsert will merge the files during write.
Thank you. Looks like it's working fine. However, is it mentioned in documentation anywhere? All that I see is the note - the bulk_insert write operation does not have auto-sizing capabilities during ingestion
Another question, by adding the following property I was able to achieve the same for insert operation type:
'write.insert.cluster' = 'true',
This property above is not greatly documented either, I found it by digging in source code..
Is there any difference really? Or overall recommendation is to use async clustering?
Thanks in advance
from hudi.
'write.insert.cluster' = 'true'
is actually inline clustering, that means it will do the clustering on each write, so the write throughput is low comparing to the async clustering.
Sorry it is not well documented.
from hudi.
Created a tracking JIRA to fix doc - https://issues.apache.org/jira/browse/HUDI-7396
Thanks @dannyhchen @vkhoroshko . Closing this.
from hudi.
Related Issues (20)
- [SUPPORT] Mandatory registration of `HoodieSparkKryoRegistrar` using PySpark for 1.0.0-beta2-rc2 HOT 5
- [SUPPORT] Failed to insert overwrite hudi table when defining partition column with int type. HOT 2
- Duplicate records in MOR (Hudi 0.12.2 on EMR)
- [SUPPORT] HoodieFlinkWriteClient null pointer for HudiRecord locationdata HOT 2
- [SUPPORT] HoodieStreamer: HoodieKeyException error when using SQL Transformer HOT 8
- [SUPPORT] Why HUDI ConsistentBucketClusteringExecutionStrategy not supported by flink engine? HOT 3
- [SUPPORT] HoodieSnapshotExporter is broken with PR HUDI-712 HOT 14
- [SUPPORT] Avoid rescan file when clean HOT 5
- [SUPPORT] Exception in running table services on metadata table (hudi0.15.0) HOT 5
- [SUPPORT]Encountered java.lang.ClassNotFoundException when using hudi through spark-sql HOT 1
- [SUPPORT] Timeout waiting for connection from pool running Hudi 0.14.0 on EMR 7.0.0 HOT 2
- [SUPPORT] Use Case Validation for CDC Ingestion from Kafka/Debezium (SQL Server) into Apache Hudi HOT 4
- [SUPPORT] Flink engine support RECORD_INDEX HOT 8
- [SUPPORT] The cleaning service takes a long time HOT 16
- [SUPPORT] PATH_NOT_FOUND Error when Running Hudi Incremental Query HOT 3
- The payloadProps of AbstractRealtimeRecordReader is missing other configurations HOT 2
- [SUPPORT] spark-sql write the table fail witch was created by flink-sql with HoodieHiveCatalog and the error is:Config conflict(key current value existing value): PreCombineKey: ts HOT 5
- [SUPPORT] After upgrading from hudi-0.14.0 to 0.15.0, an error occurred: Caused by: java.lang.NullPointerException: null value for (non-nullable) string at table_name.role_id2 HOT 4
- [SUPPORT] Flink uses BULK_INSERT method to write into Hudi to handle the error: File hdfs:/apps/hive/warehouse/hudi.db/xxx/.hoodie/20240727154013121.deltacommit.inflight does not exist! HOT 1
- [SUGGESTION] add a github bot auto deal with outdated issues HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hudi.