Comments (2)
2.i : 1.12.1 版本 connector 的 exactly-once 导入问题已在各分支最新代码中修复,1.12.1 之前的版本不受影响。
2.ii: compaction增加跟导入频率相关,可以修改flush参数尽量保证每批次数据最大并降低导入频率。同时将checkpoint周期设为分钟级别。
3.2: 这个问题比较难解释
产生的直接原因是,某个label导入的状态有可能是未知的,但是be返回了fail(比如 failed to call frontend service),所以外部系统认为此次导入失败,再次使用相同label去做重试导入时,be其实还没有最终确定这个label的状态,所以返回了label already exists。此时connector会去check label state直至查到label的最终状态后,根据最终状态做对应的操作。因此最坏的情况就是任务会阻塞直到某个未知状态label最终timeout才可以继续。而stream load timeout默认时长为10分钟,如果想修改的话可以设置'sink.properties.timeout' = '30'来减少timeout等待时长。
根本原因为be不应该在未确定最终状态的情况下就返回fail给用户,这个会在后续sr的版本中修正。
from starrocks-connector-for-apache-flink.
@hffariel 问下 1.2.3_flink-1.11_2.11 会有这个问题吗?
starrocks版本 2.3.1 fe5d830
from starrocks-connector-for-apache-flink.
Related Issues (20)
- mysql to SR , 如何支持同时导入多表?
- 支持自定义StarRocksGenericRowTransformer 清洗数据
- StarRocksSinkRowDataWithMeta is not a high performance type
- 【BUG】【Lookup Join】在使用StarRocks连接器做lookup join 的时候,select的字段必须要和表字段一样
- support MySQL's entire database synchronization without many Flink SQL
- sync add/drop schema change by debezium
- support load data from starrocks table by specific condition HOT 1
- [Feature] flink source无法读取json字段 HOT 1
- 容器部署starrocks, Flink Sink连接发生 xx.xx.xx.xx:8040 Connection timed out
- Will StarRocks source support custom SQL queries in the future
- 不支持二进制数据(binary、bytes)的读取和写入 HOT 1
- The AsyncTableFunction can be used for Flink lookup join StarRocks dimension table
- In the 3.1 separated storage and computation version, the use of the "insert into" statement in Flink SQL will lose the data with delete rowkind.
- [BUG] Error occurred in DefaultStreamLoader#getAvailableHost while trying to load a URL due to configuration issues.
- [Bug] The starrocks connector unknown datatype handle method maybe need to change
- [BugFix] Index out of range exception occurs in certain data types
- [BugFix] Causing index out of bounds
- [BugFix] predicate push-down time dimension table error
- [BUG] StarRocksSink 重试多次后死锁
- starrocks 用官网给出的 csv 例子 sink 端一直在 INITIALIZING
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from starrocks-connector-for-apache-flink.