bytedance / bitsail Goto Github PK

Is your feature request related to a problem? Please describe

Iceberg is a popular table format for analytics, which is often used with Object Store like S3 and OSS. Currently BitSail lacks relative capabilities especially writing to Iceberg.

Describe the solution you'd like

A Iceberg sink with Object Store support.

Describe alternatives you've considered

N/A

Additional context

You can refer to Flink implementations for Iceberg connector.

Is your feature request related to a problem? Please describe

Currently, kafka legacy source only support JSON format and we want to support PB format.

Describe the solution you'd like

we need to move com.bytedance.bitsail.dump.datasink.file.parser.PbBytesParser to a common place and reuse by other connector, such as kafka source

Describe alternatives you've considered

Additional context

Is your feature request related to a problem? Please describe

At present, the BytesParser of batch channel and stream channel are implemented separately and need to be unified

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

NA

Describe the solution you'd like

Migrate redis legacy connector to connector v1 interface

Describe alternatives you've considered

NA

Additional context

NA

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

We can reproduce the bug as follows (use screenshots if necessary):

1. The job conf is: '....'

2. The test data is: '....'

3. Bitsail run command: '....'

4. The error can be seen in: '....'

(Note that the above sentences are not necessary for each bug report issue.)

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Build Environment

Describe the environment where the test project was build.

• OS [e.g. Debain 11.0]
• JDK [e.g. 1.8.0_301]
• Maven [e.g. 3.6.3]
• ...

Execution Environment

Describe the environment where the test project ran.

• OS [e.g. Debain 11.0]
• JDK [e.g. 1.8.0_301]
• Deployment [e.g. Yarn | CDH 6.3.x]
• ...

Additional context

Add any other context about the problem here.

Is your feature request related to a problem? Please describe

We want to migrate the legacy kafka connector to V1 interface. Similar with #98 for redis

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

RabbitMQ有计划支持吗？

Is your feature request related to a problem? Please describe

The real-time progress can be constructed by using the split completion status and the topo original information of the task

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

We need to read local file as source. CSV file format could be a good start.

Describe the solution you'd like

Use V1 source connector interface, read CSV file as source and able to write to PrintSink

Additional context

Is your feature request related to a problem? Please describe

Doris connector ITCase test is ignored now because we need start a local Doris for test beforehand.

Describe the solution you'd like

Use DockerComposeContainer to achieve Doris test container.

• https://github.com/myfjdthink/doris-docker-compose

Describe alternatives you've considered

N/A

Additional context

N/A

Like FlinkCDC framework.

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

KerberosAuthenticator.getUgi not execute UserGroupInformation#loginUserFromKeytabAndReturnUGI, Even though kerberos parameters are passed。

this is my test

securityConfiguration.get(KerberosOptions.KERBEROS_ENABLE) is always false.because There's no place to assign。
you should add:
securityConf.set(KerberosOptions.KERBEROS_ENABLE, true);

and There's actually a problem here，

UserGroupInformation#loginUserFromKeytabAndReturnUGI should be execute in HadoopSecurityModule#login,but you return value is UserGroupInformation.getCurrentUser();

so securityModule#login is invalid

We can reproduce the bug as follows (use screenshots if necessary):

1. The job conf is: '....'

2. The test data is: '....'

3. Bitsail run command: '....'

4. The error can be seen in: '....'

(Note that the above sentences are not necessary for each bug report issue.)

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Build Environment

Describe the environment where the test project was build.

• OS [e.g. Debain 11.0]
• JDK [e.g. 1.8.0_301]
• Maven [e.g. 3.6.3]
• ...

Execution Environment

Describe the environment where the test project ran.

• OS [e.g. Debain 11.0]
• JDK [e.g. 1.8.0_301]
• Deployment [e.g. Yarn | CDH 6.3.x]
• ...

Additional context

Add any other context about the problem here.

Is your feature request related to a problem? Please describe

Currently only supports LogMetricReporter

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Got the master branch, and run
mvn clean package -D maven.test.skip=true
but got following compilation failure:
/bitsail/bitsail-connectors/bitsail-connectors-legacy/bitsail-connector-ftp/src/main/java/com/bytedance/bitsail/connector/legacy/ftp/source/FtpInputFormat.java:[75,91] incompatible types: java.lang.String cann ot be converted to com.bytedance.bitsail.common.type.TypeInfoConverter
The code snippet in line 75:

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

请问 hive 读取这块会像jdbc 那样，支持自定义sql吗？
由于可能涉及到不同库下hive表与 hive 表之间的同步的数据量比较大，整表同步的话不太现实。

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Uses reflection on InetAddress which no longer exists

final var cacheField = InetAddress.class.getDeclaredField("addressCache");
cacheField.setAccessible(true);
final var addressCache = cacheField.get(InetAddress.class);

var clazz = addressCache.getClass();
final var cacheMapField = clazz.getDeclaredField("cache");
cacheMapField.setAccessible(true);

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

Currently only supports HDFS storage, can be extended to object storage, such as S3

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

run a mysql reader job ，sql driver exception ：

source code driver name is mariadb ，is it fully compatible for connectting mysql

does mysql version occur this problem or other reason ?

Describe the bug

see below

To Reproduce

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

Build Environment

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

bitsail目前是否支持binlog connector？看了下代码里除了MysqlBinlogEventTimeExtractor没有其他相关的代码，是暂时还没有开源？

我们这边目前binlog->kafka->hudi的架构在kafka扩分区场景下存在消息乱序问题，bitsail是否能提供解决该问题的方案？

另外问下，bitsail有交流群么

Is your feature request related to a problem? Please describe

Currently we only support Flink 1.11. We want to support multiple Flink version and be able to compile with different maven profile. Some Flink API was change during the version upgrade, so we need to find a way to resolve this issue.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

Migrate Hbase legacy connector to V1 interface. We did the migration for Redis on #98

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

Now we support source reader for unified arch in bitsail. In legacy mode, we already support messenger & dirty collector for rate limit & dirty collector features.

We want support this feature in SourceReader also.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Describe the bug

CICD sometime fails due to flaky tests.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

需求: 业务场景OLAP, 计算后想输出到ck, 请问当前版本支持吗?

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

At present, the row builder of batch channel and stream channel are implemented separately and need to be unified

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is.

Example: I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

此平台没有前端？

Is your feature request related to a problem? Please describe

Migrate mangoDB connector to V1 interface. We did the same for Redis #98

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

本地docker环境已经启动。

请问执行case的流程是什么，按照文档说明报以上错误。