Comments (3)
@hangc0276 Thanks for rising up this issue. Would you like to contribute to this feature?
Yes, i have developed the first version and run on my gray env. I will contribute this feature.
Flink 1.11 have changed some interface, maybe incompatible with Flink 1.9.1.
from pulsar-flink.
@hangc0276 Thanks for rising up this issue. Would you like to contribute to this feature?
from pulsar-flink.
set global.disable.operator.chain = true;
create table test_flink_sql(
`rip` VARCHAR,
`rtime` VARCHAR,
`uid` bigint,
`rchannel` VARCHAR,
`be_time` bigint,
`be_time` VARCHAR,
`activity_id` VARCHAR,
`country_code` VARCHAR,
`os` VARCHAR,
`recv_time` bigint,
`remark` VARCHAR,
`client_ip` VARCHAR,
`day` as TO_DATE(rtime),
`hour` as date_format(rtime, 'HH')
) with (
'connector.type' = 'pulsar',
'connector.version' = '1',
'connector.topic' = 'persistent://test/test-gray/test_flink_sql',
'connector.service-url' = 'pulsar://xxx',
'connector.admin-url' = 'http://xxx',
'connector.startup-mode' = 'external-subscription',
'connector.sub-name' = 'test_flink_sql_v1',
'connector.properties.0.key' = 'pulsar.reader.readerName',
'connector.properties.0.value' = 'test_flink_sql_v1',
'connector.properties.1.key' = 'pulsar.reader.subscriptionRolePrefix',
'connector.properties.1.value' = 'test_flink_sql_v1',
'connector.properties.2.key' = 'pulsar.reader.receiverQueueSize',
'connector.properties.2.value' = '1000',
'connector.properties.3.key' = 'partitiondiscoveryintervalmillis',
'connector.properties.3.value' = '5000',
'format.type' = 'json',
'format.derive-schema' = 'true',
'format.ignore-parse-errors' = 'true',
'update-mode' = 'append'
);
insert into hive.test.test_flink_sql
select
rip, rtime,
if (uid is null, 0, uid) as uid,
if (activity_id is null, '', activity_id) as activity_id,
if (country_code is null, '', country_code) as country_code,
if (os is null, '', os) as os,
if (recv_time is null, 0, recv_time) as recv_time,
if (remark is null, '', remark) as remark,
if (client_ip is null, '', client_ip) as client_ip,
cast(`day` as string) as `day`,
cast(`hour` as string) as `hour`
from test_flink_sql;
from pulsar-flink.
Related Issues (20)
- [BUG] Set pulsar event time when it exceed zero. HOT 1
- [BUG]pulsar-flink catalog options HOT 1
- [DOC] Catalog config and DDL properties in README doc is out of date HOT 1
- [BUG] Pulsar client issue causing unacknowledged messages. HOT 1
- Json Format stores Timestamp as long but schema will be translated to Row<> type. HOT 1
- Expose metrics/profiling in docker HOT 1
- Merge pulsar-flink docker build process with sql-gateway image build process
- Upgrade pulsar-flink pulsar version to 2.10.0
- [BUG][FLINK-26980] PulsarUnorderedSourceReaderTest hang on azure
- [QUESTION] How does this project relate to apache pulsar flink connector HOT 1
- [FEATURE] Support auto failover in Flink connector
- [FEATURE] Allow USE_LATEST_WHEN_TOO_LARGE_CURSOR
- [BUG] Flink pulsar source upgrade from 1.13.1.4 to 1.13.6.2 fails HOT 1
- [BUG] Cumulative Acknowledgement is not happening in the flink-connector-pulsar
- [BUG] {"errorMsg":"org.apache.pulsar.transaction.common.exception.TransactionConflictException
- [FEATURE] 基于keyshare模式的简单使用
- [protobuf] infinite recursive call in protobuf-native parser HOT 3
- [BUG] Question about Flink Event-time support.
- [BUG]Flink connects pulsar ,that offtens responses PulsarAdminException HTTP 502 Bad Gateway HOT 1
- verify if OAuth works on 1.14 branch
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar-flink.