Open data platform based on Kubernetes. Scaleph supports SeaTunnel、Flink and Doris backended by SeaTunnel on Flink engine、Flink Kubernetes Operator and Doris operator.
Home Page: https://flowerfine.github.io/scaleph-repress-site/
License: Apache License 2.0
Arrays.asList return type is a ArrayList which is an inner class in Arrays and is not java.util.ArrayList . this will cause exception when add element into list .
see https://wenku.baidu.com/view/e44fda2ea5c30c22590102020740be1e650eccd3.html
refactor ui pages using react
dev
when new datasource added, jdbc source and jdbc sink select form can't find added datasource automatically
No response
dev
登录需要验证码
No response
add new issue for new connector support task.
integrate more seatunnel connector and support seatunnel on flink full feature.
Source
Sink
dev
Just click OK，No password changed,but meta_datasource table props change
No response
No response
add clickhouse sink connector
add clickhouse sink connector
optimize database table schema for better performance
scaleph-core-di module some table add xxx_code column with varchar type and create index on them.
CREATE TABLE `di_project` (
`id` bigint NOT NULL AUTO_INCREMENT COMMENT '自增主键',
`project_code` varchar(32) NOT NULL COMMENT '项目编码',
`project_name` varchar(64) DEFAULT NULL COMMENT '项目名称',
`remark` varchar(256) DEFAULT NULL COMMENT '备注',
`creator` varchar(32) DEFAULT NULL COMMENT '创建人',
`create_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`editor` varchar(32) DEFAULT NULL COMMENT '修改人',
`update_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
PRIMARY KEY (`id`),
UNIQUE KEY `project_code` (`project_code`)
) ENGINE=InnoDB COMMENT='数据集成-项目信息';
As we all know, out of order varchar type index is not friendly for B+ index.
For better performance, we need to replace them by primary key such as id. Introducing snowflake algorithm as our primary key generator is acceptable
1.0
作业组件编辑时不能触发版本号变化
作业发布后编辑保存，步骤属性信息没有同步存储下来
seatunnel作业的配置信息如果有错误的话，会导致整个jvm进程终止
No response
No response
dev
nginx setting config error that cause http response 401 see #147
https://github.com/flowerfine/scaleph/blob/dev/docs/build/build.md
errorCode: "401"
errorMessage: "用户认证不通过！"
showType: 4
success: false
No response
Scaleph supports flink 1.13.x, 1.14.x, 1.15.x, ...
Flink api evolves between different version and flinkful keeps up with flink through 1.13.6.xxx version or 1.14.4.xxx version.
Scaleph delegates flink cluster and job operate to flinkful and only supports specific flink version.
We need to add support multiple flink version feature to Scaleph.
Now JobController is heavy and contains much job dag logic, which is not suitable for application three layers architecture.
As an effort for more source code maintainability, try to split JobController to CRUD and operate two parts.
split JobController to CRUD and operate two parts.
check the cluster parameters when new or update a cluster
Determine whether the cluster has configured the necessary parameters based on the cluster type and the deployment mode
break barrier between frontend and backend datasource property setting.
fallbackProperty will work.
master
errorCode: "401"
errorMessage: "用户认证不通过！"
showType: 4
success: false
No response
dev
When bootstraped k3s cluster through k3d and autok3s, scaleph create flink session cluster on k3s failed.
The reason may be k3s kubeconfig certificate encryption algorithm.
No response
No response
We have evolved scaleph since 2022 year and obtain some achivements. In the last year of 2022, we want to earn more on open data platform based on flink.
We have already found a exciting flink stream contest called Beyond Stream Processing ！第四届实时计算 Flink 挑战赛, which is hosted by alicloud & tianchi. Now we are collecting players for participating in it.
Welcome everybody reply this issue and join us.
collect participators for flink stream contest
master
jvm terminated when seatunnel job conf file check result is false.
No response
No response
dev
Test task Kafka to JDBC , Incorrect page parameters:
"sink" : [
{
"url" : "jdbc:mysql://localhost:3306/testdata?",
"driver" : "com.mysql.cj.jdbc.Driver",
"username" : "root",
"password" : "123456",
"query" : "insert into test02 (id,name) values(?, ?)",
"batchSize" : "2",
"plugin_name" : "JdbcSink",
"_node_type" : "sink",
"_node_id" : 12,
"source_table_name" : "table_13"
}
In Seatunnel project Parameter is :
public static final String SINK_BATCH_SIZE = "batch_size";
No response
No response
dev
FileNotFoundException throws when startup
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:549)
at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:570)
at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:593)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:690)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78)
at org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:2170)
at org.apache.hadoop.fs.s3a.S3AUtils.patchSecurityCredentialProviders(S3AUtils.java:1174)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:240)
at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123)
at cn.sliew.scaleph.storage.configuration.FileSystemConfiguration.s3FileSystem(FileSystemConfiguration.java:59)
at cn.sliew.scaleph.storage.configuration.FileSystemConfiguration$$EnhancerBySpringCGLIB$$61a05adb.CGLIB$s3FileSystem$1(<generated>)
at cn.sliew.scaleph.storage.configuration.FileSystemConfiguration$$EnhancerBySpringCGLIB$$61a05adb$$FastClassBySpringCGLIB$$99e9f8a8.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
No response
scaleph-ui i18n
add i18n style and desc to scaleph-ui
dev/master
SeatunnelJobServiceImpl#getProjectBasePath function can't work on windows
java.lang.UnsupportedOperationException: 'posix:permissions' not supported as initial attribute
at sun.nio.fs.WindowsSecurityDescriptor.fromAttribute(WindowsSecurityDescriptor.java:358)
at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:492)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.TempFileHelper.create(TempFileHelper.java:136)
at java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:173)
at java.nio.file.Files.createTempDirectory(Files.java:991)
at cn.sliew.scaleph.engine.seatunnel.service.impl.SeatunnelJobServiceImpl.getProjectBasePath(SeatunnelJobServiceImpl.java:309)
at cn.sliew.scaleph.engine.seatunnel.service.impl.SeatunnelJobServiceImpl.submit(SeatunnelJobServiceImpl.java:172)
at cn.sliew.scaleph.engine.seatunnel.service.impl.SeatunnelFlinkJob.executeInternal(SeatunnelFlinkJob.java:43)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
No response
We need more detailed and organized docs about scaleph.
README and wiki introduce scaleph in an unsmooth way, lacks essential and detailed information which deep into scaleph
change table to partition table in log module
change table to partition table in log module
scaleph plugin property definition.
Our PropertyDescritpor is inspired by following excellent project:
As the reason of supporting seatunnel consumer.* parameter sytle, scaleph needs to add property group feature to PropertyDescriptor.
Try to unify storage interface and support local, HDFS, S3, OSS.
Storate interface based on flink FileSystem will replace various StorageService or BlobService, and support local, HDFS, S3, OSS.
In addition to that, unified storage interface will add local cache support for better performance.
dev
The latest docker image encountered failure when upload jar on project resource management.
192.168.192.1 - - [13/Aug/2022:00:29:05 +0000] "POST /api/datadev/resource/upload HTTP/1.1" 504 569 "http://localhost/scaleph/datadev/project" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
2022/08/13 00:29:05 [error] 11#11: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.192.1, server: localhost, request: "POST /api/datadev/resource/upload HTTP/1.1", upstream: "http://192.168.192.7:8080/scaleph/api/datadev/resource/upload", host: "localhost", referrer: "http://localhost/scaleph/datadev/project"
keep scaleph source code clean and robust
split scaleph-service module for better manage source code package and module responsibility
dev
datasource like kakfa can chosed in table input and output plugin and sql is not support with kafka .
No response
No response
dev
When hadoop bootstraps, scaleph would connect minio failed for network failure, even minio works normally.
No response
No response
A detailed flink cluster and job manage plan for supporting data-integration and ETL development.
multi flink version support
flink cluster
flink job
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.