laopeng2021 / castled Goto Github PK

Open source reverse-ETL platform to operationalise your data warehouse

Castled is a Reverse ETL solution which enables you to make the valuable customer insights lying in your data warehouse actionable by moving them to sales, markerting and support tools of your choice. We currently support all major data warehouses incluing Snowflake, BigQuery, Redshift and Postgres as the data source and some of the most popular sales, marketing and support solutions such as Salesforce, Hubspot, Intercom, Google Ads etc as the destination.

Some of our key offerings include

• Incremental or Full data syncs from your cloud data warehouse.
• Automated failure handling and recovery.
• Reliability and Resilience at scale.
• CFL(Castled Form Language) is a Java Annotation based UI framework, which can support the most complex UI forms from backend, thereby removing the prerequisite of having a UI developer to add a new connector.
• Kubernetes ready for easy deployment.

Quick link: https://oss-docs.castled.io/deploying-castled/deploy-local

git clone https://github.com/castledio/castled.git
cd castled
docker-compose up

You can access castled app at http://localhost:3000. Happy castling!!!

• Quick startup demo

• Creating a pipeline with Castled

You can access the documentation at: https://oss-docs.castled.io/

• Discord - For any discussions or help needed in getting started with Castled or just to hang out.
• Github - If you need to raise any bugs or feature requests please use Github. We will get back to you in no time!

The throughput/latency of a data pipeline depends mostly on the destination api limitations and/or rate limits. We do make the best effort to tune it to provide the most optimal throughput possible. But to give an indication of the throughput we support, publishing a bench mark we did to transfer upto 100 million records from BigQuery to Apache Kafka.

We are a huge fan of the open source community and we value your contributions. We have spent an insane amount of time customizing our framework to reduce the amount of time required to built a new connector. Here is all you need to do to add your own custom connector.

1. Implement java interfaces (ExternalAppConnector and DataSink) on the backend.

2. Customize the mapping configuration and app configuration on the UI using our java annotation based framework. Refer the mapping configuration and app configuration of our CustomerIO connector for more info.

• Quick video on how to write a source connector

• Quick video on how to write a destination connector

Refer our license file for queries regarding the licensing of our different modules.