Giter Club home page Giter Club logo

flink-connector-http's Introduction

SQL

CREATE TABLE UserScores (name STRING, score INT)
WITH (
'connector' = 'http-restful',
'path' = '/flink/table1',
'port' = '8080',
'format' = 'http-restful-json'
-- TODO
-- 'mode' = 'server'
);
CREATE TABLE print_table  
(id STRING,score INT)
WITH ('connector' = 'print','sink.parallelism'='1','standard-error'='true') ;

insert into print_table select name,score from UserScores;
POST
DELETE
PUT
CREATE TABLE UserScores (name STRING, score INT)
WITH (
'connector' = 'http-restful',
'remote-url' = 'http://localhost:8088/some/re',
#'method' = 'post',-- ''
'format' = 'http-restful-json'
-- 'mode' = 'client'
-- 'http-restful.headers.Cookie' = '_ga=GA1.2.2136019336.1646989403'
-- 'http-restful.headers.Referer' = 'https://www.baidu.com'
);

flink-connector-http's People

Contributors

chengkaiyang2022 avatar

Watchers

 avatar

flink-connector-http's Issues

Unifi DDL when create source table and sink table

  1. The DDL of source table should be like:
    CREATE TABLE UserScores (name STRING, score INT)
    WITH (
    'connector' = 'http-restful',
    'mode' = 'source'
    'format' = 'http-restful-json'
    'remote-url' = 'http://localhost:8088/some/re?name={#name}',
    'remote-body' = '{name: '{#name}' }'
    'source.request.body-format' = '{"token":"some token","data":DATA}'
    'source.request.headers.Cookie' = '_ga=GA1.2.2136019336.1646989403'
    'source.request.headers.Referer' = 'https://www.baidu.com'
    'source.request.headers.xxx' = 'yyyy'
    );

  2. The DDL of sink table should be like:
    CREATE TABLE UserScores (name STRING, score INT)
    WITH (
    'connector' = 'http-restful',
    'mode' = 'sink'
    'remote-url' = 'http://localhost:8088/some/re',
    'format' = 'http-restful-json'
    'sink.request.body-format' = '{"token":"some token","data":DATA}'
    'sink.request.headers.Cookie' = '_ga=GA1.2.2136019336.1646989403'
    'sink.request.headers.Referer' = 'https://www.baidu.com'
    'sink.request.headers.xxx' = 'yyyy'
    );

Ref:

  1. https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses`

Upsert mode and insert mode

Only support at most once on source side because the source side is not a queue that can be consumed from at some point again. On the sink side, it is idempotent because it is restful API which means the result will be the same how matter how many time the PUT/DELETE are sent.

Like JDBC SQL connector, and Elasticsearch SQL connector, the Http Restful SQL connector can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL.

If no primary key is defined on the DDL, the connector can only operate in append mode for exchanging INSERT only messages with external system.
Dependencies #

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.