Giter Club home page Giter Club logo

clickhaskell's Introduction

ClickHaskell

built with nix

ClickHaskell is a set of Haskell libraries and (WIP)clickhouse-lock CLI
which allows you to build reliable integrations with ClickHouse DBMS

Library API allows you to:

  1. (WIP) naturally validate backward compatability
    between application and DBMS on CI/CD
  2. avoid manual testing of queries-parsers matching
  3. generate encoders/parsers transparently
  4. generate data mappers on reading/writing
  5. avoid writing SQL without any DSL

Library API restricts you by:

  1. relying on using fixed Tables/Table functions
    as read/write models contracts
  2. relying on database first development approach
  3. benefiting via usage of CQRS pattern

Interesting

Library documentation

ClickHouse documentation

Wikipedia article about CQRS

Development

clickhaskell's People

Contributors

kovalevdima avatar 4562448 avatar lipranu avatar alleksandrgall avatar diamondy4 avatar

Stargazers

Daniel Kahlenberg avatar Anton Novikov avatar  avatar

Watchers

Artem Kanev avatar  avatar  avatar  avatar

Forkers

diamondy4 lipranu

clickhaskell's Issues

Redesign table function parameters passing

Right now our design looks like calling single function which takes all of the view parameters inferred from View definition
image

Internally its hardcoded definition for 1 to 4 number of parameters:

instance
  ( KnownSymbol name
  ) => InterpretableTable (View name columns '[Parameter paramName (chType :: Type)])
  where
  type TableInterpreter (View name columns '[Parameter paramName chType]) = Parameter paramName chType -> View name columns '[]
  interpretTable MkParameter{renderedParameter} =
    MkView
      { viewName = (BS.byteString . BS8.pack . symbolVal @name) Proxy
      , inpterpretedParameters = [renderedParameter]
      }


instance
  ( KnownSymbol name
  ) => InterpretableTable 
    ( View
      name
      columns
     '[ Parameter paramName chType
      , Parameter paramName2 chType2
      ]
    )
  where

  type TableInterpreter
    ( View
      name
      columns
     '[ Parameter paramName chType
      , Parameter paramName2 chType2
      ]
    ) 
    =  Parameter paramName chType
    -> Parameter paramName2 chType2
    -> View name columns '[]
  interpretTable parameter1 parameter2 =
    MkView
      { viewName = (BS.byteString . BS8.pack . symbolVal @name) Proxy
      , inpterpretedParameters = [renderedParameter parameter1, renderedParameter parameter2]
      }

-- The same code for 3 and 4 parameters
...

We need to reduce code duplication and provide a nice and stable interface for any number of view arguments

Make an examples of library usage in compilable Markdown Literate Haskell

It's a great approach to make examples written as compilable markdown-compatible lhs files since we could have validated examples of our API written in markdown. It could be done via markdown support plugin for Literate Haskell markdown-unlit. It would open for us an ability to generate actual library API documentation.

I think it would be great to make two example cabal projects as first step:

  1. Writing into the table
  2. Reading from the parametrized view

Redesign plan to achieve library API stability

The problems description

Right now we have an awkward library API for the ClickHouse HTTP Interface
Which brings us to the next problems:

  1. We should write heavy hardcoded extensions of general interpretClient function for every ClickHouse HTTP interface usage patterns. And then use function interpretClient by specifying the extension.
  2. Every extension of interpretClient function repeats injections of ClickHouse HTTP interface parts

ClickHouse HTTP interface usage patterns

Reading

Every reading process has the following general steps:

  1. Client sends request with statement and auth data to a DBMS server
  2. DBMS server handles statement and responses with the formatted data or an error message in http-body
  3. Client handles result of database response
flowchart LR
        http-client

        subgraph request
            http-request[[http-request
                <hr>headers: user, password, database,... 
                        body: statement
                        ]]
        end

  
        dbms-server --> isSuccess
        isSuccess --> |yes| success-http-response
        isSuccess --> |no| error-http-response
        error-http-response --> http-client
        success-http-response --> http-client
    
        http-client --> http-request
        dbms-server
        subgraph response
            isSuccess{is success}
            success-http-response[[http-response <hr> body: formatted data]]
            error-http-response[[http-response <hr> body: error message]]
        end
        http-request --> dbms-server

Loading

Writing

 
flowchart LR

      http-client

      subgraph request
        http-request[[http-request
        <hr>headers: user, password, database,... 
                body: statement+formatted data]]
      end
      dbms-server --> isSuccess
      isSuccess --> |yes| success-http-response
      isSuccess --> |no| error-http-response
      error-http-response --> http-client
      success-http-response --> http-client
      http-request --> dbms-server

      dbms-server
      subgraph response
        isSuccess{is success}
        success-http-response[[http-response <hr> body: empty]]
        error-http-response[[http-response <hr> body: error message]]
      end
      http-client --> http-request
Loading

Solutions

  1. We should provide a single function per usage case of HTTP interface which is independent from chosen Haskell HTTP client implementation.
  2. Provide functions for injecting HTTP interface usage cases into chosen Request and Response types

Migrate from TSV to RowBinary

Our (de)serialization code makes a lot of arrays manipulation since we need to work with escaping. It also makes code harder to understand and degrades performance. Rewriting the parsing and serialization to work with RowBinary format would make the code much more performant and simpler

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.