Giter Club home page Giter Club logo

core-grpc-postgres-connector's Introduction

Example Postgres GRPC Connector

As of 1 July 2020, Qlik Core is no longer available to new customers. No further maintenance will be done in this repository.

The Example Postgres GRPC Connector shows how to load data into Qlik Associative Engine from Postgres using a dockerized connector built in Golang. It streams the data asynchronously using go channels though the following components before sending it onto Qlik Associative Engine.

  • postgres_reader - reads the data from the database into reasonably sized SQL data chunks.
  • async_translator - takes the SQL data chunks and translates them into GRPC data chunks.
  • async_stream_writer - takes the GRPC data chunks and writes them onto the GRPC stream.

The reason for the division is to be able to utilize multiple CPU cores to process the different stages simultaneously.

Example

The /example directory defines a simple stack of services using docker-compose:

  • Qlik Associative Engine
  • Postgres GRPC Connector
  • Postgres Database
  • Node Test Runner (only used for automated testing)

The script in example/reload-runner is used to instruct Qlik Associative Engine to load the example data (originally defined in example/postgres-image/airports.csv) using the connector.

Steps to run the example

Run in a *nix environment (or Git Bash if on Windows), note that you must accept the Qlik Core EULA by setting the ACCEPT_EULA environment variable:

$ cd example
$ ACCEPT_EULA=yes docker-compose up -d --build
$ cd reload-runner
$ npm install
$ npm start

core-grpc-postgres-connector's People

Contributors

axelssonhakan avatar gabbaxx avatar janmiderback avatar peol avatar qlikossbuild avatar renovate[bot] avatar stefanenberg avatar sublibra avatar wennmo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

sakaga

core-grpc-postgres-connector's Issues

Readify postgres

This issue contains a checklist of tasks that is needed before repo can be open sourced.

Checklist

  • LICENSE file added to repo with a MIT license.
  • Update package.json with the right info for author, description, license and name.
  • Linting of code e.g. airbnb for node.js repo.
  • Circle CI build to secure quality e.g. linting, tests, docker image.
  • Update README.md to contain relevant info for the repo e.g. Contributions
  • Add .editorconfig, .gitignore and .gitattributes

clock_timestamp => <Unsupported format>

Running the example/test script (example/reload-runner/index.js) will create a table that contains for clock_timestamp.

This could be a possible bug and has to be investigated further

Create GRPC connector recipe

Should include a docker-compose with all needed parts like a PostGres image with a database include.

A simple nodejs service to validate that it works.

doReload method kill Qlik Core container

Hi,

I have an error with doReload() method into core-grpc-postgres-connector/example/reload-runner/index.js line 64

const reloadPromise = app.doReload();

Docker Qlik Core Error return

{"timestamp":"2019-01-04T16:27:20.520Z","log_type":"System","logseverity":"INFO","thread_id":1820976896,"user_id":"Personal\\Me","tenant_id":"default","app_id":"0","message":"CGenericConnection: Call to Terminate process (reason 996) with invalid handle; "}
{"timestamp":"2019-01-04T16:27:20.520Z","log_type":"System","logseverity":"INFO","thread_id":1820976896,"user_id":"Personal\\Me","tenant_id":"default","app_id":"0","message":"CGenericConnection: Call to Terminate process (reason 996) with invalid handle; "}

Regards

Readify postgres-grpc-connector

This issue contains a checklist of tasks that is needed before repo can be open sourced.

Checklist

  • LICENSE file added to repo with a MIT license.
  • Update package.json with the right info for author, description, license and name.
  • Linting of code e.g. airbnb for node.js repo.
  • Circle CI build to secure quality e.g. linting, tests, docker image.
  • Update README.md to contain relevant info for the repo e.g. Contributions
  • Add .editorconfig, .gitignore and .gitattributes

Automated: Protex scan failed

A BlackDuck Protex scan failed on this repository.

Number of files needing identification: 2

If you are unsure how this process works, please contact trunk team on Slack.

No more automated issues will be created on this repository until this issue
is closed.

Please prioritize and solve this as soon as possible. If the files
has been identified already, you may ignore and close this issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.