Giter Club home page Giter Club logo

avrokdb's Introduction

avrokdb

Avro

GitHub release (latest by date) Travis (.com) branch

Introduction

This interface allows kdb+ to users encode and decode Apache Avro serialized data.

This is part of the Fusion for kdb+ interface collection.

New to kdb+ ?

Kdb+ is the world's fastest time-series database, optimized for ingesting, analyzing and storing massive amounts of structured data. To get started with kdb+, please visit https://code.kx.com/q/ for downloads and developer information. For general information, visit https://kx.com/

New to Apache Avro?

Apache Avro is a data serialization system.

Avro provides:

  • Rich data structures.
  • A compact, fast, binary data format.

Avro relies on schemas which are defined with JSON. When Avro data is read, the schema used when writing has to be provided. This permits each datum to be written with no per-value overheads, making serialization both fast and small.

Installation

Requirements

  • kdb+ ≥ 3.5 64-bit (Linux/macOS/Windows)

Installing a release

It is recommended that a user install this interface using a release package. This is completed in a number of steps:

  1. If running on Windows, ensure you have downloaded/installed the Avro C++ libraries following the instructions here. This step is not necessary on Linux or macOS because their avrokdb release packages are statically linked with libavrocpp.
  2. Download a release for your system architecture.
  3. Install script avrokdb.q to $QHOME, and binary file lib/arrowkdb.(so|dll) to $QHOME/[mlw](64), by executing the following from the unzipped release package directory:
## Linux/macOS
chmod +x install.sh && ./install.sh

## Windows
install.bat

Building and installing from source

Requirements

  • kdb+ ≥ 3.5 64-bit (Linux/macOS/Windows)
  • Apache Avro C++ libraries
  • C++11 or later and
  • CMake ≥ 3.1.3

Third-party library installation

Linux

On linux avrocpp should be built from source.

  1. Install the avrocpp dependencies (boost and compression libraries).

    With an apt package manager:

    sudo apt -y update
    sudo apt -y install libboost-dev libboost-filesystem-dev libboost-iostreams-dev libboost-program-options-dev libsnappy-dev

    With a yum package manager:

    sudo yum -y update
    sudo yum -y install boost-devel boost-filesystem boost-iostreams boost-program-options snappy-devel
  2. Clone the avro repo and switch to the release-1.11.2 tag (which is the one used to build the avrokdb linux package)

    git clone https://github.com/apache/avro.git
    cd avro
    git checkout refs/tags/release-1.11.2 --
  3. Create cmake build and install directories:

    cd lang/c++
    mkdir build
    mkdir install
    export AVRO_INSTALL=$(pwd)/install
    cd build
  4. Generate the cmake build scripts:

    cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DCMAKE_INSTALL_PREFIX=$AVRO_INSTALL ..
  5. Build and install to $AVRO_INSTALL:

      cmake --build . --config Release
      cmake --build . --config Release --target install
  6. Copy the shared object to $QHOME/l64:

    cp $AVRO_INSTALL/lib/libavrocpp.so $QHOME/l64

macOS

On macOS avrocpp can be installed using brew:

brew install avro-cpp

Windows

On Windows avrocpp should be built using vcpkg:

  1. Clone the vcpkg repo and bootstrap it:

    git clone https://github.com/microsoft/vcpkg.git
    cd vcpkg
    bootstrap-vcpkg.bat
  2. Install avrocpp:

    vcpkg install avro-cpp:x64-windows
    set AVRO_INSTALL=%cd%\installed\x64-windows
  3. Copy the DLLs to $QHOME\w64:

    cd installed\x64-windows\bin
    copy avrocpp.dll %QHOME%\w64
    copy boost_iostreams-vc143-mt-x64-*.dll %QHOME%\w64
    copy liblzma.dll %QHOME%\w64
    copy zstd.dll %QHOME%\w64
    copy bz2.dll %QHOME%\w64
    copy zlib1.dll %QHOME%\w64

Building avrokdb

In order to successfully build and install this interface from source, the following environment variables must be set:

  1. AVRO_INSTALL = Location of the Avro C++ API release (only required if avrocpp is not installed globally on the system, e.g. on Linux or Windows where avrocpp was built from source)
  2. QHOME = Q installation directory (directory containing q.k)

From a shell prompt (on Linux/macOS) or Visual Studio command prompt (on Windows), clone the avrokdb source from github:

git clone https://github.com/KxSystems/avrokdb.git
cd avrokdb

Create the cmake build directory and generate the build files (this will use the system's default cmake generator):

mkdir build
cd build

## Linux (using the Arrow installation which was build from source as above)
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DAVRO_INSTALL=$AVRO_INSTALL

## macOS
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11

## Windows (using the Arrow installation which was build from source as above)
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=11 -DAVRO_INSTALL=%AVRO_INSTALL%

Start the build:

cmake --build . --config Release

Create the install package and deploy to $QHOME:

cmake --build . --config Release --target install

Documentation

Documentation outlining the functionality available for this interface can be found in the docs folder.

Status

The avrokdb interface is provided here under an Apache 2.0 license.

If you find issues with the interface or have feature requests, please consider raising an issue.

If you wish to contribute to this project, please follow the contribution guide.

avrokdb's People

Contributors

nmcdonnell-kx avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.