Giter Club home page Giter Club logo

vertica-hadoop-connector's Introduction

****************************
* Vertica Analytic Database
*
* Apache Hadoop & Pig Connector
*
* Copyright 2012 Vertica Systems, an HP company 2012
****************************

This directory contains jars, examples, doc & source code for 
Apache Hadoop & Pig connectors to the Vertica Analytic Database.

IMPORTANT: If you wish to contribute anything to this repository, in
order for us to accept your pull request you MUST sign and send a copy
of the appropriate Contributor License Agreement to Vertica
([email protected]):

license/PersonalCLA.pdf: If you are contributing for yourself
license/CorporateCLA.pdf: If you are contributing on behalf of your company

Directory structure:
  ./hadoop-vertica    Hadoop to Vertica Connector
  ./pig-vertica       Pig to Vertica Connector
  ./hadoop-vertica-example Example for using the Hadoop Connector
  ./squeal            Squeal - a pig to sql translator.

Compile:
   Set the following environment variables: 
   HADOOP_HOME - Directory containing the hadoop installation.
   PIG_HOME   - Directory containing Apache Pig installation.
   PIG_JAR-     Name of pig jar.
   BUILDDIR   - Temporary directory for storing build files.
   pig_version- Pig version.
   VERTICA_JAR- Path to Vertica JDBC jar.
   JAR_DIR    - Directory where the new jar files & docs should be stored.

   Run - make jar
TODO: 
    Use Maven to download dependencies.
	helpful commands for running examples.
	Examples for pig connector.
	Examples for Hadoop streaming.

vertica-hadoop-connector's People

Contributors

alamb avatar pxuan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vertica-hadoop-connector's Issues

Using VerticaRecord get("String name") function

I wrote a Map Class using Vertica as Map input source.
when I used vertica.get(int pos) this function, the program can run and the result is correct.
But When I used vertica.get(String name) function, the program cannot run and told me the column is not exist.

My input query is
VerticaInputFormat.setInput(job,"select user_id,movie_id,rating from trainer.movie_rating");

In my Map Class code i just changed
Long rating=(Long) value.get(2);
to
Long rating=(Long) value.get("rating"); // I also tried movie_rating.rating

So I checked VerticaRecord Class code. I found that VerticaRecord use ResultSetMetaData.getCatalogName to get column name. But return the column name is empty String. Howerer if change to ResultSetMetaData.getColumnName that is right.

I guess may be this is a issue. If there is any way to help me resolve it. Very thanks for your help.

Performace tips

Hi, guys!

Can you provide some tips for improving performance of data loading from hive table to Vertica cluster?

My use case of using Vertica is
Spark jobs for preparing data (some transformations of business data), and result of jobs write to hive table in ORC format. After that, I run the sh script for loading to vertica
COPY vertica_schema.$my_table_new FROM $table_path_hdfs ON ANY NODE ORC.

Performance of this solution is terrible, for example loading of data with value of 1.5B rows takes > 4 hours. So, how to improve it?

Vertica cluster hosts: 6 machines
Hadoop (hive) cluster: 30 machines

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.