vertica / vertica-hadoop-connector Goto Github PK
View Code? Open in Web Editor NEWVertica Hadoop Connector
License: Apache License 2.0
Vertica Hadoop Connector
License: Apache License 2.0
**************************** * Vertica Analytic Database * * Apache Hadoop & Pig Connector * * Copyright 2012 Vertica Systems, an HP company 2012 **************************** This directory contains jars, examples, doc & source code for Apache Hadoop & Pig connectors to the Vertica Analytic Database. IMPORTANT: If you wish to contribute anything to this repository, in order for us to accept your pull request you MUST sign and send a copy of the appropriate Contributor License Agreement to Vertica ([email protected]): license/PersonalCLA.pdf: If you are contributing for yourself license/CorporateCLA.pdf: If you are contributing on behalf of your company Directory structure: ./hadoop-vertica Hadoop to Vertica Connector ./pig-vertica Pig to Vertica Connector ./hadoop-vertica-example Example for using the Hadoop Connector ./squeal Squeal - a pig to sql translator. Compile: Set the following environment variables: HADOOP_HOME - Directory containing the hadoop installation. PIG_HOME - Directory containing Apache Pig installation. PIG_JAR- Name of pig jar. BUILDDIR - Temporary directory for storing build files. pig_version- Pig version. VERTICA_JAR- Path to Vertica JDBC jar. JAR_DIR - Directory where the new jar files & docs should be stored. Run - make jar TODO: Use Maven to download dependencies. helpful commands for running examples. Examples for pig connector. Examples for Hadoop streaming.
I wrote a Map Class using Vertica as Map input source.
when I used vertica.get(int pos) this function, the program can run and the result is correct.
But When I used vertica.get(String name) function, the program cannot run and told me the column is not exist.
My input query is
VerticaInputFormat.setInput(job,"select user_id,movie_id,rating from trainer.movie_rating");
In my Map Class code i just changed
Long rating=(Long) value.get(2);
to
Long rating=(Long) value.get("rating"); // I also tried movie_rating.rating
So I checked VerticaRecord Class code. I found that VerticaRecord use ResultSetMetaData.getCatalogName to get column name. But return the column name is empty String. Howerer if change to ResultSetMetaData.getColumnName that is right.
I guess may be this is a issue. If there is any way to help me resolve it. Very thanks for your help.
Hi, guys!
Can you provide some tips for improving performance of data loading from hive table to Vertica cluster?
My use case of using Vertica is
Spark jobs for preparing data (some transformations of business data), and result of jobs write to hive table in ORC format. After that, I run the sh script for loading to vertica
COPY vertica_schema.$my_table_new FROM $table_path_hdfs ON ANY NODE ORC
.
Performance of this solution is terrible, for example loading of data with value of 1.5B rows takes > 4 hours. So, how to improve it?
Vertica cluster hosts: 6 machines
Hadoop (hive) cluster: 30 machines
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.