vertica / vertica-hadoop-connector Goto Github PK

I wrote a Map Class using Vertica as Map input source.
when I used vertica.get(int pos) this function, the program can run and the result is correct.
But When I used vertica.get(String name) function, the program cannot run and told me the column is not exist.

My input query is
VerticaInputFormat.setInput(job,"select user_id,movie_id,rating from trainer.movie_rating");

In my Map Class code i just changed
Long rating=(Long) value.get(2);
to
Long rating=(Long) value.get("rating"); // I also tried movie_rating.rating

So I checked VerticaRecord Class code. I found that VerticaRecord use ResultSetMetaData.getCatalogName to get column name. But return the column name is empty String. Howerer if change to ResultSetMetaData.getColumnName that is right.

I guess may be this is a issue. If there is any way to help me resolve it. Very thanks for your help.

Hi, guys!

Can you provide some tips for improving performance of data loading from hive table to Vertica cluster?

My use case of using Vertica is
Spark jobs for preparing data (some transformations of business data), and result of jobs write to hive table in ORC format. After that, I run the sh script for loading to vertica
COPY vertica_schema.$my_table_new FROM $table_path_hdfs ON ANY NODE ORC.

Performance of this solution is terrible, for example loading of data with value of 1.5B rows takes > 4 hours. So, how to improve it?

Vertica cluster hosts: 6 machines
Hadoop (hive) cluster: 30 machines

https://github.com/vertica/Vertica-Hadoop-Connector/blob/master/hadoop-connector/com/vertica/hadoop/VerticaConfiguration.java#L209