A simple way to test Hadoop and Hive using maven.

By default, we're set to download a local copy of Hadoop when you first build Hive Test, or whenever the project is cleaned, just before running our test cases.

You can force a redownload and installation of Hadoop by manually activating the download-hadoop profile

mvn --activate-profiles download-hadoop test

You can also perform the download and extraction process independent of testing.

Download Hadoop (into the maven target directory)

mvn --activate-profiles download-hadoop wagon:download-single

Extract Hadoop (into the maven target directory)

mvn --activate-profiles download-hadoop exec:exec

We'll skip attempting to download and use a local copy of Hadoop if any of the following are true

• set your HADOOP_HOME environment variable to a hadoop distribution
• hadoop tar extracted to $home/hadoop/hadoop-0.20.2_local

Hive Test will work so long as you have Hadoop 0.20.X in your path, i.e. /usr/bin/hadoop. In this case, you'll want to deactivate the hadoop download.

mvn --activate-profiles -download-hadoop test

Hive_test gives us an embedded Hive including an embedded Derby database, and a local HiveThriftService. This allows us to create unit tests very easily.

public class ServiceExampleTest extends HiveTestService {

  public ServiceExampleTest() throws IOException {
    super();
  }

  public void testExecute() throws Exception {
    Path p = new Path(this.ROOT_DIR, "afile");

    FSDataOutputStream o = this.getFileSystem().create(p);
    BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(o));
    bw.write("1\n");
    bw.write("2\n");
    bw.close();

    client.execute("create table  atest  (num int)");
    client.execute("load data local inpath '" + p.toString() 
      + "' into table atest");
    client.execute("select count(1) as cnt from atest");
    String row = client.fetchOne();
    assertEquals(row, "2");
    client.execute("drop table atest");
  }
}