| tags | projects | ||||
|---|---|---|---|---|---|
|
|
This guide walks you through testing a simple Spring YARN project.
What you’ll build
You’ll use an existing sample application and create a simple JUnit test to verify application is executed properly.
What you’ll need
https://raw.githubusercontent.com/spring-guides/getting-started-macros/master/prereq_editor_jdk_buildtools.adoc - Local single-node instance based on Hadoop 2.2.0 or later
|
Note
|
Testing this sample application don’t need existing or running Hadoop instance. |
If you’re not familiar with gradle, refer to Building Java Projects with Gradle.
Set up the project
Focus on this guide is to use JUnit to test your sample application. We have already prepared a project for gradle and application files based on sample YARN Basic Sample. What is left for you is to create a JUnit test class.
Create a JUnit Test Class
gs-yarn-testing-dist/src/test/java/hello/AppIT.java
link:complete/gs-yarn-testing-dist/src/test/java/hello/AppIT.java[]Let’s go through step by step what’s happening in this JUnit class. As already mentioned earlier we don’t need any existing or running Hadoop instances, instead testing framework from Spring YARN provides an easy way to fire up a mini cluster where your tests can be run in an isolated environment.
We named a class as AppIT which plays nice with both gradle and
maven when tests are executed during the build. Tests will need a real
artifacts to be build which means test can only run after jar files
has been build and processed by a Spring Boot’s repackage plugin.
In gradle build file we make sure artifacts are build before tests are
run. In maven build file we use failsafe plugin which is meant for
integration testing and tests handled by it are executed after all
project artifacts has been build.
-
We used
spring.yarn.client.filesproperty to override settings defined inapplication.ymlfile because test is run fromgs-yarn-testing-distproject. -
@MiniYarnClusterTestis a composed annotation telling Spring to start a Hadoop’s mini cluster having components forHDFSandYARN. Hadoop’s configuration from this minicluster is automatically injected into your testing context. -
AbstractBootYarnClusterTestsis a class containing a lot of base functionality what you need in your tests.
Then it’s time to deploy the application into a running minicluster
-
submitApplicationAndWait()method simply runs yourClientApplicationand expects it to do an application deployment. On default it will wait 60 seconds an application to finish and returns a current state. -
We make sure that we have a correct application state
We use ContainerLogUtils to find our container logs files from a minicluster.
-
We assert count of a log files
-
We expect some specified content from log file
-
We expect
stderrfiles to be empty
Run Test
If using gradle:
./gradlew clean buildIf using maven:
mvn clean packageIf test is executed succesfully you’d be able to see YARN container logs within the minicluster logging directory:
$ find gs-yarn-testing-dist/target/|grep std
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stderr
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stderr$ grep Hello gs-yarn-testing-dist/target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
[2014-02-14 16:37:54.278] boot - 18453 INFO [main] --- HelloPojo: Hello from HelloPojo
[2014-02-14 16:37:54.278] boot - 18453 INFO [main] --- HelloPojo: About to list from hdfs root content
[2014-02-14 16:37:55.157] boot - 18453 INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
[2014-02-14 16:37:55.157] boot - 18453 INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/app; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}Summary
Congratulations! You have now created a simple yet effective JUnit
test for building Spring YARN projects.
See Also
The following guides may also be helpful:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent