aist-oceanworks / mudrod Goto Github PK

EsipSRIOntology.java sets the basis for utilizing the entire ESIP community ontology repository.

The implementation needs to be finished off such that it can interact with the ORR Ont REST API.

We are using Elasticsearch version 2.3, which only supports up to spark 1.6.1. If we would like to use the Spark 2.0 compatibility, we will need to switch to using the 5.0.0-alpha5 version. However, it is an alpha release version and should be used for testing purposes only (i.e. not to be used in production).

Do you have any suggestion?

refer: https://discuss.elastic.co/t/write-es-error-with-spark-2-0-release/56967/3

As discussed in todays ESIP Mudrod Meetup, we should build a Docker container which packages the entire framework and makes it easy to use.
@Yongyao @quintinali I'll work with you both to achieve this.

Error:
Description Resource Path Location Type
org.apache.maven.archiver.MavenArchiver.getManifest(org.apache.maven.project.MavenProject, org.apache.maven.archiver.MavenArchiveConfiguration) pom.xml /mudrod-core line 1 Maven Configuration Problem

I am not familiar with Maven config, but I found a discussion here @lewismc :
http://stackoverflow.com/questions/37555557/m2e-error-in-mavenarchiver-getmanifest

an integrated GUI, DB,

@lewismc

1. In eclipse, right click on murdrod-service, run as -> run on server -> tomcat 7, you would see the error below.

This tomcat problem has been solved. Looks like tomcat messed up old mudrod project and the new code. Removing servers and start a new configuration has solved this issue. But jetty:run issue never go away, there might be something wrong with pom, I guess.

java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mudrod-service]]

1. With mvn jetty:run, it says no plugin found for prefix "jetty" in the current project ...

Link to the very first version of SDD: https://docs.google.com/document/d/1fMxCUW1gdOU14u0CVQ_Q3uHpFbBSIpR7rGmUytAujVc/edit

web log ingesting; (Yongyao)
session reconstruction; (Yun)
vocabulary semantic relationship extraction; (Yongyao)
ontology navigation; (to be added, Lewis)
search ranking; (to be extended, Yongyao)
recommendation (to be added, Yun)

Perform log ingestion tests to benchmark log ingestion speed.

This is when running a logIngest from the temp0907 branch with some local modifications I've been working on.

lmcgibbn@LMC-032857 /usr/local/mudrod(temp0907) $ ./core/target/appassembler/bin/mudrod -f -logDir /usr/local/mudrod_resources/logDir
java.lang.RuntimeException: Cannot determine the number of rows because it is not specified in the constructor and the rows RDD is empty.
        at scala.sys.package$.error(package.scala:27)
        at org.apache.spark.mllib.linalg.distributed.RowMatrix.numRows(RowMatrix.scala:77)
        at esiptestbed.mudrod.utils.MatrixUtil.exportToCSV(MatrixUtil.java:504)
        at esiptestbed.mudrod.weblog.pre.ClickStreamGenerator.execute(ClickStreamGenerator.java:64)
        at esiptestbed.mudrod.discoveryengine.WeblogDiscoveryEngine.preprocess(WeblogDiscoveryEngine.java:125)
        at esiptestbed.mudrod.main.MudrodEngine.startFullIngest(MudrodEngine.java:158)
        at esiptestbed.mudrod.main.MudrodEngine.main(MudrodEngine.java:357)

@lewismc In case you missed the issue in pull request. This is the command I use: -l -logDir C:\mudrodLogIngest

ERROR main.MudrodEngine - MudrodEngine: 'logDir' argument is mandatory. User must also provide either 'logIngest' or 'fullIngest'.
java.lang.NullPointerException
at esiptestbed.mudrod.main.MudrodEngine.main(MudrodEngine.java:213)

Am I doing it wrong?

@lewismc @quintinali This is the link to the most up-to-date user manual. Please update it with your newly developed functions.

https://docs.google.com/document/d/1iwWa8Rem9ktOZmUsA1tvlMKcnZcwDBJjN2AFeM5rW3o/edit

0. installation or docker container (Lewis)
   

1. web log ingesting; (CLI, yongyao)
2. session reconstruction; (add how to run command line, Yun)
3. vocabulary semantic relationship extraction; (add how to run command line, Yongyao)
4. ontology navigation; (Lewis)
5. search ranking; (Yongyao)
6. recommendation (Yun)

Implement a $MUDROD_HOME/bin/mudrod script which will provide users with options to execute log ingestion, log deletion, etc. from the command line e.g. $MUDROD_HOME/bin/mudrod logimport /path/to/log, this will then run https://github.com/mudrod/mudrod/blob/war/core/src/main/java/esiptestbed/mudrod/weblog/pre/ImportLogFile.java with the relevant input parameter(s) in an ad-hoc fashion rather than EVERY time mud rod is started.

Right now the InportLogFile.java is a bit of a mess in terms of use of efficient data structures, documentation and general presentation.
PR coming up to solve this.

Slf4j is a simple but powerful logging framework fro Java. When writing Java code I typically always use Slf4j-over-Log4j for implementing application logging.
I am going to begin working on replacing all of the System.out.println with correct application logging.

The following screenshot is an example of the issues flagged within the findbugs report when we execute mvn clean install.
We should address these issues and improve the quality of the mudrod codebase.

The Commons Command Line Interface library is a very widely used implementation for all CLI logic.
https://commons.apache.org/proper/commons-cli/
I propose to use it for all Mudrod tools such that we can create better CLI code.

Write the implementations and also write tests to validate.

Hi @Yongyao can you please add a publications page to the Mudrod wiki page?
https://github.com/mudrod/mudrod/wiki
Then link to it from https://github.com/mudrod/mudrod#publications

At a bare minimum we should have auto completion functionality.

We need to add in the ability for users to select suggested entities being provided by the underlying ontology processing. These should be hyperlinked such that they invoke faceted search and can be used to slice datasets.

Looking at the ranked query results as below

We need to

• choose a 'brighter' color of green... the dark green does not stuck out very well and may be difficult for some people to read.
• ensure that all columns are uniformly formatted. Right now the text seems to loop back on itself... this is untidy and can be improved.
• have an expansion button for every abstract such that users can simply click to expand the abstract text should they wish. I think that this is called 'accordion' in JQuery UI... however I may be wrong.

@quintinali can you please have a look at this as i think you are much better at UI than me?

Based on the availability of ontology-enhanced search within Mudrod. We should at some stage have an underlying triples store. This needs to be discussed with the project team.

@lewismc I tried to deploy MUDORD on VM(199.26.254.146) following docker user guide https://github.com/mudrod/mudrod/tree/master/docker

When I run the command docker run -t -i mudrod/mudrod /bin/bash, the system considered it as unknow command.

Could you please take a look?

Right now, we are working on this branch, which is the only branch that has ranking and recommendation functions. https://github.com/mudrod/mudrod/tree/temp0907
After service is started, it works well in the beginning, but automatically shuts down after 2~4 hours. No error log is found either in jetty log or mudrod.log.

Just to let you know, in the servlet web listener, we started one ES transport client and one spark driver. I guess this is might be something about memory leak. So the solution I come up with is to modify JVM heap memory size when starting jetty. (not tested yet)

export MAVEN_OPTS="-Xmx2048m -Xms2048m" && mvn jetty:run

Do this make sense to you? BTW, if we copy mudrod.snapshot.war to tomcat webapp, it can be started.

FindBugs looks for bugs in Java programs. It is based on the concept of bug patterns. A bug pattern is a code idiom that is often an error. Bug patterns arise for a variety of reasons:

Difficult language features
Misunderstood API methods
Misunderstood invariants when code is modified during maintenance
Garden variety mistakes: typos, use of the wrong boolean operator

FindBugs uses static analysis to inspect Java bytecode for occurrences of bug patterns. We have found that FindBugs finds real errors in most Java software. Because its analysis is sometimes imprecise, FindBugs can report false warnings, which are warnings that do not indicate real errors. In practice, the rate of false warnings reported by FindBugs is generally less than 50%.

http://gleclaire.github.io/findbugs-maven-plugin/index.html

SonarQube.com is a cloud service offered by SonarSource and based on SonarQube. SonarQube is a widely adopted open source platform to inspect continuously the quality of source code and detect bugs, vulnerabilities and code smells in more than 20 different languages.

The Mudrod codebase would benefit significantly from an automated quality metric from SonarQube.

More documentation can be found at
https://docs.travis-ci.com/user/sonarqube/
and
https://sonarqube.com/

The speed for most of the queries are fine, but for queries that have lots of returned results such as "ocean temperature", the speed is very slow (>10s).

I am trying solve this problem through multi-threading, but there is an issue that I cannot fix. Maybe I can use some different eyes.

Detailed problem statement: normally, if a>b and b>c, a>c. But this may not be true in our case, because our ranking model is not overfitting, so what I did before is simple but costs too much time.

for(int j=0; j< resultList.size(); j++)
    {
      for(int k=0; k< resultList.size(); k++)
      {
        if(k!=j)
        {
          resultList.get(j).below += comp (resultList.get(j), resultList.get(k));
        }
      }
    }

What I am doing right now is to divide the resultList into multiple parts and run the same code simultaneously, which kind of improved the speed a bit, but not much.

Let me know if you have any ideas.

This includes adding more features, code quality improvement, and so on.

@lewismc The same error is still here. I even tried it on a different machine.

SEVERE: A child container failed during start
java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mudrod-service]]
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1123)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:816)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mudrod-service]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
... 6 more
Caused by: java.lang.NoClassDefFoundError: com/google/gson/JsonElement
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredFields(Unknown Source)
at org.apache.catalina.util.Introspection.getDeclaredFields(Introspection.java:106)
at org.apache.catalina.startup.WebAnnotationSet.loadFieldsAnnotation(WebAnnotationSet.java:270)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationServletAnnotations(WebAnnotationSet.java:139)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:65)
at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:415)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:892)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:386)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5380)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 6 more
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonElement
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 20 more

@lewismc I just installed the SonarQube plugin of eclipse. The question I have is, why am I only seeing 60 bugs on my local SQ, but there are 200+ on the remote one. Is it because different setting are used in local and remote ones? I am also trying to New a SQ server, but I don't have the information below.

BTW, let's say I have another project, how am I supposed to connect it with SQ? Didn't find much document on this issue.

The document I am referencing is http://www.sonarlint.org/eclipse/

Right now, when I attempt to pull and build the Docker container on the remote NSF host, I run into permission issues from within the running container process when attempting to run jobs from the $MUDROD_HOME/bin/mudrod script.
Pull request coming up.

I found this probably is a mistake.
@quintinali Can you check on this? Is the highlighted code supposed to be config.get("clickstreamSVDMatrix_tmp")

SVDAnalyzer svd = new SVDAnalyzer(config, es, spark);
svd.GetSVDMatrix(config.get("clickstreamMatrix"),
Integer.parseInt(config.get("clickstreamSVDDimension")),
config.get("clickstreamSVDMatrix_tmp"));
List tripleList = svd
.CalTermSimfromMatrix(config.get("clickstreamMatrix"));
svd.SaveToES(tripleList, config.get("indexName"),
config.get("clickStreamLinkageType"));

@lewismc I am trying to load the ML classifier model, which is a binary file, from resource folder, but it gives me an reading error. After looking into it, I found the size gets changed after maven build.

java.io.StreamCorruptedException: invalid stream header: EFBFBDEF
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
at java.io.ObjectInputStream.(ObjectInputStream.java:299)
at weka.core.SerializationHelper.read(SerializationHelper.java:275)
at weka.core.SerializationHelper.read(SerializationHelper.java:257)
at esiptestbed.mudrod.ssearch.ranking.Learner.(Learner.java:45)
at esiptestbed.mudrod.ssearch.Ranker.(Ranker.java:46)
at esiptestbed.mudrod.ssearch.Searcher.main(Searcher.java:332)

It looks like an maven problem from this thread: http://stackoverflow.com/questions/19500458/maven-resource-binary-changes-file-size-after-build

I tried a few solution I found online, but all of them would lead to some other maven error. Can you help me fix it? If need, I can send the binary file to you. Thanks.

@lewismc I have searched for ML libraries that can provide "Learning to Rank" APIs for days, but I still cannot find a good one.

http://www.cs.cornell.edu/people/tj/svm_light/svm_rank.html (Written in C++)
https://github.com/jattenberg/RankLib (Written in Java, but does not provide API, and no comment at all)

Inspired by this Python code: http://fa.bianp.net/blog/2012/learning-to-rank-with-scikit-learn-the-pairwise-transform/

I decided to implement a RankSVM by myself, which won't be very hard. I would say 2 days is enough.

So, what are your thoughts on this?

Both @quintinali and I failed to start the service after updating the code, already spent several hours on it, but still didn't get any clue. @lewismc , can you help take a look? This might be related to some changes recently committed.

It saysSLF4J cannot be found, but I didn't see anyone change the maven dependency.

?? 28, 2016 5:44:35 ?? org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class esiptestbed.mudrod.webservlet.MudrodWebListener
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory

Every time Spark starts, the error below is printed out in the logs. It doesn't not affect the results, and Spark still works, but I think this looks bad to the testers. @quintinali didn't find a way to fix it, @lewismc can you help take a look.

I am testing the new code right now, and should be able to get back to you by the end of Monday.

2016-07-24 19:37:08,322 WARN org.spark-project.jetty.util.component.AbstractLifeCycle: FAILED [email protected]:4040: java.net.BindException: Address already in use: bind
java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.Net.bind(Unknown Source)
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:228)
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.(SparkContext.scala:448)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:61)
at esiptestbed.mudrod.driver.SparkDriver.(SparkDriver.java:31)
at esiptestbed.mudrod.main.MudrodEngine.(MudrodEngine.java:48)
at esiptestbed.mudrod.main.MudrodEngine.main(MudrodEngine.java:122)

Upon analysis of local catalina.out log, I see the following

SEVERE: Error listenerStart
Sep 15, 2016 2:01:18 PM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [/mudrod-service-0.0.1-SNAPSHOT] startup failed due to previous errors
Sep 15, 2016 2:01:18 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/mudrod-service-0.0.1-SNAPSHOT] appears to have started a thread named [elasticsearch[Mar-Vell][[timer]]] but has failed to stop it. This is very likely to create a memory leak.

There are a number of other issues noted within my local log which I am going to work through however for the time being this one will be dealt with first.

We should be able to use ontology(ies), in particular classes, to enhance a users search through knowledge-based refinement. We need to implement some sort of term suggestion based on classes present within earth science ontologies as per the Ontology Integration Action Plan.

Right now in ESDriver.java we are ALWAYS creating a new Elasticsearch Node as per the following constructors

  public ESDriver(String clusterName){
    node =
        nodeBuilder()
        .settings(ImmutableSettings.settingsBuilder().put("http.enabled", false))
        .clusterName(clusterName)
        .client(true)
        .node();

    client = node.client();
  }

  public ESDriver(Map<String, String> config){
    Settings settings = System.getProperty("file.separator").equals("/") ? ImmutableSettings.settingsBuilder()
        .put("http.enabled", "false")
        .put("transport.tcp.port", config.get("ES_Transport_TCP_Port"))
        .put("discovery.zen.ping.multicast.enabled", "false")
        .put("discovery.zen.ping.unicast.hosts", config.get("ES_unicast_hosts"))
        .build() : ImmutableSettings.settingsBuilder()
        .put("http.enabled", false)
        .build();;

        node =
            nodeBuilder()
            .settings(settings)
            .clusterName(config.get("clusterName"))
            .client(true)
            .node();
        client = node.client();
  }

This is not the preferred way to interact with an existing Elasticsearch cluster and we should use the more efficient TransportClient instead.

    // Prefer TransportClient
    if (hosts != null && port > 1) {
      TransportClient transportClient = TransportClient.builder().settings(settings).build();
      for (String host: hosts)
        transportClient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(host), port));
      client = transportClient;
    } else if (clusterName != null) {
      node = nodeBuilder().settings(settings).client(true).node();
      client = node.client();
    }

This will significantly improve the performance of Mudrod initiation.

In mudrod-core, we DO NOT declare but actually DO use both of the following dependencies

[WARNING]    commons-lang:commons-lang:jar:2.5:compile
[WARNING]    org.apache.commons:commons-lang3:jar:3.3.2:compile

... the issues is however that we only use one function from each dependency, and only on one occasion. We should probably try to write out own implementations and reduce the dependency usage.

lmcgibbn@LMC-056430 /usr/local/mudrod/core(master) $ grep -lr "org.apache.commons.lang" .
./src/main/java/esiptestbed/mudrod/driver/ESDriver.java
./src/main/java/esiptestbed/mudrod/recommendation/structure/OHCodeExtractor.java

add comments for utils, semantics, metadata, web logs

integrate similarity results

Right now, application logs just sit and wait, for quite some time... ideally we would have a command line progress bar which would indicate progress. This issue should provide this functionality for system administrators to see job progress.

I was trying to start service in eclipse. Please see the error below.

2016-09-16 20:35:48,460 ERROR [localhost]./mudrod-service - Exception sending context initialized event to listener instance of class esiptestbed.mudrod.webservlet.MudrodWebListener
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: jar:file:/C:/Eclipse%2520workspace2/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/mudrod-service/WEB-INF/lib/mudrod-core-0.0.1-SNAPSHOT.jar!/javaSVMWithSGDModel
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.(Path.java:172)
at org.apache.hadoop.fs.Path.(Path.java:89)
at org.apache.spark.mllib.util.Loader$.metadataPath(modelSaveLoad.scala:95)
at org.apache.spark.mllib.util.Loader$.loadMetadata(modelSaveLoad.scala:129)
at org.apache.spark.mllib.classification.SVMModel$.load(SVM.scala:100)
at org.apache.spark.mllib.classification.SVMModel.load(SVM.scala)
at esiptestbed.mudrod.ssearch.ranking.Learner.(Learner.java:43)
at esiptestbed.mudrod.ssearch.Ranker.(Ranker.java:45)
at esiptestbed.mudrod.webservlet.MudrodWebListener.contextInitialized(MudrodWebListener.java:61)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5077)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5591)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1574)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1564)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: jar:file:/C:/Eclipse%2520workspace2/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/mudrod-service/WEB-INF/lib/mudrod-core-0.0.1-SNAPSHOT.jar!/javaSVMWithSGDModel
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 18 more

Using master branch, when I am navigating the WebApp, I get the following... which is not right. This needs to be addressed.

Hi @Yongyao, we should establish an automated process which polls the Github master branch, fetches (upon commit to master branch), builds and deploys the Web application and codebase to the Testbed server we have running.
Can you please post here the link to the testbed server.
We can work on this when we talk on Friday.

Currently the logging is configured within log4j.properties we need to ensure that application output is also printed to the terminal.

@quintinali Can you move these two searchbyQuery() used by datasetDetail to somewhere else. Maybe a separate class in search package. It's bit messy here.

https://github.com/mudrod/mudrod/blob/master/core/src/main/java/esiptestbed/mudrod/driver/ESDriver.java#L249