Giter Club home page Giter Club logo

ingestion3's People

Contributors

audreyaltman avatar codacy-badger avatar dominicbm avatar eslao avatar guegueng avatar katelynnsinclair avatar markbreedlove avatar mdellabitta avatar moltude avatar pickleandmustard avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

ingestion3's Issues

`sbt package`: "Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?"

Hi,

I was trying to follow the README -- but I get this error. Any idea what I'm going wrong?

sbt package
[info] Loading project definition from /Users/tingle/code/ingestion3/project
java.lang.NullPointerException
	at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1769)
	at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
	at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
	at java.base/java.util.regex.Pattern.matcher(Pattern.java:1130)
	at java.base/java.util.regex.Pattern.split(Pattern.java:1249)
	at java.base/java.util.regex.Pattern.split(Pattern.java:1322)
	at sbt.IO$.pathSplit(IO.scala:797)
	at sbt.IO$.parseClasspath(IO.scala:912)
	at sbt.compiler.CompilerArguments.extClasspath(CompilerArguments.scala:66)
	at sbt.compiler.MixedAnalyzingCompiler$.withBootclasspath(MixedAnalyzingCompiler.scala:188)
	at sbt.compiler.MixedAnalyzingCompiler$.searchClasspathAndLookup(MixedAnalyzingCompiler.scala:166)
	at sbt.compiler.MixedAnalyzingCompiler$.apply(MixedAnalyzingCompiler.scala:176)
	at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
	at sbt.Compiler$.compile(Compiler.scala:152)
	at sbt.Compiler$.compile(Compiler.scala:138)
	at sbt.Defaults$.sbt$Defaults$$compileIncrementalTaskImpl(Defaults.scala:860)
	at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:851)
	at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:849)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:844)
[error] (compile:compileIncremental) java.lang.NullPointerException
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
java -version
java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

Remove log4j

log4j can't be used in distributed mode and should be removed from project

IngestRemap - Select most recent harvest data

With the implementation of #200 and #202 the IngestRemap entry needs to be able to find and select the most recent harvest data output to provide as the source data for mapping, Currently, this invocation relies upon a fixed path and deleting previous run data. This code should look within the ./harvest/ folder and identify the most recent folder using the date time in the folder name (20180906_131309-pa-OriginalRecord.avro)

Since harvests are the only process that currently uses this timestamped folder name, the other processes do not need to search for the most recent data for their input. They will continue to use the hard-coded paths.

// Outputs
val harvestDataOut = cmdArgs.getInput()
val mapDataOut = baseDataOut+"/mapped"
val enrichDataOut = baseDataOut+"/enriched"
val jsonlDataOut = baseDataOut+"/json-l"
val baseRptOut = baseDataOut+"/reports"

executeMapping(sparkConf, harvestDataOut, mapDataOut, shortName, logger)
// Enrichment
executeEnrichment(sparkConf, mapDataOut, enrichDataOut, shortName, logger, conf)
// Json-l
executeJsonl(sparkConf, enrichDataOut, jsonlDataOut, logger)
// Reports
executeAllReports(sparkConf, enrichDataOut, baseRptOut, logger)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.