jamedge / moonlight Goto Github PK
View Code? Open in Web Editor NEWData lineage REST API and its client libraries for ETL jobs.
License: Apache License 2.0
Data lineage REST API and its client libraries for ETL jobs.
License: Apache License 2.0
Add a table of contents so that navigating through md report is easier.
Endpoint returning information saying which lines are processed for the selected processing framework would be useful.
If the line doesn't exist, response with empty HTML table is still returned. This shouldn't happen - no HTML should be returned for this case, or maybe just a caption.
Add API endpoint and client function that save the HTML/Markdown report to Notion.
Create an endpoint which would return HTML page containing a comprehensive representation of the lineage data, formatted in a way making the following present:
NOTES:
Remove rows from resulting line tables if they have no values.
Add endpoints for retrieving all information from IO Elements. Preferrably in json, md and html.
Create endpoints which would return JSON and HTML representations of all line elements saved in the database.
Add tests for testing correct and bad requests for getting line and lineage information as json, md and html where applicable.
Lineage endpoint is not working correctly. Error with the containing request is below. Add the test for this functionality as part of fixing this issue.
Stack trace:
2020-07-11 23:48:18,161 [scala-execution-context-global-146] ERROR com.github.jamedge.moonlight.core.api.Module - Unable to process request. Response message: There was an internal server error. Check API logs for details.
Additional details:
Source message: NULL is not iterable
Source stacktrace:
org.neo4j.driver.internal.value.ValueAdapter.values(ValueAdapter.java:401)
org.neo4j.driver.internal.value.ValueAdapter.values(ValueAdapter.java:395)
neotypes.implicits.mappers.ValueMappers$$anon$2.to(ValueMappers.scala:120)
neotypes.implicits.mappers.ResultMappers$$anon$3.to(ResultMappers.scala:144)
neotypes.implicits.mappers.ResultMappers$$anon$3.$anonfun$to$8(ResultMappers.scala:145)
scala.util.Either.flatMap(Either.scala:352)
neotypes.implicits.mappers.ResultMappers$$anon$3.to(ResultMappers.scala:145)
neotypes.implicits.mappers.ResultMappers$$anon$1.to(ResultMappers.scala:98)
neotypes.Transaction$.$anonfun$collectAsImpl$4(Transaction.scala:119)
neotypes.internal.utils.traverse$.loop$1(traverse.scala:14)
neotypes.internal.utils.traverse$.traverseAs(traverse.scala:20)
neotypes.internal.utils.traverse$.traverseAsList(traverse.scala:25)
neotypes.Transaction$.$anonfun$list$2(Transaction.scala:38)
neotypes.Transaction$.$anonfun$collectAsImpl$3(Transaction.scala:118)
neotypes.internal.syntax.stage$CompletionStageOps$.$anonfun$acceptImpl$1(stage.scala:15)
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
org.neo4j.driver.internal.handlers.PullAllResponseHandler.completeFailureFuture(PullAllResponseHandler.java:326)
org.neo4j.driver.internal.handlers.PullAllResponseHandler.onSuccess(PullAllResponseHandler.java:88)
org.neo4j.driver.internal.async.inbound.InboundMessageDispatcher.handleSuccessMessage(InboundMessageDispatcher.java:82)
org.neo4j.driver.internal.messaging.v1.MessageReaderV1.unpackSuccessMessage(MessageReaderV1.java:75)
org.neo4j.driver.internal.messaging.v1.MessageReaderV1.read(MessageReaderV1.java:56)
org.neo4j.driver.internal.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:83)
org.neo4j.driver.internal.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:35)
org.neo4j.driver.internal.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
org.neo4j.driver.internal.async.inbound.MessageDecoder.channelRead(MessageDecoder.java:47)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1389)
org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1203)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945)
org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146)
org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:745)
Request: HttpRequest(HttpMethod(GET),http://localhost:8080/lineage/graph/md?root_io=user_data,List(Timeout-Access: <function1>, Accept: application/moonlight.v1+json, */*;q=0.0, User-Agent: PostmanRuntime/7.26.1, Postman-Token: 0d8d2bde-97e9-43da-8d12-22621a6bae0b, Host: localhost:8080, Accept-Encoding: gzip, deflate, br, Connection: keep-alive),HttpEntity.Strict(application/moonlight.v1+json,0 bytes total),HttpProtocol(HTTP/1.1))
Add a helm chart containing all components for easier k8s deployment. Add all crucial configuration params as env vars.
Add a docker-compose file to ease the setup of the environment and the service. Configure the necessary envs as well.
Set up CI/CD pipeline to pack the API to docker container and deploy to docker hub.
Current library for querying data from neo4j (neotypes
) requires producing a lot of boilerplate and/or complex DSL-like logic to not repeat the same queries for different objects multiple times.
The goal of this issue is to replace this tedious part with some library which could provide these functionalities out of the box.
So far, the only library I found that provides this is this one, but doesn't seem actively maintained.
Another possibility is to explore neo4j java connector and what could be done by using it.
Having this endpoint will enable user to get all information about storage. Preferred in json, md and html.
Implement, version and deploy/distribute a simple scala library which will make the usage of the functionality this project provide even easier for developers. Ideally, it would be something like annotation or a call which could make adding (and/or extracting lineage information from context) as easy as possible.
To understand the data sources (IOElements) better, the concept of column or some smaller chunk should be thought through as well.
Notes:
Add marshaller that enables returning of line data as markdown.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.