bill-cc / metadata-hive-hook Goto Github PK
View Code? Open in Web Editor NEWHive hook, obtain task information from Hive, fetch input/output tables and lineage information from HSQL.
License: Apache License 2.0
Hive hook, obtain task information from Hive, fetch input/output tables and lineage information from HSQL.
License: Apache License 2.0
你好,我看列级血缘部分注释掉了,请问可以支持列级血缘吗~
您好,以下是我执行的SQL:
insert overwrite table bi_tmp.t_tmp_000184_order
select concat(create_time,'123') -- order
from bi_tmp.t_tmp_000189_order
where order is not null
limit 1;
通过Hook得到的字段血缘信息如下:
"columnLineage":[{"expression":["concat(bi_tmp.t_tmp_000189_order.create_time,'123')"],"qualifiedName":"bi_tmp.t_tmp_000184_order.","inputs":[{"qualifiedName":"bi_tmp.t_tmp_000189_order.create_time"}],"name":""}]
可以看到qualifiedName的值为bi_tmp.t_tmp_000184_order.
没有拿到输出表的字段名
如果加入SQL中被注释掉的别名 order ,血缘信息中qualifiedName的值为bi_tmp.t_tmp_000184_order.order
但这并不一定是输出表中真正的物理字段名,只是SQL中的临时别名
请问这个问题之后会进行维护嘛,期待回复!
请问作者,我copy了源代码的lineagelogger到自己的工程,但是得到的edges和vertices都是空,请问您知道什么原因吗?另外我在源代码同位置copy了lineagelogger类到mylineagelogger,也是有同样的问题,将hive.exec.post.hooks配置改为hive自己的lineagelogger就可以得到完整的edges和vertices。期待回答,多谢!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.