Comments (1)
Hey Miguel,
Great work, I think this could be very useful for many people.
I have a question:
"Unit test" for me implies that this is part of a CI suite. As a dev, I make a change to ETL code and before it gets merged, my changes are tested on data using hooqu. But would it not make sense to use this for runtime checks as well? Maybe that's the intent, then it didn't become clear to me.
Yeah it says "unit test" for data to say that is a on the data (quality). Now realistically when working with large amounts of data is impossible to test on the whole data (Deequ, the project this is based on, runs on Spark and targeted toward large datasets). So the idea is to add a QA step as part of your ETL, data processing pipeline, not as part of your unit tests (though nothing would prevent you from doing so).
I could imagine this being used like this:
verification_suite = VerificationSuite().add_check( Check(CheckLevel.ERROR, "Basic Check") .has_size(lambda sz: sz == 5) # we expect 5 rows .is_complete("id") # should never be None/Null .is_complete("productName") # should never be None/Null .has_mean("numViews", lambda mean: mean <= 10) ) @verification_suite.check_input(lambda df, *args, **kwargs: df) def my_fun(df, foo, bar=123): df = ... return dfThe idea would be that at runtime, when
my_fun
is called, the verification suite is performed on the input df (same idea could apply to the output df). Through theCheckLevel
, you could control if this should raise an error or just produce an error log, for example. I know this would need a bit of redesign of the API, since at the moment, it seems thatVerificationSuite
needs reference to the data to be tested viaadd_data
(but I think this isn't necessary and could prove problematic down the line).
You could potentially run the tests like this, I honestly don't like so much decorators for data pipelines. I would rather wrap this into its own step (say a Luigi or Airflow task) and do that at the beginning of the pipeline and/or somewhere in between. However if someone wants to use Hooqu using decorators I think I/we could extend it to make it possible 😉 .
This way, it's less of a "unit test" and more of a runtime test for data. It would not only catch errors that stem from changes in the code, but also from changes in the data. Again, maybe that's the intent, then you could make it more explicit in the README.
Thanks for the suggestion. I will do so! 😃
Some minor comments:
* I saw widespread use of lambdas in your code, be careful to store references to them, unless you think it will never make sense to pickle the objects.
You are right, for now I don't see a need to pickle but I will keep that in mind.
* typo `dupliucatees` in README
It will be fixed!
from hooqu.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hooqu.