Comments (10)
So far I have solved this issue by simply instantiating the analyzer on the node that does the actual execution. So this is what you see in the Pig UDF.
I'll have a look if I can make it serializable. No promises at this moment.
from yauaa.
Thanks for answering, since we're using Spark SQL and UDFs that would make it "once by execution" which is pretty bad in terms of executions. How can I help ?
from yauaa.
btw thx your work is the best we've been trying so far
from yauaa.
we may be blocked by antlr/antlr4#233
from yauaa.
The Parse Tree does not need to be serializable. I'll have a look on how I can solve this.
from yauaa.
I did some experimentation and getting this to work right would be 'really hard'.
Background:
Parsing the config files may yield parse errors (so you want to do that early) and the resulting data structures need the parser system to validate it all. Then 'only' serializing what NEEDS to be serialized becomes tricky.
from yauaa.
I found a way to do this that seems to work.
I test it by running a significant part of the tests with a UserAgentAnalyzer that was serialized and then deserialized: https://github.com/nielsbasjes/yauaa/blob/master/analyzer/src/test/java/nl/basjes/parse/useragent/TestSerialization.java
Can you please verify if this also works in your context?
from yauaa.
Question: You say "best so far". What makes this library for you the best?
from yauaa.
@ogirardot Have you been able to check if it works now?
from yauaa.
These changes are in the 1.4 release.
So for now I'm closing this as 'fixed'.
Please reopen if you find any issues.
from yauaa.
Related Issues (20)
- Abstract logging with SLF4j HOT 2
- Incorrect Yahoo crawler HOT 2
- Release a new version with updated dependencies HOT 2
- Fix maven-artifact-plugin 3.5.0 project.build.outputTimestamp error. HOT 3
- Normalize.replaceString is vulnerable to out of denial of service (DoS) attack HOT 1
- AntPathMatcher.match is vulnerable to out of denial of service (DoS) attack HOT 1
- TODO: Remove Antlr4 - Java 21 workaround HOT 2
- Reasons for Windows NT on mobile devices HOT 1
- In k8s+istio environment, parsing does not respond HOT 2
- 系统版本号解析异常 HOT 1
- why not DeviceName is not V2311 HOT 2
- Wrong typification for ZTE T1002 HOT 7
- Errors When Loading Files HOT 4
- Supermium HOT 1
- Frozen SamsungBrowser HOT 9
- Fix long header value in webapp HOT 1
- Errors When Loading Files. HOT 2
- Detecting email clients from user-agent HOT 7
- Fix reproducibility HOT 1
- Browsers to investigate
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yauaa.