woothee / woothee-java Goto Github PK
View Code? Open in Web Editor NEWWoothee Java implementation and Hive UDF
Woothee Java implementation and Hive UDF
Woothee-java wrongly detects iOS devices as Mac OS X because of case sensitivity in the user agent parsing.
Example below is taken from a real web service getting requests from mobiles devices. Both user agents are valid user agents but only one is detected as an iOS device.
Current behaviour:
Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15G77
-> iPhoneMozilla/5.0 (iPhone; CPU IPhone OS 11_4_1 Like Mac OS X) AppleWebKit/605.1.15 (KHTML, Like Gecko) Mobile/15G77
-> Mac OS XExpected behaviour:
Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15G77
-> iPhoneMozilla/5.0 (iPhone; CPU IPhone OS 11_4_1 Like Mac OS X) AppleWebKit/605.1.15 (KHTML, Like Gecko) Mobile/15G77
-> iPhoneWe noticed that the version of user agents for Android 9 wasn't parsed properly by Woothee. This because the user agent contains Android 9
and not Android 9.0
.
For example, the version for the following user agent would be null
instead of 9
: Mozilla/5.0 (Linux; Android 9; SM-N960F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.105 Mobile Safari/537.36
We tested it with our own phones so we are sure that these are valid user agents for Android 9 devices (and not some fake user agents :) )
(A similar PR is open in the Ruby version of Woothee)
I can not get Android version when FireFox Browser.
I request UserAgent to "Mozilla/5.0 (Android 7.0; Mobile; rv:50.0) Gecko/50.0 Firefox/50.0"
Result,
{name=Firefox, category=smartphone, os=Android, version=50.0, vendor=Mozilla}
not found "os_version" parameter
missing file DataSet.java?
We find about 1.6% of incoming traffic to be identified as Internet Explorer UNKNOWN.
The UA strings lookslike these:
select p.useragent, count(*) as cnt, count(distinct user_id)
from php_logs p
join woothee_useragent_dim w
on p.useragent = w.useragent
where dt >= '20150401' and dt <= '20150430'
and name = 'Internet Explorer' and version = 'UNKNOWN'
group by p.useragent
order by cnt desc
limit 30;
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; LCJB; rv:11.0) like Gecko 12691436 13036
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MASMJS; rv:11.0) like Gecko 5470585 4994
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; ASU2JS; rv:11.0) like Gecko 5052566 4996
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MAARJS; rv:11.0) like Gecko 4481126 4056
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MALNJS; rv:11.0) like Gecko 4194491 4110
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; NP06; rv:11.0) like Gecko 3969474 4023
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; MAAU; rv:11.0) like Gecko 3438683 3266
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; MATM; rv:11.0) like Gecko 3085392 3057
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko 2188720 2821
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; LCJB; rv:11.0) like Gecko 1695819 1555
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MASEJS; rv:11.0) like Gecko 1460236 1615
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MATMJS; rv:11.0) like Gecko 1447890 1167
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MDDCJS; rv:11.0) like Gecko 963311 620
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; Touch; rv:11.0) like Gecko 927900 1599
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; ASU2JS; rv:11.0) like Gecko 772250 821
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; LCJB; rv:11.0) like Gecko 751545 855
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MAARJS; rv:11.0) like Gecko 746599 617
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; Touch; rv:11.0) like Gecko 677468 714
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko 522275 515
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MALNJS; rv:11.0) like Gecko 463573 458
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; MASMJS; rv:11.0) like Gecko 462232 571
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; ASU2JS; rv:11.0) like Gecko 457776 848
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; NP07; NP07; rv:11.0) like Gecko 385464 544
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MALCJS; rv:11.0) like Gecko 369572 293
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; MAMD; rv:11.0) like Gecko 352423 239
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; TNJB; rv:11.0) like Gecko 293674 342
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; MANM; rv:11.0) like Gecko 291762 218
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MAPBJS; rv:11.0) like Gecko 286097 234
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; MDDRJS; rv:11.0) like Gecko 245578 299
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; Touch; LCJB; rv:11.0) like Gecko 226963 277
It seems to me that the problem is that the regular expression used in
https://github.com/woothee/woothee-java/blob/5a7de46936f5f6e4b0e76620bc4b991e344ff53d/src/main/java/is/tagomor/woothee/browser/MSIE.java does not allow for tokens between "Trident/7.0;" and "rv:11.0", such as "Touch;" or "MASMJS;".
The later seem to be manufacturer's codes http://www.whatismybrowser.com/developers/unknown-user-agent-fragments
Honestly, I want to stop supporting the build option for Hive UDF. Its dependency problem in pom.xml
is huge.
I don't use it in the workload around myself, and it's not so difficult to build your own UDF build with woothee-java if you need it.
Does someone have objectives about it?
T/O
I tried out some Crawler UserAgent as described here:
https://www.keycdn.com/blog/web-crawlers/
Most of them failed to classify as Crawler, examples:
Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
I am getting this error while generating the jar using these two steps.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project woothee-java: Compilation failure
[ERROR] /home/kakn01/wootheejar-test/woothee-java-master/src/main/java/is/tagomor/woothee/DataSet.java:[43,106] reached end of file while parsing
[ERROR] -> [Help 1]
I am assuming there is another step before the above two steps: download the source code from master, then run the "mvn" command from the root directory of the extracted source. We need a pom.xml file to run the mvn command right? Please let me know if I am wrong.
Depends on #25
I cannot find Hive part of the package in the latest released jar(1.0.0). This is avoiding us from creating some hive UDFs.
This parser reasonably well in finding different browsers, bots, system os and versions.
It is able to find for Android too.
But it somehow gives ipad for ipad's ios and iphone for iphone's ios
please fix the above problem
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.