agoragames / bnet_scraper Goto Github PK
View Code? Open in Web Editor NEWA Nokogiri-based scraper of Battle.net profiles. Currently this only includes Starcraft2.
License: MIT License
A Nokogiri-based scraper of Battle.net profiles. Currently this only includes Starcraft2.
License: MIT License
Hello there again,
please would you kindly check and update all examples on the first main page ?
Some of them are not working.
Thanks a lot!
Cheers
The grandmaster page for each realm is accessible and chock full of information for scraping. The API I'm imagining looks something like an array of Grandmasters that contain the points, wins, rank, and a Profile object with the basic information filled in, but not fully scraped.
The hydra portrait in row 4, column 6 of this block (http://media.blizzard.com/sc2/portraits/4-90.jpg) is reported as "Level 11 Locust."
Hello there,
seems to be a nice tool you are doing.
I am trying to use it.
It seems that date format are not handled well
It have errors (see details below) for achievments dates:
When 'us' region page format is : mm/dd/yyyy
(Works well here: http://us.battle.net/sc2/en/profile/2377239/1/Demon/achievements/)
In several others region/locale : dd/mm/yyyy
(Fails here, even if local put to en:
http://eu.battle.net/sc2/en/profile/1229243/1/Stephano/achievements/)
Its more like a bnet website issue... (since 'en' locale is set but not handled)
but still as such one cannot scrap achievements properly.
Thanks a lot!
Cheers
C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scraper/starcraf
t2/achievement.rb:20:in new': invalid date (ArgumentError) from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:20:in
convert_date'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement.rb:15:in earned=' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:10:in
block in initialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement.rb:9:in each_key' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:9:in
initialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:78:in new' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:78:in
extract_recent_achievement'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:65:in block in scrape_recent' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:64:in
times'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:64:in scrape_recent' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:41:in
scrape'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/profile.rb:18:in achievements' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/profile.rb:22:in
recent_achievements'
from C:/Users/xxx/Desktop/a.rb:17:in `
The API for Profile is very erratic. The scraper holds the data, but makes no attempt to structure the data beyond a basic hash with concentration of data. The API needs to be refactored to consider a Profile, League, etc.
Thoughts on improved API:
ProfileScraper returns a Profile object. Profile has:
swarm_levels
achievements
leagues
match_history
Calls to these, unless already retrieved, will scrape the relevant page(s). In addition, give ProfileScraper an option to eager-scrape these subsections. This removes the need to have BnetScraper::Starcraft2.full_scrape
and provides a better use-case for the library.
I'm getting two sets of values returned for this profile URL (http://us.battle.net/sc2/en/profile/273698/1/Omni/) for career games and games this season.
Set 1 (correct): {career_games: '205', games_this_season: '41'}
Set 2 (incorrect): {career_games: '226', games_this_season: '47'}
This is interesting to me because it's always from these two sets; there's no further variation. I haven't seen this happen in a single-threaded model, and it suddenly appeared once I went multi-thread, so I think that might be the likely culprit. Maybe I'm wrong, though.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.