agoragames / bnet_scraper Goto Github PK

Hello there again,
please would you kindly check and update all examples on the first main page ?
Some of them are not working.

Thanks a lot!
Cheers

The grandmaster page for each realm is accessible and chock full of information for scraping. The API I'm imagining looks something like an array of Grandmasters that contain the points, wins, rank, and a Profile object with the basic information filled in, but not fully scraped.

The hydra portrait in row 4, column 6 of this block (http://media.blizzard.com/sc2/portraits/4-90.jpg) is reported as "Level 11 Locust."

Hello there,
seems to be a nice tool you are doing.
I am trying to use it.

It seems that date format are not handled well
It have errors (see details below) for achievments dates:

When 'us' region page format is : mm/dd/yyyy
(Works well here: http://us.battle.net/sc2/en/profile/2377239/1/Demon/achievements/)

In several others region/locale : dd/mm/yyyy
(Fails here, even if local put to en:
http://eu.battle.net/sc2/en/profile/1229243/1/Stephano/achievements/)

Its more like a bnet website issue... (since 'en' locale is set but not handled)
but still as such one cannot scrap achievements properly.

Thanks a lot!
Cheers

C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scraper/starcraf
t2/achievement.rb:20:in new': invalid date (ArgumentError) from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:20:inconvert_date'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement.rb:15:in earned=' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:10:inblock in initialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement.rb:9:in each_key' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement.rb:9:ininitialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:78:in new' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:78:inextract_recent_achievement'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:65:in block in scrape_recent' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:64:intimes'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/achievement_scraper.rb:64:in scrape_recent' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/achievement_scraper.rb:41:inscrape'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr
aper/starcraft2/profile.rb:18:in achievements' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bnet_scraper-0.6.0/lib/bnet_scr aper/starcraft2/profile.rb:22:inrecent_achievements'
from C:/Users/xxx/Desktop/a.rb:17:in `

The API for Profile is very erratic. The scraper holds the data, but makes no attempt to structure the data beyond a basic hash with concentration of data. The API needs to be refactored to consider a Profile, League, etc.

Thoughts on improved API:

ProfileScraper returns a Profile object. Profile has:

• swarm_levels
• achievements
• leagues
• match_history

Calls to these, unless already retrieved, will scrape the relevant page(s). In addition, give ProfileScraper an option to eager-scrape these subsections. This removes the need to have BnetScraper::Starcraft2.full_scrape and provides a better use-case for the library.

I'm getting two sets of values returned for this profile URL (http://us.battle.net/sc2/en/profile/273698/1/Omni/) for career games and games this season.

Set 1 (correct): {career_games: '205', games_this_season: '41'}
Set 2 (incorrect): {career_games: '226', games_this_season: '47'}

This is interesting to me because it's always from these two sets; there's no further variation. I haven't seen this happen in a single-threaded model, and it suddenly appeared once I went multi-thread, so I think that might be the likely culprit. Maybe I'm wrong, though.

Code: http://pastebin.com/Zjx2iVFi