HTML crawler broken about twitter-tools HOT 3 CLOSED

hey folks! i'm downloading the tweets2011 collection for a project for @mattlease's course at the university of texas ischool. so i'll need to figure out how to resolve this issue. i'm wondering if you guys thought it might be better to use the twitter api vs scraping the actual web page -- it seems like it'd be much more efficient, but i don't know if you guys have run into any rate-limiting issues w/ the api.

thanks!

from twitter-tools.

the smallest microblog collection is 16 million tweets, so the twitter api rate-limiting (something like 180 tweets every 15 min) would make it intractable for these collections.

as the issue implies, twitter stopped embedding the tweet's json in the html, so the only other option seems to be parsing the html. this is tricky because:

• there could be several tweets in the conversation on the page and finding the correct one is nearly impossible with simple string matching
• the contents of the tweet element have twitter's formatting applied, and the parser needs to differentiate between, say, a link to an @-reference and a link to an external site (whose text content is not the actual link)

@lintool do you think using something like jsoup to parse and select the contents would be the correct solution? i come from a javascript background, so it seems like node.js would be perfect for this, but i wouldn't want to re-implement all of the retry and redirect logic.

from twitter-tools.

Fixed by 826d693

from twitter-tools.