Using futures requests instead of raw to speed data loading about rpscrape HOT 12 CLOSED

Have implemented this, seems to be working as expected so far.

from rpscrape.

The site limits requests per minute, there are no gains to be made there, vast majority of running time is waiting for requests to be allowed. I think writing iteratively is also preferable given the precarious nature of web scraping with edge cases and connection, you dont want to risk losing hours of work in memory.

from rpscrape.

Yeah I'm less sure on the iterative vs one-off writing being an improvement for those reasons, however I'm also not sure whether an incomplete data set is worth much more than none at all if there's a break!

However on async vs sync, loading 2019-2020 GB jumps seems to take ~840s sequentially vs <200s when I use asynchronous requests calls, with (as far as I can tell) the same output. It's possible (likely?) I'm missing something, but there definitely appears to be an improvement.

from rpscrape.

Interesting, its possible they have removed the limit or changed it as it has been a few years since I checked. Do you have a fork with your changes I can test?

from rpscrape.

Ran a little test there for multiple minutes and got no 403 responses which suggests the limiting has been removed and async will dramatically improve speed as you have seen. Ill start working on it just now, or you can make a pull request if youve already done it.

from rpscrape.

Awesome, glad you could replicate. Sorry was about to push my version then noticed my auto formatter had gone a little rogue with the quotation mark formatting. Can fix up and make a PR tomorrow if you haven’t done already by then.

from rpscrape.

Awesome, and more properly async than I did too! Thanks

from rpscrape.

Many thanks for the update.

I can fetch yesterday's results fine.

But should I be getting this noise awell?:

$ cd rpscrape/scripts/

$ python rpscrape.py -d 2021/04/19

Exception ignored in: <function _ProactorBasePipeTransport.del at 0x0000026D175A7E50>
Traceback (most recent call last):
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 116, in del
self.close()
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 719, in call_soon
self._check_closed()
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Finished scraping. 2021_04_19.csv saved in rpscrape/data//all
Exception ignored in: <function _ProactorBasePipeTransport.del at 0x0000026D175A7E50>
Traceback (most recent call last):
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 116, in del
self.close()
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 108, in close
self._loop.call_soon(self._call_connection_lost, None)
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 719, in call_soon
self._check_closed()
File "C:\Users\garry\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

...

from rpscrape.

No you shouldnt be getting that, seems to be a windows specific issue, will have a look.

from rpscrape.

Many thanks!

from rpscrape.

I have tested on windows and runs fine with no errors, can you confirm?

from rpscrape.

Confirmed working now. Thanks again!

from rpscrape.