Yet another simple parser to parse and save news from Ycombinator to disk. It has following options:
| Short option | Long option | Description |
|---|---|---|
| -o | --output-dir | dir where to put parsed data. [default cwd] |
| -i | --interval | interval to check the main page in seconds [default 30] |
| -l | --log-dir | dir for logs [default ./ycrawler.log] |
| -d | --debug | turn on debug mode |
You should have installed Python 3.6 on your system.
$ git clone https://github.com/ligain/14_asyncio
$ cd 14_asyncio/
$ python3.6 -m venv .env
$ . .env/bin/activate
$ pip install -r requirements.txt
$ python crawler.py
[2018.09.15 22:27:16] run: I Started downloader worker
[2018.09.15 22:27:16] run: I Started crawling worker
[2018.09.15 22:27:16] run: I Start crawling from main page: https://news.ycombinator.com
...
$ cd 14_asyncio/
$ python3.6 tests.py
The code is written for educational purposes.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent