Crawler for HUAWEI AppGallery Web pages
- hwtop_spider.py: crawl app information from https://appgallery.huawei.com/#/Top
- hwapp_spider.py: crawl app information directly from only the initial page https://appgallery.huawei.com/#/Apps
- hwapp_spider2.py: switch various categoriy pages at https://appgallery.huawei.com/#/Apps, jump into inividual app mainpages to crawl more comprehensive information
- Python
- Scrapy, Splash, Lua
- Elasticsearch