dqsdatalabs Goto Github PK

advertools

advertools - online marketing productivity and analysis tools

aio-scrapy

将基于twisted的scrapy/scrapy-redis改成基于asyncio,使用aiohttp发送请求

almaren-framework

The Almaren Framework provides a simplified consistent minimalistic layer over Apache Spark. While still allowing you to take advantage of native Apache Spark features. You can still combine it with standard Spark code.

androidcrawler

crawler android apps (.apk file) from android markets using scrapy, support ip proxy pool

apachecn-python-zh

:books: ApacheCN Python 译文集

awesome-crawler

A collection of awesome web crawler,spider in different languages

awesome-scraping

A good start point of resources for make scraping.

awesome-scrapy

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

awesome-web-data-extractor

A curated list of promising Web Data Extractors resources

awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

awesome_apis

:octocat: A collection of APIs

burplist

Web crawlers for Burplist, a search engine or craft beers in Singapore

coursework

crawlerx

CrawlerX - Develop Extensible, Distributed, Scalable Crawler System which is a web platform that can be used to crawl URLs in different kind of protocols in a distributed way.

darkweb-search-engine

Dark Web & Deep Web Search Engine. Data Crawler and indexer for Darkweb , OSINT Tools for the Dark Web

deliciousdockerfiles

Some of prepared various dockerfiles which it will be used by any DevOps engineers in the world.

design

爬虫

design-1

docker-scrapyd

🕷️ Scrapyd is an application for deploying and running Scrapy spiders.

eqt-enrichment

examples-of-web-crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

fireprox

AWS API Gateway management tool for creating on the fly HTTP pass-through proxies for unique IP rotation

flower

Real-time monitor and web admin for Celery distributed task queue

fuli

福利信息网赚爬虫

generalscrapy

gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

gerapyautoextractor

Auto Extractor Module

glotpress-scraper

Web scraper to download all the WordPress translation files from GlotPress.org

house

Use scrapy to crawl Lianjia house price data

httpbin

HTTP Request & Response Service, written in Python + Flask.