Giter Club home page Giter Club logo

pyloom's Introduction

PyLoom，爬龙！

PyLoom想为有价值的网站编写爬虫，让开发者便捷地获取结构化的数据。

PyLoom由三个部分组成，

  1. 框架，减少编写、运行、维护爬虫的工作量。

  2. 爬虫，寻找有价值的目标为其开发爬虫，并维护既有爬虫的可用性。

    预期19年底，PyLoom将拥有围绕电子商务、房屋租售、社交网络、新闻媒体的数十个爬虫。

  3. 升级爬虫，对于频繁使用的爬虫，增强其能力

    • 增强定制能力，例如支持限定地区、类别、关键字抓取；
    • 增强抓取策略，减少对代理、打码接口的使用；
    • 增强更新策略，更细粒度地计算重复抓取的时间。

目前进度，

①部分完成，开发常见爬虫够用了，随爬虫的开发迭代出更多功能；

②已有几款爬虫，放置于spiders目录。

安装

  1. 环境要求

    • python 3.6.0+
    • redis 2.6+
    • 类unix系统

  2. 安装PyLoom

    git clone https://github.com/spencer404/PyLoom.git
python3.6 -m pip install -e ./PyLoom

    添加 -i https://pypi.douban.com/simple 参数，利用豆瓣镜像提速。

    出现错误fatal error: Python.h: No such file or directory时，

    需安装对应平台的python3.x-devel包

运行

以运行spiders/WeiBo为例，

  1. 最简参数启动爬虫

    pyloom run -s PyLoom/spiders/WeiBo

    在爬虫目录中执行run时，可省略-s参数。

  2. 启动代理池

    pyloom proxy run

  3. 添加代理

    根据命令提示，添加名为"xxx"的代理

    pyloom proxy add

  4. 使用代理启动爬虫

    pyloom run --proxy xxx

    命令run的部分常用参数：

    -l, --level    日志级别
-s, --spider   指定爬虫目录
-r, --redis    指定redis地址(URL形式)
-C, --clear    清空队列、代理数据后运行
--proxy        使用指定代理运行，逗号分隔多个代理
--damon        作为守护进程运行
-p             子进程数量
-t             每个子进程的线程数量

    在多台服务器上运行时，若参数-s、-r所指向的目标相同，即可横向扩容性能。

    默认地，PyLoom将抓到数据打印在日志中，你可以修改on_save函数自定义如何保存。

pyloom's People

Contributors

flytutu avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

flytutu alpha-ya smallning saf111r luckes-yang qiaojiao suthony hakaman188 yzw1 jooooooop herny88 fathui kekenalog edgar08 yuknight yukidengyu qzth-zxjun emaiqi pr0214 pygsec topone37 limuitech cao-ming kingking888 zerooo0 aron-long llzhi001 lxy7065809 lstarby fortheskyking nathonnot lo420 meihaomain aigeorgeli fire-zhy pimcms kyqv zofuthan dancelion ice-a panziqiang007 pengjinfu tools-env camoryang jyc099 lostloch wucheng139 flamingyouth ajunfork w312033591 fengyuwuqing tdlist cansnow123 melodyjerry guxiaoyong sumsung007 charmer21 pyer-at-gh shakeyo liganping onecupof nomenofear onehumanaha cccccpu abspider zhangyanlin2022

pyloom's Issues

PyCharm 直接运行爬龙

我想请问一下，如何在 PyCharm 直接运行爬龙呢,我直接在__main__.py运行,会报错：
ModuleNotFoundError: No module named 'main.entry'; 'main' is not a package
谢谢！

爬虫跑不起来 

C:\Users\PC>pyloom run -s G:\PyLoom\spiders\WeiBo
Traceback (most recent call last):
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\Scripts\pyloom-script.py", line 11, in <module>
    load_entry_point('pyloom', 'console_scripts', 'pyloom')()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\pkg_resources\__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\pkg_resources\__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\pkg_resources\__init__.py", line 2324, in load
    return self.resolve()
  File "C:\Users\PC\AppData\Local\Programs\Python\Python37\lib\site-packages\pkg_resources\__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "c:\users\pc\pyloom\pyloom\__init__.py", line 1, in <module>
    from .tasks import Task
  File "c:\users\pc\pyloom\pyloom\tasks.py", line 8, in <module>
    from .utils import *
  File "c:\users\pc\pyloom\pyloom\utils.py", line 9, in <module>
    import readline
ModuleNotFoundError: No module named 'readline'

请问者跑不起来

拼多多

拼多多的不行了，找不到了

环境装好了， 运行出现这个错误

Traceback (most recent call last):
File "/usr/local/bin/pyloom", line 9, in
load_entry_point('pyloom', 'console_scripts', 'pyloom')()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 542, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2569, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2229, in load
return self.resolve()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2235, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/root/PyLoom/pyloom/init.py", line 1, in
from .tasks import Task
File "/root/PyLoom/pyloom/tasks.py", line 119
raise `errors.TaskError(f"不支持从'{type(root)}'类型构造CSS")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.