Giter Club home page Giter Club logo

python-spider's Introduction

Spider 高效爬虫实践

演示

  • 豆瓣电影 Top 250 爬取并进行结果可视化:

  • 数据分析:

    • 高分电影最多的十位导演:
    • 高分电影最多的十位主演:
    • 高分电影最多的十个国家:
    • 前十影片评分与评论数的关系:

  • 浏览器下拉行为模拟 + cookie 伪造登录, 爬取知乎动态刷新内容

功能

  • 根据爬取数据特性建 Model 聚集数据, 方便后续高效利用
  • 爬取多页面
  • 页面缓存提升效率及可靠性
  • 浏览器行为模拟爬取动态页面
  • cookie 身份验证, 爬取保护内容
  • 利用 jinja2 渲染出可视化的结果

使用方法

  • 豆瓣 250:
    • 运行 douban_visual.py
    • 在浏览器打开生成的 visualising.html
  • 知乎:
    • 补上 serect.py 于根目录, 放自己的知乎 cookie (在浏览器直接复制粘贴即可)
      • 格式为: cookie = 'k=v;k=v;...'
    • 执行

python-spider's People

Contributors

nanrayjack avatar

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.