Giter Club home page Giter Club logo

xianyu_spider's Introduction

xianyu_spider

闲鱼APP数据爬虫

特性 | 技术架构 | 快速开始 | 使用说明 | 免责声明 | 常见问题 | 学习交流

🚀「闲鱼APP数据爬虫」一个基于uiautomator2的闲鱼APP爬虫项目,严禁将所得数据商用! 本项目只供学习使用,作者不承担任何由此带来的法律风险和损失!如果本项目对你有用,点个Start,方便回顾知识~

👻特性

  • 支持任意关键词采集
  • 支持数据导出Excel
  • 支持自定义上滑次数

✨技术架构

  • Python3.6+
  • uiautomator2

🚨快速开始

1、环境准备

  1. clone本项目代码
  2. 在项目根目录创建Python虚拟环境venv
    1. cd xianyu_spider
    2. pip install virtualenv
    3. virtualenv venv
  3. 安装依赖库pip install -r requirements.txt
  4. 本次使用的是uiautomator2和USB真机调试,请打开Android手机USB调试模式并连接到电脑
  5. 首次通过uiautomator2连接手机时,会自动在手机上安装一个小黄车APP
  6. 打开cmd输入adb devices获取设备ID

2、运行项目

  1. 将设备ID输入xianyu.py的d = u2.connect("设备ID")
  2. 运行python xianyu.py即可进行采集,如需更改采集数据,可更改main函数keyword即可main(keyword='餐饮券')

🍔使用说明

  1. 本次使用的是uiautomator2和USB真机调试,请打开手机USB调试模式并连接到电脑
  2. 本次只采集了每个商品的【标题、价格、图片】
  3. 如果想采集其他信息,可使用weditor进行调试,自行更改相应代码,命令行输入weditor自动打开调试辅助工具 整体截图
  4. 程序运行完成后,会在当前目录下生成YYYY-MM-dd结果.xlsx

运行截图

  • 运行截图 整体截图
  • 运行结果 整体截图

👻常见问题

错误:atxagent 下次必须通过adb启动

电脑USB连接安卓设备;

1. 查看设备是否成功连接:adb devices
   出现的真实设备名称在device前即连接成功;
2. 进入手机shell环境:adb shell
3. 赋予设备权限:chmod 775 /data/local/tmp/atx-agent
4. 后台启动atx-agent: /data/local/tmp/atx-agent server -d

错误:device显示unauthorized

如果输入adb devices显示unauthorized表示usb调试模式授权有问题,按如下方式调整

1、手机里点击【撤销USB调试授权】
2、电脑上重启adb
 adb kill-server
 adb start-server
3、然后重新连接

免责声明

请勿将xianyu_spider应用到任何可能会违反法律规定和道德约束的工作中,请友善使用xianyu_spider,遵守蜘蛛协议,不要将xianyu_spider用于任何非法用途。如您选择使用xianyu_spider 即代表您遵守此协议,作者不承担任何由于您违反此协议带来任何的法律风险和损失,一切后果由您承担。

学习交流

扫码入群交流学习,如群二维码失效请加作者微信:FearlessMan888,备注:python群

xianyu_spider's People

Contributors

fearlesspeople avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

xianyu_spider's Issues

程序报错

2024-01-12 02:39:08 INFO:正在获取【餐饮券】关键字信息...
-32002 Client error: <> data: Selector [resourceId='com.taobao.idlefish:id/title'], method: None
2024-01-12 02:39:33 ERROR:程序运行异常:{'code': -32002, 'data': "Selector [resourceId='com.taobao.idlefish:id/title']", 'method': 'wait'}
我在运行您的示例时出现了这个错误,可能是什么原因导致的呢

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.