opennaslab / kubespider Goto Github PK

我非常喜欢这个项目,目前有没有用户交流群,很想参与进来,啊啊啊啊啊啊

Describe the background

Currently, the download trigger is only executed once, which may not be optimal. It would be better to attempt it at least three times.

Describe the feature

Currently, the download trigger is only executed once, which may not be optimal. It would be better to attempt it at least three times.

20 18:15:22,705-INFO: Get webhook trigger:magnet:?xt=urn:btih:4ba1e93b9f4680e2e91df56d7e4a3b4292de5fcc&dn=[%E7%94%B5%E5%BD%B1%E5%A4%A9%E5%A0%82www.dytt89.com]%E8%9A%81%E4%BA%BA%E4%B8%8E%E9%BB%84%E8%9C%82%E5%A5%B3%EF%BC%9A%E9%87%8F%E5%AD%90%E7%8B%82%E6%BD%AE-2023_HDTC%E4%B8%AD%E8%8B%B1%E5%8F%8C%E5%AD%97.mp4&tr=http://t.t789.me:2710/announce&tr=http://t.t789.co:2710/announce&tr=http://t.t789.vip:2710/announce
20 18:15:22,705-INFO: Download mangent file
20 18:15:22,706-INFO: Start magnet download:magnet:?xt=urn:btih:4ba1e93b9f4680e2e91df56d7e4a3b4292de5fcc&dn=[%E7%94%B5%E5%BD%B1%E5%A4%A9%E5%A0%82www.dytt89.com]%E8%9A%81%E4%BA%BA%E4%B8%8E%E9%BB%84%E8%9C%82%E5%A5%B3%EF%BC%9A%E9%87%8F%E5%AD%90%E7%8B%82%E6%BD%AE-2023_HDTC%E4%B8%AD%E8%8B%B1%E5%8F%8C%E5%AD%97.mp4&tr=http://t.t789.me:2710/announce&tr=http://t.t789.co:2710/announce&tr=http://t.t789.vip:2710/announce
20 18:15:24,513-WARNING: Please ensure your motrix server is ok:Unauthorized

用的迅雷3.4.0

试了几次，姿势不对啊

不支持Docker直接部署吗？

蜜柑订阅有些繁体中文简体还有720和1080 混在一起 只想下载需要的

Xxx

目前通过开发者手动适配资源难免会存在资源不够多不够广的问题，如果能适配rss订阅源，那么就可以获得广泛的资源支持。例如群晖的download station或者asustor的download center一样，可以让用户自主添加想要的rss订阅，并且可以设定资源刷新时间自主下载。

通过修改download_provider.yaml

aria2:
  type: aria2_download_provider
  enable: false
  download_base_path: "/downloads/"
  rpc_endpoint_host: http://127.0.0.1
  rpc_endpoint_port: 6800
  secret: kubespider
  priority: 3
qbittorrent:
  type: qbittorrent_download_provider
  enable: true
  download_base_path: "/downloads/"
  http_endpoint_host: http://127.0.0.1
  http_endpoint_port: 8080
  username: admin
  password: adminadmin
  verify_webui_certificate: false
  priority: 0
  tags:
    - kubespider

但是自己测试，还是不行，手动添加任务到qbittorrent可以。
然后访问
http://192.168.0.111:3080/api/v1/downloadproviders

回复：

{
    "aria2": true,
    "qbittorrent": false,
    "transmission": false,
    "xunlei": false,
    "youget": false,
    "yt-dlp": false
}

我目前自己部署了几个qb和aria2容器，用来分别下载不同类别的资源，如qb-1下载pt站的，qb-2下载mikan的动画资源。

请问现在是否支持这样的指定下载器功能？

aria2 bt下载不如qbittorent好用

配置怎么改 , 运行脚本如下提示:

Unable to find image 'cesign/aria2-pro:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
[root@localhost kubespider]# ls

Describe the background

The magic source provider has significant potential for development, and therefore we should allocate a space to store the configuration of every adapted website.

Describe the feature

Where we should put this demo configuration? cc @qingchoulove

Describe the background

Send a notification to the user when a task is successfully added, or when a download is accepted from rss

Describe the feature

• Easy to extend more notification platforms
• It is possible to configure for each provider
• Stable and easy-to-use api
• More features from the community

/Users/lingjiatong/code/kubespider/venv/bin/python3.10 /Users/lingjiatong/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/231.9011.38/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 49590 --file /Users/lingjiatong/code/kubespider/kubespider/app.py 
Connected to pydev debugger (build 231.9011.38)
2023-06-21 14:44:10,385-INFO: Config files(source_provider.yaml,download_provider.yaml,pt_provider.yaml,kubespider.yaml) miss, try to init them
Traceback (most recent call last):
  File "/Users/lingjiatong/code/kubespider/kubespider/core/config_handler.py", line 158, in prepare_config
    shutil.copy(template_cfg, target_cfg)
  File "/usr/local/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 417, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lingjiatong/.config_template/source_provider.yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lingjiatong/code/kubespider/kubespider/core/config_handler.py", line 160, in prepare_config
    raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err
TypeError: decoding str is not supported

for cfg in miss_cfg:
        template_cfg = values.CFG_TEMPLATE_PATH + cfg
        target_cfg = values.CFG_BASE_PATH + cfg
        try:
            shutil.copy(template_cfg, target_cfg)
        except Exception as err:
            raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err

应该是这里shutil.copy报的路径不存在问题

Describe the background

Support automatic downloading of updated videos by bloggers of Bilibili.

Describe the feature

Support automatic downloading of updated videos by bloggers of Bilibili.

填域名能收到数据包。日志提示

2023-04-25 15:48:22,127-ERROR: 125.70.202.219 - - [25/Apr/2023 15:48:22] code 400, message Bad request version ('ºº\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x93\x1a\x1a\x00\x00\x00\x00\x00\x14\x00\x12\x00\x00\x0f公网域名替换显示\x00\x17\x00\x00ÿ\x01\x00\x01\x00\x00')
2023-04-25 15:48:22,127-INFO: 125.70.202.219 - - [25/Apr/2023 15:48:22] "üq&<lhÂäüø¸û[øQdøÄiægýkÁ}! £VWns@ÏÜA4V:$õe໡wH8tÙ
y0
ººÀ+À/À,À0̨̩ÀÀ公网域名替换显示ÿ" HTTPStatus.BAD_REQUEST -
2023-04-25 15:48:22,142-ERROR: 125.70.202.219 - - [25/Apr/2023 15:48:22] code 400, message Bad request version ('JJ\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x93úú\x00\x00\x00\x00\x00\x14\x00\x12\x00\x00\x0f公网域名替换显示\x00\x17\x00\x00ÿ\x01\x00\x01\x00\x00')
2023-04-25 15:48:22,143-INFO: 125.70.202.219 - - [25/Apr/2023 15:48:22] "üïÐsM#D`v/ûÆEB8ù Z=Ä_»ìîH AíçOͬûWæá
vÞöÓç3"¯^ JJÀ+À/À,À0̨̩ÀÀúú公网域名替换显示ÿ" HTTPStatus.BAD_REQUEST -

填IP地址的话倒是正常，公网IP，和内网IP都是正常的
023-04-25 15:51:38,820-INFO: 10.253.0.4 - - [25/Apr/2023 15:51:38] "GET /healthz HTTP/1.1" 200 -

浏览器插件的方式依赖人肉浏览页面

是否可以提供api方式供爬虫直接调用

Ask your question here:

环境信息：在宿主机上挂载了一个 NFS 文件夹，其目录为 /root/kubespider/nas，然后安装该工具，登录aira 的容器，可以正常创建文件和读取已有文件，担心和这个 NFS 挂载有关所以特意说明
下载时提示错误，容器输出的日志信息：
05/06 23:08:19 [NOTICE] Serialized session to '/config/aria2.session' successfully.
[#d8fb97 56KiB/56KiB(100%) CN:5 SD:1]
05/06 23:08:20 [NOTICE] Download complete: [MEMORY][METADATA]梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [NOTICE] Saved metadata as /downloads/57c7d3f11ac1c98cc9ef52a3ccf5091e14b97854.torrent.
05/06 23:08:20 [NOTICE] Serialized session to '/config/aria2.session' successfully.
05/06 23:08:20 [ERROR] CUID#86 - Exception caught while allocating file space.
Exception: [AbstractDiskWriter.cc:557] errNum=95 errorCode=17 fallocate failed. cause: Operation not supported
05/06 23:08:20 [ERROR] CUID#86 - Download not complete: /downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [NOTICE] Download GID#6d32f38fc9378fd2 not complete: /downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [WARRING] Number of files is zero, maybe a Magnet Link.
05/06 23:08:20 [INFO] Download task error, deleting files...
removed '/downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4.aria2'
removed '/downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4'
05/06 23:08:20 [INFO] Deleting .torrent file ...
removed '/downloads/57c7d3f11ac1c98cc9ef52a3ccf5091e14b97854.torrent'
05/06 23:08:20 [INFO] Deleting empty directory ...

用的就是示例中的资源

请问我应该如何排查，谢谢。

Ask your question here:

RT

我想尝试改善下二开体验，开发时支持代码热更新，结果发现当前指定的 python版本是3.1
而 watchdog 只支持，2.6，2.7 以及 3.4 以上的版本,原因似乎是 3.1 collections 模块中不存在的 MutableSet 属性。
我找了个 downloaders 的子项目，直接升到 3.8 去似乎没有问题

demo

直接 docker-compose up 可以启动 /app 内的文件改变会热更新

系统Unraid, docker-compose.yml参照教程有做修改

services:
  kubespider:
    image: cesign/kubespider:latest
    depends_on:
      - aria2
    environment:
      - PUID=1000
      - PGID=100
      - TZ=Asia/Shanghai
    ports:
      - 3080:3080
    volumes:
      - /mnt/user/Files/Config/kubespider/config:/app/.config
    networks:
      - kb
    
  aria2:
    container_name: aria2-qb
    image: abcminiuser/docker-aria2-with-webui:latest-ng
    network_mode: "bridge"
    ports:
      - 6800:6800
      - 6880:80
    volumes:
      - /mnt/user/Files/Download/kubespider:/data
      - /mnt/user/Files/Config/kubespider/aria2:/conf
    environment:
      - PGID=100
      - PUID=1000
    restart: unless-stopped

networks:
  kb:
    name: kb

浏览器插件Aria2 explorer已经安装，能够正常连接

kubespider插件已加载并配置api地址

在测试网址https://www.meijutt.tv/content/meiju28582.html 右键触发下载后，aria2没有显示正在下载的任务
docker logs kubespider日志如下：

2023-07-06 21:44:37,432-INFO: meijutt tv link is:https://www.meijutt.tv/content/meiju28582.html
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:96EC4CB3707382B9945D...
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:9B762A8C71AEFE413DC3...
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:3F4F1EFB8104B557B66F...
2023-07-06 21:44:38,295-INFO: Find new resource:meijutt_source_provider/magnet:?xt=urn:btih:96EC4CB3707382B9945D...
2023-07-06 21:44:38,299-INFO: filtering downloader by name None type None, result ['aria2']
2023-07-06 21:44:38,302-INFO: download link type magnet, with provider size 1
2023-07-06 21:44:38,302-INFO: Download mangent file with downloader(aria2)
2023-07-06 21:44:38,306-INFO: Start magnet download:magnet:?xt=urn:btih:96EC4CB3707382B9945DA60439DD0B1F19A187D3
2023-07-06 21:44:38,309-WARNING: Please ensure your aria2 server is ok:HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jsonrpc (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1496c7a8c430>: Failed to establish a new connection: [Errno 111] Connection refused'))
2023-07-06 21:44:38,309-WARNING: Download torrent file error:HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jsonrpc (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1496c7a8c430>: Failed to establish a new connection: [Errno 111] Connection refused'))

Telegram

Invite link: https://t.me/+RR09tPuCvI9iNTc1

Describe the background

Currently, the notification is unfriendly. Let's improve it.

Describe the feature

Currently, the notification is unfriendly. Let's improve it.

Describe the background

Now after the configuration is changed, users need to restart the kubespider container, and it's a redundant step.

Describe the feature

• Support configuration reload automatically after changed, and do not need to restart the kubespider container.

小白一个，参考docker-compose折腾了一下，但是没有调试成功。请问有大佬出一个Unraid的配置教程么

Describe the background

To make this project more international, we should provide an English version. This will encourage the participation of foreign developers.

Describe the feature

doc: Provide the english version of all the docs

Hey, @DDSDerek ! Mind sharing your experience with using the main.sh script to help us enhance the installation section of the README document? We want to make it super easy for more people to get the hang of Kubespider. thx!

Describe the background

Automatically download the video pushed by telegram channel.

Describe the feature

Automatically download the video pushed by telegram channel.

/assign @DDSRem

使用的是docker run 收到安装，aira2使用的是以前就有的不是脚本自带的
配置如下:
aria2:
type: aria2_download_provider
enable: true
download_base_path: "/downloads/"
rpc_endpoint_host: http://ip
rpc_endpoint_port: '6800'
secret: pass
priority: 1

Ask your question here:

在文件 source_provider/meijutt_source_provider/provider.py 第69行，筛选下载链接时，筛选器为 tabs-list current-tab

通过检查美剧天堂html文件，存在多套下载链接是，class并不会随着切换替换 current-tab class，而是修改 style = display: block;

这部分是不是不符合预期的？

Ask your question here:

2023-06-01 15:56:35,618-INFO: Get xunlei token:xxxxx
2023-06-01 15:56:37,818-ERROR: Create tasks error:{"error":"task_daily_create_limit","error_code":11,"error_description":"您今日的免费使用次数已达上限","error_details":[{"@type":"type.googleapis.com/google.rpc.DebugInfo","detail":"LIMIT:3","stack_entries":[]}],"HttpStatus":403}
2023-06-01 15:56:37,819-WARNING: Download torrent file error:Create task error

现在在pr合入之后，上传docker的workflow基本都是失败的：actions，并且每次都在dockerhub上覆盖旧的镜像tag，较难随时更新版本使用，考虑优化一下上传的workflow？

@miRemid 有兴趣来搞搞这个吗

可否下载m3u8地址视频，需要批量下载那种

containerd as the container runtime, but the container detection method used in #225 is not working anymore.

FileNotFoundError: [Errno 2] No such file or directory: '/.config/source_provider.yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/kubespider/app.py", line 8, in <module>
    main()
  File "/app/kubespider/app.py", line 5, in main
    runner.run_with_config_handler()
  File "/app/kubespider/core/runner.py", line 34, in run_with_config_handler
    config_handler.prepare_config()
  File "/app/kubespider/core/config_handler.py", line 177, in prepare_config
    raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err
TypeError: decoding str is not supported

我这执行脚本会一直闪屏，麻烦大佬帮忙看下是什么问题呢？

@DDSDerek

mac系统和QNAP都有这个问题。

请教一下，通过浏览器插件发送到配置里的美剧可以正常添加到aria2中并下载操作，但直接将配置文件中的mikanani启动并替换为自己的rss后，再重启容器，aria2无法接受到任何mikanani的下载项，还有有日志文件吗？如果发送异常了怎么排错？

Ask your question here:

大家讨论下配置项：

1. 代理
2. 周期时间
3. 服务端口
4. 目前我看有人在公网用，设置一下密码认证？

是否还有其他的？
@qingchoulove @xiaozhikang0916 @miRemid @ijwstl

Describe the background

feat: download YouTube channel video automatically

Describe the feature

feat: download YouTube channel video automatically

Describe the background

现有模式，资源站（source_provider）依赖开发人员为每一个网站适配，成本较高。
是否有模式可以通过配置项支持有相同模式的资源站。

Describe the feature

根据source_provider.provider.SourceProvider定义，需要实现should_handle和get_links两个关键方法。
阅读MeijuttSourceProvider代码，get_links其实是通过BeautifulSoup进行解析。
综上，是否可以通过BeautifulSoup的select支持的能力，通过用户配置xpath或css selector来支持不同资源站。
例如：

mjtt

xpath: /html/body/div[2]/div[6]/div/div/div[3]/div[1]/ul/li
css selector: body > div.warp > div.o_cn2 > div > div > div.tabs-list.current-tab > div.down_list > ul > li > input

dygod

xpath: //*[@id="downlist"]/table/tbody/tr/td/a
css selector: #downlist > table > tbody > tr > td > a

表达式确认

可以通过chrome devtool进行预选则，然后根据实际情况进行调整。

附录

css selector
xpath

source provider里面没看到有b站

Ask your question here:

请问kubespider能否自定义迅雷下载的文件夹？目前是下载到common文件夹下。

Ask your question here:

google浏览器设置server保存时失败

从README的过程和个人使用情况来看，只能通过浏览器插件来发送URL形式的文件链接或种子链接给后端，如何发送文件形式的种子文件呢？

Ask your question here:

#173

kubespider/core/config_handler.py line: 143

The config_path is missing when determining kubespider.yaml file exists

Ask your question here:

官方群晖流程报错如下

docker: Error response from daemon: Bind mount failed: '/volume2/kubespider/.config' does not exists.

环境：群晖 920
流程：参考官方的群晖部署流程，进入 bash install 环节就一直报这个错误，容器倒是都安装好了，可是报错后就没法看到 aria2 的链接密钥。
尝试：

• chmod 777 给了 kubespider 共享文件夹和 bash 文件，不行
• kubespider 的共享文件夹赋予 everyone 权限，也还是不行

Describe the background

支持下载收藏视频(B站)

Describe the feature

支持下载收藏视频(B站)
@acegao9527 有兴趣来实现一下吗？

Ask your question here:

2023-06-20 22:59:42,941-ERROR: Exception on /api/v1/download [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/app/kubespider/core/webhook_server.py", line 22, in decorated
    return func(*args, **kwargs)
  File "/app/kubespider/core/webhook_server.py", line 92, in download_handler
    err = download_trigger.kubespider_downloader.download_file(source, path, link_type)
  File "/app/kubespider/core/download_trigger.py", line 88, in download_file
    return self.handle_magnet_download(url, path, downloader_list, extra_param)
  File "/app/kubespider/core/download_trigger.py", line 129, in handle_magnet_download
    provider.load_config()
  File "/app/kubespider/download_provider/xunlei_download_provider/provider.py", line 73, in load_config
    with open(token_js_path, 'r', encoding='utf-8') as js_file:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/dependencies/xunlei_download_provider/get_token.js'

感觉是创建 container 的时候没有映射对路径？ 我是把 get_token.js 复制到了映射的目录里，然后改了一下配置就可以下载了