Giter Club home page Giter Club logo

Comments (17)

AAndyProgram avatar AAndyProgram commented on May 24, 2024 2

@lucacham @NormanMises
Try this script and tell me if any files are downloaded

  1. Create a temporary directory (e.g. "D:\\OutputFolder\\")
  2. Replace the <SCRAWLER_SETTINGS_PATH> text with the actual SCrawler settings path
  3. Replace the <URL> text with the actual URL to the user profile
  4. Replace the <PATH_TO_GDL> test with the actual path to the gallery-dl
cd /d "D:\\OutputFolder\\"
<PATH_TO_GDL>\gallery-dl --no-download --no-skip --cookies "<SCRAWLER_SETTINGS_PATH>\Responser_Twitter_Cookies_Netscape.txt" --write-pages <URL> -D "D:\\OutputFolder\\"

@Alexo88 Good to know that everything is fine.

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024 1

This script should download information in JSON files. If there is nothing there, so I think you've configured cookies incorrectly and/or something wrong with the Japanese location in GDL. Sorry. I have no issues with Twitter. Everything works for me.

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024
  1. The bug report is incomplete. Add version information (click the top right info button in the main window, then the Environment button, then the Copy button, and paste the copied text here).
  2. I don't know what and how you configured GDL through python. SCrawler doesn't need special GDL configurations.

from scrawler.

lucacham avatar lucacham commented on May 24, 2024

Thank you for your reply
Below is my version info.

SCrawler v2023.8.27.0 (x64)
OS: Microsoft Windows 11 Pro (x64)
ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
yt-dlp version: 2023.07.06
gallery-dl version: 1.25.8
cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0

I don't know what and how you configured GDL through python. SCrawler doesn't need special GDL configurations.

Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024

Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.

It is, but SCrawler handles it on its own. SCrawler doesn't use user configurations. SCrawler creates and uses its own GDL configuration. It is possible that your GDL configuration conflicts with SCrawler's GDL configuration. I think you should remove your GDL configuration and everything will be fine.

from scrawler.

lucacham avatar lucacham commented on May 24, 2024

As shown below, I have deleted the gallery-dl I configured myself and restarted my computer

20230831115501

I re-downloaded SCrawler, re-imported the cookies, and then tried to download Twitter pictures. The result was the same as before: a new Twitter account folder was created during the download. Under the folder was a Settings folder, and the settings folder contained xml. file

20230831135033
20230831135045

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024

Can't confirm. I can download this profile https://twitter.com/hajisan0525
Show me a screenshot of your environment settings.

image

from scrawler.

NormanMises avatar NormanMises commented on May 24, 2024

The same cannot be done now, but the log is as follows

2023-08-31 22:39:09: [Twitter] - tuzaibaby
	: user information saving error
	路径中具有非法字符。
2023-08-31 22:39:09: Message: 路径中具有非法字符。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
	在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:38: [Twitter] - xixi8848
	: user information saving error
	路径中具有非法字符。
2023-08-31 22:39:38: Message: 路径中具有非法字符。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
	在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:44: gallery-dl configuration file creating error
	不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
	在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
	: [Twitter] - tuzaibaby
	: GetTimelineFromGalleryDL()
	不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
	在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
	: user information saving error
	路径中具有非法字符。
2023-08-31 22:39:44: Message: 路径中具有非法字符。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
	在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:40:38: gallery-dl configuration file creating error
	不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
	在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:40:38: [Twitter] - tuzaibaby
	: [Twitter] - tuzaibaby
	: GetTimelineFromGalleryDL()
	不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
	HelpLink:
	StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
	在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
	在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)

environment

SCrawler v2023.8.6.0 (x64)
OS: Microsoft Windows 11 家庭中文版 (x64)
ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
yt-dlp version: 2023.07.06
gallery-dl version: 1.25.8
cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024

@NormanMises
Google translate:
路径中具有非法字符 - Illegal characters in path
不支持给定路径的格式 - The format of the given path is not supported

Try to specify the path in Latin characters

from scrawler.

NormanMises avatar NormanMises commented on May 24, 2024

my path is E:\SCrawler\Twitter\
image

from scrawler.

lucacham avatar lucacham commented on May 24, 2024

The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos)
I haven't changed these settings

20230901090040

In addition, the pictures below are my settings. Could you please help me see if there is anything wrong with my settings? Thank you.

Screenshots 
11


1


2


3


4


5


7


8


9


10

from scrawler.

Alexo88 avatar Alexo88 commented on May 24, 2024

I used to use a chrome plugin before but since Elon restricted its API, the pluning doesn't do anything, and I tried to download from this program but it doesn't work, I hit start download and it ends abruptly.

Uploading image.png…

from scrawler.

Alexo88 avatar Alexo88 commented on May 24, 2024

The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos) I haven't changed these settings

try downloading new version, i doit and works.

from scrawler.

mykeehu avatar mykeehu commented on May 24, 2024

Unfortunately this script didn't help me, nothing downloads :(

from scrawler.

mykeehu avatar mykeehu commented on May 24, 2024

Strange. I can't get anything from three Twitter users...

from scrawler.

NormanMises avatar NormanMises commented on May 24, 2024

Unable to download, even if a proxy is used

D:\Apps\SCrawler\Environment\gallery-dl --no-download --no-skip --cookies "D:\Apps\SCrawler\Settings\Responser_Twitter_Cookies_Netscape.txt" --write-pages https://twitter.com/kkkl6587 -D "E:\SCrawler\test\" --proxy "http://127.0.0.1:12345"
[twitter][error] HttpError: SSLError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: /i/api/graphql/XA6F1nJELYg65hxOC2Ekmg/UserByScreenName?variables=%7B%22screen_name%22%3A%22kkkl6587%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

from scrawler.

AAndyProgram avatar AAndyProgram commented on May 24, 2024

Have you tested without a proxy? No luck?

from scrawler.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.