fr75s / bigscraperqt Goto Github PK

View Code? Open in Web Editor NEW

7.0 3.0 1.0 1.79 MB

Easy to use GUI Scraping tool; supports LaunchBox and more

Home Page: https://fr75s.github.io/bigscraperqt

License: GNU General Public License v3.0

QML 33.74% Python 66.24% Shell 0.02%

bigscraperqt's Introduction

bigscraper-qt

VISIT THE WEBSITE HERE

GUI Scraping tool

Bigscraper-qt is a tool you can use to scrape the LaunchBox Games Database and more. With a simple GUI, you can easily scrape metadata for a variety of systems.

If you want more information, head to the website. This README mainly serves as information regarding the development of bigscraper-qt.

Need help? For quick support or light feature suggestions, Visit the Discord Server. If it is a problem of a big enough scope, you may put it into the issue tracker..

Running from Source

Ensure you have the following dependencies installed:

Python 3
Qt5
The following python packages:
- PyQt5
- Unidecode
- requests
- inputs
- yt_dlp
- lxml
- xdg

Then, download the source code.

$ git clone https://github.com/Fr75s/bigscraperqt.git
$ cd bigscraperqt

After you download the source code, simply run the following. To see available flags, append -h.

$ python3 -m bsqt

Building from Source

To build from source, first clone the repository.

$ git clone https://github.com/Fr75s/bigscraperqt.git
$ cd bigscraperqt

Then, simply run one of the following, depending on which version to build:

Appimage:
$ appimage-builder --skip-tests

Flatpak:
$ flatpak-builder build io.github.fr75s.bigscraper-qt.json --force-clean [...]

Planned Features

A list of new planned features are below.

Flathub Release
LaunchBox Asynchronous Scraping
Clearing Cached data by system
UI Refresh

bigscraperqt's People

Contributors

Stargazers

Watchers

Forkers

sebio1980

bigscraperqt's Issues

Suggestion: Arcade Database as source

I made the suggestion earlier in the Pegasus discord as I'd forgotten there was a github for this project. :)

For Arcade, it would be very nice to support the database here: http://adb.arcadeitalia.net/

Skyscraper, which is included with Retropie as optional, supports this source (and others) using priority settings. Its codebase is essentially frozen, but would make a good resource for this project with a GUI. Having LB as a source with the others already supported (like ScreenScraper.fr) would be amazing

Suggestion: change folder result

Hello,
in the interface, I select the location of my games for a system.
Example: "/recalbox/share/roms/snes"
But the scrape result doesn't go in there, it goes in "/recalbox/share/system/.local/share/bigscraper-qt/"
It would be better for systems to have the following tree as a result.
Example:
"/recalbox/share/roms/snes/media/video/f-zero.mp4"
"/recalbox/share/roms/snes/media/screenshot/f-zero.png"
"/recalbox/share/roms/snes/media/fanart/f-zero.png"
"/recalbox/share/roms/snes/media/box3d/f-zero.png"
"/recalbox/share/roms/snes/media/box2d/f-zero.png"
etc...
"/recalbox/share/roms/snes/media/video/Mario kart.mp4"
"/recalbox/share/roms/snes/media/screenshot/Mario kart.png"
"/recalbox/share/roms/snes/media/fanart/Mario kart.png"
"/recalbox/share/roms/snes/media/box3d/Mario kart.png"
"/recalbox/share/roms/snes/media/box2d/Mario kart.png"
etc...
Thank you for your help.

suggestions: translate in other language

Hello,
do you have the possibility to share the file with the text to translate it?
Thank you

Issue: screenscraper error after 5 games

Hello,
i have this issue when i scrap sega32x

# /usr/bin/bigscraper-qt-1.4.1-x86_64.AppImage -d
[I] Program Init... 2022-10-25 14:36:02
[I] Starting application
[I] bigscraper-qt v1.4.1. Made by Fr75s. Licensed under GPLv3
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
[I] Reading options file
[D] Saved Options to File
[I] QML Loaded
[U] Selected Folder (file:///recalbox/share/roms/sega32x)
[U] Selected System (Sega 32X)
-----
[I] Begin Task: Scrape Folder
-----
[I] Data is ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[I] Will write to /recalbox/share/system/.local/share/bigscraper-qt/
[I] Scraping from ScreenScraper
[O] Starting...
[I] Data formatted to [['AFTER_BURNER_COMPLETE', 'AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE', 'BC_RACERS', 'BLACKTHORNE', 'BRUTAL_ABOVE_THE_CLAW', 'COSMIC_CARNAGE', 'DARXIDE', 'DOOM', 'FIFA_SOCCER_96', 'GOLF_MAGAZINE_PRESENTS_36_GREAT_HOLES_STARRING_FRED_COUPLES', 'KNUCKLES_CHAOTIX', 'KOLIBRI', 'METAL_HEAD', 'MORTAL_KOMBAT_II', 'MOTHERBASE', 'MOTOCROSS_CHAMPIONSHIP', 'NBA_JAM_TOURNAMENT_EDITION', 'NFL_QUARTERBACK_CLUB', 'PITFALL_THE_MAYAN_ADVENTURE', 'PRIMAL_RAGE', 'RBI_BASEBALL_95', 'SANGOKUSHI_IV', 'SPACE_HARRIER', 'STAR_TREK_STARFLEET_ACADEMY_STARSHIP_BRIDGE_SIMULATOR', 'STAR_WARS_ARCADE', 'STELLAR_ASSAULT', 'T_MEK', 'TEMPO', 'TIME_WARNER_32X_CMD_DOWNLOAD_CARTRIDGE', 'TOUGHMAN_CONTEST', 'VIRTUA_FIGHTER', 'VIRTUA_RACING_DELUXE', 'WORLD_SERIES_BASEBALL_STARRING_DEION_SANDERS', 'WWF_RAW', 'WWF_WRESTLEMANIA_THE_ARCADE_GAME'], ['/recalbox/share/roms/sega32x/After Burner Complete (Europe).zip', '/recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip', '/recalbox/share/roms/sega32x/BC Racers (USA).zip', '/recalbox/share/roms/sega32x/Blackthorne (USA).zip', '/recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip', '/recalbox/share/roms/sega32x/Cosmic Carnage (Europe).zip', '/recalbox/share/roms/sega32x/Darxide (Europe) (En,Fr,De,Es).zip', '/recalbox/share/roms/sega32x/Doom (Europe).zip', '/recalbox/share/roms/sega32x/FIFA Soccer 96 (Europe) (En,Fr,De,Es,It,Sv).zip', '/recalbox/share/roms/sega32x/Golf Magazine Presents - 36 Great Holes Starring Fred Couples (Europe).zip', "/recalbox/share/roms/sega32x/Knuckles' Chaotix (Europe).zip", '/recalbox/share/roms/sega32x/Kolibri (USA, Europe).zip', '/recalbox/share/roms/sega32x/Metal Head (Europe) (En,Ja).zip', '/recalbox/share/roms/sega32x/Mortal Kombat II (Europe).zip', '/recalbox/share/roms/sega32x/Motherbase (Europe).zip', '/recalbox/share/roms/sega32x/Motocross Championship (Europe).zip', '/recalbox/share/roms/sega32x/NBA Jam - Tournament Edition (World).zip', '/recalbox/share/roms/sega32x/NFL Quarterback Club (World).zip', '/recalbox/share/roms/sega32x/Pitfall - The Mayan Adventure (USA).zip', '/recalbox/share/roms/sega32x/Primal Rage (USA, Europe).zip', "/recalbox/share/roms/sega32x/RBI Baseball '95 (USA).zip", '/recalbox/share/roms/sega32x/Sangokushi IV (Japan).zip', '/recalbox/share/roms/sega32x/Space Harrier (Europe).zip', '/recalbox/share/roms/sega32x/Star Trek - Starfleet Academy - Starship Bridge Simulator (USA).zip', '/recalbox/share/roms/sega32x/Star Wars Arcade (Europe).zip', '/recalbox/share/roms/sega32x/Stellar Assault (USA, Europe).zip', '/recalbox/share/roms/sega32x/T-MEK (USA, Europe).zip', '/recalbox/share/roms/sega32x/Tempo (Japan, USA).zip', '/recalbox/share/roms/sega32x/Time Warner 32X CMD Download Cartridge (USA).zip', '/recalbox/share/roms/sega32x/Toughman Contest (USA, Europe).zip', '/recalbox/share/roms/sega32x/Virtua Fighter (Europe).zip', '/recalbox/share/roms/sega32x/Virtua Racing Deluxe (Europe).zip', '/recalbox/share/roms/sega32x/World Series Baseball Starring Deion Sanders (USA).zip', '/recalbox/share/roms/sega32x/WWF Raw (World).zip', '/recalbox/share/roms/sega32x/WWF WrestleMania - The Arcade Game (USA).zip'], 'sega32x', 19]
[I] Getting User Page
[O] Showing Main Bar
[O] Hiding Game Bar
[O] Getting User Page
[D] Page Request Successful
[D] ScreenScraper Requests As of this line: 19
[D] Maximum alotted requests: 300000
[D] Your Level: 11 (will be helpful for me to mimic your thread count)
[D] Max number of threads: 7
[D] Threadpool Maximum Capacity: 7
[I] Initializing Thread for AFTER_BURNER_COMPLETE
[D] GAME FILE: /recalbox/share/roms/sega32x/After Burner Complete (Europe).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['AFTER_BURNER_COMPLETE', '/recalbox/share/roms/sega32x/After Burner Complete (Europe).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: 0a8cbd7835e53bfc0ac2363b2d9abb7b
[D] SHA1 Hash: 06d3034ef60b032e755ec0e3e715a50906d07e7e
[D] CRC32 Hash: 708147714, CRC32 Hex-Formatted Hash: 2a357a02
[I] Attempting to get page for AFTER_BURNER_COMPLETE
[D] (AFTER_BURNER_COMPLETE) Page Request Successful
[I] (AFTER_BURNER_COMPLETE) Collecting Game Metadata
[D] Checking for Game Name
[D] Adding Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Game Name: ['After Burner Complete']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for After Burner Complete
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Initializing Thread for AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE
[D] GAME FILE: /recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE', '/recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: e03077aef49dd67445d96d76ad6e457c
[D] SHA1 Hash: fc63b1cb18af47959541bd3a6157467790ca00bb
[D] CRC32 Hash: 471177555, CRC32 Hex-Formatted Hash: 1c159953
[I] Attempting to get page for AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] (AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE) Page Request Successful
[I] (AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Adding Game Name
[D] Checking for Game Release
[D] Game Name: ['Spider-man - Web Of Fire']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (2)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Spider-man - Web Of Fire
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Initializing Thread for BC_RACERS
[D] GAME FILE: /recalbox/share/roms/sega32x/BC Racers (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BC_RACERS', '/recalbox/share/roms/sega32x/BC Racers (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: cef7e91bedaba3e87e02da09cd8246d3
[D] SHA1 Hash: ad6089e370c887e11428259b57447529f50c95b7
[D] CRC32 Hash: 378450490, CRC32 Hex-Formatted Hash: 168eb23a
[I] Attempting to get page for BC_RACERS
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] (BC_RACERS) Page Request Successful
[I] (BC_RACERS) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Checking for Game Name
[D] Checking for Game Name
[D] Adding Game Name
[D] Game Name: ['Bc Racers']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Bc Racers
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Initializing Thread for BLACKTHORNE
[D] GAME FILE: /recalbox/share/roms/sega32x/Blackthorne (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BLACKTHORNE', '/recalbox/share/roms/sega32x/Blackthorne (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: 16511f98621ff43adbca331985a99b43
[D] SHA1 Hash: 4f25dda03075f544ef4ad6034225decd619696df
[D] CRC32 Hash: 1531738562, CRC32 Hex-Formatted Hash: 5b4c79c2
[I] Attempting to get page for BLACKTHORNE
[D] Download Successful, Writing to file
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (2 / 40) (for Bc Racers)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Initializing Thread for BRUTAL_ABOVE_THE_CLAW
[D] GAME FILE: /recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BRUTAL_ABOVE_THE_CLAW', '/recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: b9cfb502d47caf92f49db67cf5330162
[D] SHA1 Hash: 088dee9cd777b2951431810f975a6c69c46dec53
[D] CRC32 Hash: 2859814636, CRC32 Hex-Formatted Hash: aa754eec
[I] Attempting to get page for BRUTAL_ABOVE_THE_CLAW
[D] Download Successful, Writing to file
[D] (BLACKTHORNE) Page Request Successful
[I] (BLACKTHORNE) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Checking for Game Name
[D] Checking for Game Name
[D] Adding Game Name
[D] Game Name: ['Blackthorne']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Blackthorne
[I] Downloading Media (3 / 48) (for Blackthorne)
[I] Downloading Media (3 / 48) (for Blackthorne)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (2 / 40) (for Bc Racers)
[D] (BRUTAL_ABOVE_THE_CLAW) Page Request Successful
Traceback (most recent call last):
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/bsqt/scrapemany.py", line 804, in run
    page_content = page.json()
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] Erreur : Rom/Iso/Dossier non trouv?e !  : 0
Aborted

Suggestion: hide password screenscraper

Hello,

In your software for account screenscraper, it would be good to hide the password in the options and log part.

Thank you

Error with screenscraper folder

Hello,
OS Version :
[pixL-Beta28] Linux OS Debian Ubuntu 18.04

Machine Configuration
CPU : I7-6820
GPU : GeForce GTX 1080 mobile
System Storage device : USB box SSD M.2

when i launch scrap i have this issue :

[I] Program Init... 2022-10-15 18:01:07
[I] Starting application
[I] bigscraper-qt v1.4.0. Made by Fr75s. Licensed under GPLv3
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
[I] Reading options file
[D] Saved Options to File
[I] QML Loaded
[U] Selected Folder (file:///recalbox/share/roms/snes)
[U] Selected System (Super Nintendo Entertainment System)
-----
[I] Begin Task: Scrape Folder
-----
[I] Data is ['file:///recalbox/share/roms/snes', 'Super Nintendo Entertainment System']
[I] Will write to /recalbox/share/system/.local/share/bigscraper-qt/
[I] Scraping from ScreenScraper
[O] Starting...
[I] Data formatted to [['MICRO_MACHINES', 'MEGA_MAN_7'], ['/recalbox/share/roms/snes/Micro Machines (Europe).zip', '/recalbox/share/roms/snes/Mega Man 7 (Europe).zip'], 'snes', 4]
[I] Getting User Page
[O] Showing Main Bar
[O] Hiding Game Bar
[O] Getting User Page
[D] Page Request Successful
[D] ScreenScraper Requests As of this line: 10
[D] Maximum alotted requests: 300000
[D] Max number of threads: 7
[D] Threadpool Maximum Capacity: 7
[I] Initializing Thread for MICRO_MACHINES
[D] GAME FILE: /recalbox/share/roms/snes/Micro Machines (Europe).zip
[D] SYSTEM INFO: 4, snes, Super Nintendo Entertainment System
[D] Thread Start
[D] ['file:///recalbox/share/roms/snes', 'Super Nintendo Entertainment System']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'screenScraperUser': 'xxxxxxx', 'screenScraperPass': 'xxxxxxx'}
[D] ['MICRO_MACHINES', '/recalbox/share/roms/snes/Micro Machines (Europe).zip', 'https://www.screenscraper.fr/api2/jeuInfos.php?devid=Fr75s&devpassword=YlxdDSIEUgv&softname=bigscraperqt&output=json&ssid=xxxxxx&sspassword=xxxxxx', [4, 'snes', 'Super Nintendo Entertainment System']]
[D] MD5 Hash: 17aee32a91e4555d5aa3c75863247db8
[D] SHA1 Hash: f8a29e5113d2142d6562229e5d2f5915287ca824
[D] CRC32 Hash: 1906809994, CRC32 Hex-Formatted Hash: 71a79c8a
[I] Attempting to get page for MICRO_MACHINES
[I] Initializing Thread for MEGA_MAN_7
[D] GAME FILE: /recalbox/share/roms/snes/Mega Man 7 (Europe).zip
[D] SYSTEM INFO: 4, snes, Super Nintendo Entertainment System
[D] Thread Start
[D] ['file:///recalbox/share/roms/snes', 'Super Nintendo Entertainment System']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'screenScraperUser': 'xxxxxxx', 'screenScraperPass': 'xxxxxxxx'}
[D] ['MEGA_MAN_7', '/recalbox/share/roms/snes/Mega Man 7 (Europe).zip', 'https://www.screenscraper.fr/api2/jeuInfos.php?devid=Fr75s&devpassword=YlxdDSIEUgv&softname=bigscraperqt&output=json&ssid=xxxxxx&sspassword=xxxxxxx', [4, 'snes', 'Super Nintendo Entertainment System']]
[D] MD5 Hash: 99709a16eea3374f456d9ba7eef36ae4
[D] SHA1 Hash: 6e2e13fd072c39605d0fcddf4cca88b435c987ba
[D] CRC32 Hash: 817841714, CRC32 Hex-Formatted Hash: 30bf4632
[I] Attempting to get page for MEGA_MAN_7
[D] (MICRO_MACHINES) Page Request Successful
[I] (MICRO_MACHINES) Collecting Game Metadata
[D] Checking for Game Name
[D] Adding Game Name
[D] Checking for Game Release
[D] Checking for Game Release
[D] Checking for Game Release
[D] Checking for Game Release
Traceback (most recent call last):
  File "/tmp/.mount_bigscrVxKUg5/usr/lib/python3.10/site-packages/bsqt/scrapemany.py", line 856, in run
    meta["Release Date"] = [calendar_month_rev[rd_raw[1]] + " " + rd_raw[2] + ", " + rd_raw[0]]
IndexError: list index out of range
Aborted

Thank you for help

[feature request] Scrape lutris games

Hi, thanks for this amazing tool you made !

I was wondering if it was a plan to scrape games that are imported in lutris ? This differs from PC games as it allows to tweak launching options in lutris while still launching them from pegasus (or any other frontend).

Issue: Export

Hello,

i have this issue when i export the scrap for my game.

[O] Hiding Game Bar
[O] Downloading Video
[D] Download Successful, Writing to file
[D] Writing Metadata to File
[O] Scraping Complete. Exiting...
[O] Exiting...
[W] Input Error: No gamepad found.
[W] Input Error: No gamepad found.
[U] Selected Folder (file:///recalbox/share/roms/n64)
[W] Input Error: No gamepad found.
[W] Input Error: No gamepad found.
[U] Selected System (Nintendo 64)
[U] Selected Export (EmulationStation)
-----
[I] Begin Task: Export
-----
[I] Data is ['file:///recalbox/share/roms/n64', 'Nintendo 64', 'EmulationStation']
[I] Will write to /recalbox/share/roms/n64
[O] Starting...
[I] Format: EmulationStation
[D] Reading Metadata File
[O] Exporting For EmulationStation...
[O] Showing Main Bar
[D] Copying Mario Party - Box - Front (Europe).png to folder
Traceback (most recent call last):
  File "/tmp/.mount_bigscrc2C9P0/usr/lib/python3.10/site-packages/bsqt/bsqt.py", line 79, in run
    self.t.run()
  File "/tmp/.mount_bigscrc2C9P0/usr/lib/python3.10/site-packages/bsqt/export.py", line 433, in run
    shutil.copyfile(os.path.join(paths["MEDIA"], system, game_form) + "/" + img + ".png", os.path.join(out_folder, "media", "boxFront") + "/" + img + ".png")
  File "/tmp/.mount_bigscrc2C9P0/usr/lib/python3.10/shutil.py", line 256, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/recalbox/share/roms/n64/media/boxFront/Mario Party - Box - Front (Europe).png'
Aborted

Thank you for help

Suggestion : Keep launch command

If a system has already been scraped and a launch command has been added to the metadata file. New games are added and rescraped. When exporting the metadata file, the launch command is removed in the new file. It would be great to keep it in the new metadata file !

Suggestions: add other system

Hello,

In your software i don't see :

MAME
ATOMISWAVE
SEGA MODEL1
SEGA MODEL2
SEGA MODEL3
SEGA CHIHIRO
FBNEO
NEOGEO

Thank you for add this.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.