evertonstz / pynps Goto Github PK

Hello.

I've compiled pkg2zip v2.3 myself by running sudo make on the directory with the source files downloaded from the aforementioned release. 25 files were generated, out of which I believe only pkg2zip is needed. Is that correct? I moved pkg2zip by itself and tested it on a small .pkg I had on my computer, and it worked fine. I transferred pkg2zip by itself to /home/$USER/.config/pyNPS/lib/ and everything seems to be working fine, but could anyone tell me if the other 24 files are in fact required?

Finally, as stated in the title, the pre-compiled binary is not downloadable anymore, as the file seems to be removed from the hosting service. I don't think it's a problem, it's just as a kind of heads up. I'm still sort of a beginner on Linux, so it's still useful to me to compare your binary pkg2zip with mine to know if

1. I did it correctly
2. The other files are needed.

Thanks.

title

Hey! I know this repo hasn't seen any updates for a while, but I was thinking that pyNPS should download the work.bin too, or a text-file with the zRIF string. It might not be necessary on a hacked PSV, but it's needed on Vita3k in order to install updates.

Duplication steps:
pynps -G -c psp -r usa "talkman travel tokyo"

[SEARCH] you're going to download the following files:

1. PSP | NPUG80155 | USA | Game | TALKMAN TRAVEL TOKYO [17.72 MiB]
   Download files? [y/n]: y

[DOWNLOAD] TALKMAN TRAVEL TOKYO (US) [NPUG80155] for PSP
w2kCvtC25dQdGqSeGgl1oJel2ywCHKxvecsJJTIje4Pc6AMGmMJdIeyxX2pgY7d97pnjrYwVKUkGBj1 100%[====================================================================================================================================================================================================>] 17.72M 43.8MB/s in 0.4s

Traceback (most recent call last):
File "d:\python3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "d:\python3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "D:\Python3\Scripts\pynps.exe_main.py", line 9, in
File "d:\python3\lib\site-packages\pynps\cli\cli.py", line 561, in cli_main
sha256_dl = checksum_file(downloaded_file_loc)
File "d:\python3\lib\site-packages\pynps\functions\functions.py", line 532, in checksum_file
with open(file, 'rb') as f:
OSError: [Errno 22] Invalid argument: 'G:/Games/pynps/PKG/PSP/GAMES/w2kCvtC25dQdGqSeGgl1oJel2ywCHKxvecsJJTIje4Pc6AMGmMJdIeyxX2pgY7d97pnjrYwVKUkGBj1Aca5vWHX9PeD7Sc099Dx7e.pkg?product=0084&country=us'

'?' in the URL gets turned into '@', but pynps is still looking for the file with '?'.

This can be fixed by adding
dl_dile_loc = dl_dile_loc.replace("?", "@") in cli.py, after
dl_dile_loc = f"{DLFOLDER}/PKG/{i['System']}/{i['Type']}/{i['PKG direct link'].split('/')[-1]}" as well as replacing

open(file, 'rb') as f:
with
open(file.replace("?", "@"), 'rb') as f:

in cli.py.

Traceback (most recent call last):
File "site-packages/PyInstaller/loader/rthooks/pyi_rth__tkinter.py", line 30, in
FileNotFoundError: Tcl data directory "/tmp/_MEIw55QPa/tcl" not found.
[27062] Failed to execute script pyi_rth__tkinter

Requested.

I just love pynps so badly! Really great work!!!
By using this for a day now and tinkering a bit I thought I didn't want to clutter the other PS3 Enhancement request thread too much. So I made this new wishlist here instead!

These are the main 2 things I would love to see.

1. Automatic download of all found matches, without writing what to download or pressing y.
2. Multithread download (This would be soo sweet to use on a 1gbit network)

Duplication steps:
run
C:\Program Files (x86)>pynps -D -zip -c psv -r jap _all

Enter the number for what you want to download, you can enter multiple numbers using commas: 2205-2206

App will try to download the 2205th entry:

[SEARCH] you're going to download the following files:
2205) PSV | PCSG01116 | JAP | DLC | 探索者たちの口伝集 [ 100 KiB]
2206) PSV | PCSG01116 | JAP | DLC | 追加クエスト『経験値どっさりクエスト』 [ 100 KiB]
Download files? [y/n]: y

[DOWNLOAD] 探索者たちの口伝集 (JP) [PCSG01116] for PSV
JP0106-PCSG01116_00-ADVSYSTEM0000001_bg_1_f5f2e3ae4bdc667bf29f992d933ad6c24e3ce 100%

[CHECKSUM] checksum not matching, pkg file is probably corrupted, delete it at your download folder and redownload the pkg
[CHECKSUM] corrupted file location: G:/Games/pynps/PKG/PSV/DLCS/JP0106-PCSG01116_00-ADVSYSTEM0000001_bg_1_f5f2e3ae4bdc667bf29f992d933ad6c24e3cedd4.pkg

Done! <- app stops, doesn't try to download 2206

This is caused by break in the cli.py file:
if sha256_dl != sha256_exp:
loc = f"{DLFOLDER}/PKG/{i['System']}/{i['Type']}/{i['PKG direct link'].split('/')[-1]}"
printft(HTML("[CHECKSUM] checksum not matching, pkg file is probably corrupted, delete it at your download folder and redownload the pkg"))
printft(HTML("[CHECKSUM] corrupted file location: %s") %loc)
break # skip file

Without that break, app fails gracefully and continues processing.

Hi.

Apparently, pyNPS defaults to ~/Downloads/pyNPS as the download location. I read the documentation and found nothing about being able to change the default directory, so is it possible to implement this feature? pyNPS is great, keep up the good work!

Thanks.

BertrandHustle has a nice way of extracting psx game names straight from eboot files.
Source: https://github.com/BertrandHustle/psx_scraper/blob/master/psx_scraper.py

TODO:

• need to test how trustworthy the code can be on detecting the name in different eboots;
• check if the function works for psp eboots, if not, search for a implementation that does;

NPS publishes a set of PS3 tsv files as well. Do you plan to add support for those? I say this as a layman with Python, but it looks like it wouldn't be a difficult addition. I may try to add it myself this week, unless you're already in the middle of something similar.

Thought: does pkg2zip support PS3 pkgs? EDIT: it appears not to.

The downloader is not checking if the game already exists in the final PS3 pkg folder or Extracted folder.
Which means it downloads the game again and extracts it again.

If I'm not mistaken, when the file already exists in the extracted folder it means the download was successful right?
Probably need to input a check if the folder/file exists in the final destination.
Then have the downloader skip the unnecessary download and extra extraction.

I get the following error for 3) PS3 | NPUB30314 | USA | Game | Dead Space Extraction . The rap file never appears where it's supposed to.

me@mypc:~/Downloads/pyNPS/PS3# pynps -G -c ps3 "Dead Space Extraction"

[PKG2ZIP] you don't have a valid pkg2zip installation or binary in your system, extraction will be skipped. PS3 games don't need pkg2zip
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[SEARCH] here are the matches:
1) PS3 | NPEB00318 | EUR | Game | Dead Space Extraction (Trial to Full)                                                                           [3.447 GiB]

2) PS3 | NPUB30314 | USA | Game | Dead Space Extraction                                                                                           [3.464 GiB]

3) PS3 | NPUB30314 | USA | Game | Dead Space Extraction                                                                                           [3.464 GiB]

Enter the number for what you want to download, you can enter multiple numbers using commas: 3
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[SEARCH] you're going to download the following files:
3) PS3 | NPUB30314 | USA | Game | Dead Space Extraction                                                                                           [3.464 GiB]

Download files? [y/n]: y
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[DOWNLOAD] Dead Space Extraction (US) [NPUB30314] for PS3
zZyBxuyRDPGzGCrzuxCqEUFvplMyLZsMphAGCpl 100%[============================================================================>]   3.46G  95.1MB/s    in 39s
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[CHECKSUM] No checksum provided by NPS, skipping check
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[EXTRACTION] skipping extraction since pkg2zip doesn't work with pkg2zip
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[MOVE] PS3 pkg file moved to: /home/username12345/Downloads/pyNPS/PS3/GAMES/packages/Dead Space Extraction (US) - UP0006-NPUB30314_00-DSEXTRACTIONKEY1.pkg
[MOVE] attempting to delete .pkg file from old location
[MOVE] success, the compressed .pkg was deleted
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[RAP] downloaing RAP file
4e5044000000000300000003000000005550303 100%[============================================================================>]     352  --.-KB/s    in 0s
Traceback (most recent call last):
  File "/usr/local/bin/pynps", line 33, in <module>
    sys.exit(load_entry_point('pynps==1.6.2', 'console_scripts', 'pynps')())
  File "/usr/local/lib/python3.6/dist-packages/pynps-1.6.2-py3.6.egg/pynps/cli/cli.py", line 727, in cli_main
    get_rap(i, WGET, rap_folder, rap_url)
  File "/usr/local/lib/python3.6/dist-packages/pynps-1.6.2-py3.6.egg/pynps/functions/functions.py", line 277, in get_rap
    copyfile(f"{dl_tmp_folder}{i['RAP']}", rap_folder)
  File "/usr/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
OSError: [Errno 36] File name too long: '/tmp/tmp5r3rymwv/4e5044000000000300000003000000005550303030362d4e50554233303331345f30302d445345585452414354494f4e4b455931000000000000000000000000476c6f72794f664272756e6573747564dadac5ce3234df54702a451a970faa39af0bf3c53b1ac16e5286d3457872eb1e000000000000000000000000000000000000000c0000400000000000000000408598216d2c40548905889dd0e44ae5125fe6242c087ce197ada23c2d85ad2e710000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d4beebd71f9c53a30699e98e969e45e7dd32a9679cffc126e47db2ecbc3e454f1698d0d0cd7ba63504fe17a7ced68fd0558fbe366a116de61355a79a0fb786825eb10a135fbeef61731fb212bc9ff4b1454441544120332e332e302e57000000'

Ubuntu 18.04.6 LTS
Linux 5.4.0-1058-gcp
pynps version 1.6.1

Can be done with a -nc/--noconfirm flag, initially requested at #15

Traceback (most recent call last):
  File "pynps.py", line 1071, in <module>
    main()
  File "pynps.py", line 782, in main
    maybe_download = search_db(system, what_to_dl, args.search, reg, DBFOLDER)
  File "pynps.py", line 357, in search_db
    system_database = database[system]
  File "C:\Users\macfa\Documents\GitHub\pynps\pynps\lib\site-packages\sqlitedict.py", line 245, in __getitem__
    raise KeyError(key)
KeyError: 'PSV'

To reproduce: delete database file, update theme database for PSP, run "pynps aa"

pynps -G -zip -c psp -r eur "Extreme Pinball"

[DOWNLOAD] Extreme Pinball (EU) [NPEF00068] for PSP
3sE4F8Tg3kOFfsobRrF1VMUf7hAfYDeDX62mguUM5WEb83auLcpwXcEQQNWP4Jrjfu52tydMworI7oW 100%
[CHECKSUM] No checksum provided by NPS, skipping check
[PKG2ZIP] Attempting to extract [NPEF00068]Extreme Pinball
[] loading...
[] unpacking PSX
Traceback (most recent call last):
File "d:\python3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "d:\python3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "D:\Python3\Scripts\pynps.exe_main.py", line 9, in
File "d:\python3\lib\site-packages\pynps\cli\cli.py", line 667, in cli_main
delete = run_pkg2zip(dl_dile_loc, dl_location, PKG2ZIP, pkg2zip_args, extraction_folder, i)
File "d:\python3\lib\site-packages\pynps\functions\functions.py", line 694, in run_pkg2zip
process = runner(run_lst, cwd=output_location)
File "d:\python3\lib\site-packages\pynps\functions\functions.py", line 643, in runner
out = line.rstrip().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 29: invalid start byte

Replacing out = line.rstrip().decode() with out = line.rstrip().decode('cp1252') in functions.py appears to correct this (though it breaks Japanese title downloading).

I am donwloading a PS3 game directly into a external USB drive and I am having this error. Any ideas?

--------------------------------------------------------------------------------------------------------------------------
[DOWNLOAD] Batman Arkham Origins (EU) [NPEB01414] for PS3
OmyVmsLhSkcJEZlLcSDGIwwawjOIbV 100%[==================================================>]  14,08G  11,2MB/s    in 27m 39s 
--------------------------------------------------------------------------------------------------------------------------
[CHECKSUM] No checksum provided by NPS, skipping check
--------------------------------------------------------------------------------------------------------------------------
[EXTRACTION] skipping extraction since pkg2zip doesn't work with pkg2zip
--------------------------------------------------------------------------------------------------------------------------
[MOVE] PS3 pkg file moved to: /media/victor/PS3external/pyNPS/PS3/GAMES/packages/Batman Arkham Origins (EU) - EP1018-NPEB01414_00-BATMANORIGINS001.pkg
[MOVE] attempting to delete .pkg file from old location
[MOVE] success, the compressed .pkg was deleted
--------------------------------------------------------------------------------------------------------------------------
[RAP] downloaing RAP file
E94D3BCCFF0A5A60A8E4E4F1D6B049 100%[==================================================>]      16  --.-KB/s    in 0s      
Traceback (most recent call last):
  File "/home/victor/.local/bin/pynps", line 11, in <module>
    sys.exit(cli_main())
  File "/home/victor/.local/lib/python3.7/site-packages/pynps/cli/cli.py", line 715, in cli_main
    get_rap(i, WGET, rap_folder, rap_url)
  File "/home/victor/.local/lib/python3.7/site-packages/pynps/functions/functions.py", line 262, in get_rap
    os.rename(f"{dl_tmp_folder}{i['RAP']}", rap_folder)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpjqo89k69/E94D3BCCFF0A5A60A8E4E4F1D6B0499D' -> '/media/victor/PS3external/pyNPS/PS3/GAMES/exdata/EP1018-NPEB01414_00-BATMANORIGINS001.rap'

Hi, I am having the followig error when executing any pynps command:

Traceback (most recent call last):
  File "/usr/local/bin/pynps", line 7, in <module>
    from pynps import main
  File "/usr/local/lib/python3.5/dist-packages/pynps.py", line 145
    print(f"Processing {type}: {progress_bar( int(index_file/(len(file) - 1) * 100) )}", 
                                                                                      ^
SyntaxError: invalid syntax

I installed it via pip3 in the Linux installation that is part of ChromeOS. I am in an arm architecture if that may help.

Thanks!

Tittle

Hello
I think it would be great if the app would skip download if the .zip file (created with the -zip option) exists and is valid.
This would make keeping the content up to date with new entries.

I noticed that there is an error with the following function when downloading psv themes. I was able to fix this by creating a temp file in my theme folder. Please note I tried to fix the code myself so there may be a +/- difference in the lines in the errors.

Traceback (most recent call last):
  File "/usr/bin/pynps", line 8, in <module>
    sys.exit(cli_main())
  File "/usr/lib/python3.8/site-packages/pynps/cli/cli.py", line 582, in cli_main
    theme_folder_name = get_theme_folder_name(f"{DLFOLDER}/Extracted/bgdl/t/")
  File "/usr/lib/python3.8/site-packages/pynps/functions/functions.py", line 731, in get_theme_folder_name
    comp = list(range(1, a[-1] + 1))
IndexError: list index out of range

From what I can tell it caused by the following line because the list 'a' is empty:
comp = list(range(1, a[-1] + 1))

NPS Browser has this feature, which makes it easier to track backups I have already. I believe it records all downloaded backups, DLC and updates under npsSettings.dat .

Hi.
What you think about adding a compression flag for psp iso images to convert them into .cso format using maxcso or similar software?

Hello,

I had an error while using pynps recently on Ubuntu 20.10. Could you please what I'm doing wrong with pynps?
I have a desktop with GNU/Linux Debian and everything is fine... I can't find the reason why it failes on Ubunut...

Regards!

thank you for your help!

Preliminary information:

installed version : 1.6.2 // through "pip3 install pynps" (updates frequently made)

pkg2zip installation : /home/$USER/.config/pyNPS/lib/ and make it executable by runing:

• ~/.local.bin/pkg2zip (64 bits - chmod'ed to be executable)
  OR
• ~.config/pyNPS/lib/pkg2zip (64 bits - chmod'ed to be executable)

Desktop: Linux HP 5.8.0-29-generic #31-Ubuntu SMP Fri Nov 6 12:37:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Command used : pynps -c psv -GDC -r eur
Command Steps:

• Finding the expected game : OK
• Downloading or Resuming downlaod : OK
• Extracting through pkg2zip : KO

Command output :
PKG2ZIP] Attempting to extract [PCSBxxxxxx] xxxxxxxxx
Traceback (most recent call last):
File "/home//.local/bin/pynps", line 8, in
sys.exit(cli_main())
File "/home//.local/lib/python3.8/site-packages/pynps/cli/cli.py", line 662, in cli_main
delete = run_pkg2zip(dl_dile_loc, dl_location, PKG2ZIP, pkg2zip_args, extraction_folder, i, zrif)
File "/home//.local/lib/python3.8/site-packages/pynps/functions/functions.py", line 699, in run_pkg2zip
process = runner(run_lst, cwd=output_location)
File "/home//.local/lib/python3.8/site-packages/pynps/functions/functions.py", line 636, in runner
p = subprocess.Popen(list, cwd=cwd,
File "/usr/lib/python3.8/subprocess.py", line 854, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home//.local/bin/pkg2zip'

Hi!
Can u include a query of macosx at get_system=="Darwin" -> like linux?