xforce / eve-echoes-tools Goto Github PK

Hi

I now have my image:

$ docker images

REPOSITORY      TAG      IMAGE ID    CREATED      SIZE
cookiemagic/evee-tools  latest    3a77af58e520    13 minutes ago  85.6MB
evee-tools        latest    21f8d10fe6cf    4 hours ago    1.25GB

evee-tools (old version) works fine
but cookiemagic/evee-tools (new version) crashes (on the same eve.xapk, in a different folder).

Here are errors I receive:

$ docker run --rm -v$(pwd):/data cookiemagic/evee-tools dump_static /data/eve.xapk /data/staticdata
2020-10-06 11:35:55,964 WARN [npktool] Handle unknown special case! Changing field 28
2020-10-06 11:35:58,535 INFO [npktool] Done.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "scripts/dump_static_data.py", line 158, in dump_script_unpack
return dump_script(*args)
File "scripts/dump_static_data.py", line 89, in dump_script
script_redirect_out = execute_stdout([sys.executable, "neox-tools/scripts/script_redirect.py", os.path.join(script_extract_dir,
File "scripts/dump_static_data.py", line 75, in execute_stdout
raise e
File "scripts/dump_static_data.py", line 70, in execute_stdout
output = subprocess.check_output(
File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'neox-tools/scripts/script_redirect.py', '/tmp/tmp200xj8ot/unknown_file_name/E302128602225F51.nxs']' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "scripts/dump_static_data.py", line 344, in
dump_scripts(apk)
File "scripts/dump_static_data.py", line 183, in dump_scripts
pool.map_async(dump_script_unpack, files).get(9999999)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'neox-tools/scripts/script_redirect.py', '/tmp/tmp200xj8ot/unknown_file_name/E302128602225F51.nxs']' returned non-zero exit status 1.
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'
b'Traceback (most recent call last):\n File "neox-tools/scripts/script_redirect.py", line 45, in \n main()\n File "neox-tools/scripts/script_redirect.py", line 36, in main\n data = script_redirect.unnpk(open(input_file, "rb").read())\nAttributeError: module 'script_redirect' has no attribute 'unnpk'\n'

It seems data url hosted aws is broken. Would you please update data url or give some suggestion on how to create xapk from apk and obb files?

Hoping to start a discussion around understanding some of the API endpoints that can be found in the data dump. Specifically:

• https://unisdk.update.easebar.com/
• https://optsdk.gameyw.netease.com/

Has anyone had any success reverse engineering these endpoints for useful information? I've been running some proxies for monitoring game traffic and have really only seen a few calls to some of these endpoints while the game loads but nothing that provides any immediate useful data (i.e. market data). The references to the APIs in the dumped Python code seem to be incomplete. For example, I see references to an envsdk module which seems to use the ENV_SDK_HOST endpoint but the code for it is missing in the dump.

Would be interested in seeing what others have found.

Absolutely amazing work! I've learned an incredible amount on how the map data is saved and presented because of this. I'm curious though, where are the GUI files? Icons and background images and such...

When I run your docker with the command:
docker run -v$(pwd):/data cookiemagic/evee-tools fsd2json xx.sd

I recieve the following error:
standard_init_linux.go:211: exec user process cause "exec format error"

looking that up, there may be a .sh file somewhere that doesn't lead with #!/bin/bash

Have all the data, just want to write some script to automatically extract it from the python scripts and dump it out into json.
Has some nice stuff

Hi.

First thanks for this tools, it's really awesome.

Maybe I miss something but I don't find a way to use res/staticdata/gettext/msg_index/index.json (once extracted).
Values match locales res/staticdata/gettext/<locale>/* but keys doesn't match any objects id.

Have you any idea ?

Does not in staticdata, I can not find it

run ./build.sh

[+] Building 144.7s (16/25)
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 1.17kB                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 34B                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:edge                                                                                  3.1s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                   0.0s
 => CACHED https://api.github.com/repos/xforce/eve-echoes-tools/compare/main...HEAD                                                             0.0s
 => [internal] load build context                                                                                                               0.2s
 => => transferring context: 177.72kB                                                                                                           0.1s
 => CACHED [cargo-build  1/10] FROM docker.io/library/alpine:edge@sha256:2f77b6664f181b246244f9cd052155e74fb3f26d2a502edecd5fff0fc4bda388       0.0s
 => CACHED [cargo-build  2/10] RUN apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing rust cargo                                      0.0s
 => CACHED [cargo-build  3/10] WORKDIR /app                                                                                                     0.0s
 => CACHED [cargo-build  4/10] ADD Cargo.toml Cargo.toml                                                                                        0.0s
 => CACHED [cargo-build  5/10] ADD fsd2json/src /app/fsd2json/src                                                                               0.0s
 => CACHED [cargo-build  6/10] ADD fsd2json/Cargo.toml /app/fsd2json/Cargo.toml                                                                 0.0s
 => CACHED [cargo-build  7/10] ADD neox-tools/src /app/neox-tools/src                                                                           0.0s
 => CACHED [cargo-build  8/10] ADD neox-tools/Cargo.toml /app/neox-tools/Cargo.toml                                                             0.0s
 => CACHED [cargo-build  9/10] RUN cargo install --path fsd2json --root /usr/local/                                                             0.0s
 => ERROR [cargo-build 10/10] RUN cargo install --path neox-tools --root /usr/local/                                                          141.3s
------
 > [cargo-build 10/10] RUN cargo install --path neox-tools --root /usr/local/:
#15 0.341   Installing npktool v0.1.0 (/app/neox-tools)
#15 0.473     Updating crates.io index
#15 2.798  Downloading crates ...
#15 4.281   Downloaded memchr v1.0.2
#15 4.327   Downloaded murmur3 v0.5.1
#15 4.415   Downloaded num-bigint v0.3.2
#15 4.550   Downloaded memchr v2.4.0
#15 4.591   Downloaded nom v3.2.1
#15 4.603   Downloaded num-iter v0.1.42
#15 4.622   Downloaded ppv-lite86 v0.2.10
#15 4.638   Downloaded rand_chacha v0.2.2
#15 4.656   Downloaded rand_core v0.5.1
#15 4.668   Downloaded terminal_size v0.1.17
#15 4.678   Downloaded scopeguard v1.1.0
#15 4.701   Downloaded smallvec v1.6.1
#15 4.738   Downloaded tree_magic v0.2.3
#15 4.745   Downloaded cfg-if v0.1.10
#15 4.759   Downloaded fixedbitset v0.2.0
#15 4.773   Downloaded lock_api v0.3.4
#15 4.806   Downloaded indicatif v0.16.2
#15 4.828   Downloaded num-rational v0.3.2
#15 4.860   Downloaded parking_lot_core v0.7.2
#15 4.864   Downloaded fnv v1.0.7
#15 4.871   Downloaded number_prefix v0.4.0
#15 4.892   Downloaded getrandom v0.1.16
#15 4.927   Downloaded parking_lot v0.10.2
#15 5.119   Downloaded petgraph v0.5.1
#15 5.163   Downloaded rand v0.7.3
#15 5.423   Downloaded regex-syntax v0.6.25
#15 5.445   Downloaded num-complex v0.3.1
#15 5.638   Downloaded regex v1.5.4
#15 5.671   Downloaded console v0.14.1
#15 11.53   Downloaded compress v0.2.1
#15 11.63   Downloaded num v0.3.1
#15 11.64   Downloaded aho-corasick v0.7.18
#15 11.66    Compiling libc v0.2.98
#15 11.66    Compiling getrandom v0.1.16
#15 11.99    Compiling memchr v2.4.0
#15 12.25    Compiling scopeguard v1.1.0
#15 12.35    Compiling cfg-if v0.1.10
#15 12.39    Compiling smallvec v1.6.1
#15 12.41    Compiling ppv-lite86 v0.2.10
#15 12.81    Compiling fixedbitset v0.2.0
#15 13.06    Compiling regex-syntax v0.6.25
#15 14.08    Compiling number_prefix v0.4.0
#15 14.36    Compiling fnv v1.0.7
#15 14.45    Compiling murmur3 v0.5.1
#15 14.62    Compiling num-traits v0.2.14
#15 14.84    Compiling num-integer v0.1.44
#15 15.06    Compiling num-bigint v0.3.2
#15 15.53    Compiling num-rational v0.3.2
#15 15.75    Compiling num-iter v0.1.42
#15 15.99    Compiling lock_api v0.3.4
#15 25.00    Compiling atty v0.2.14
#15 25.11    Compiling time v0.1.44
#15 28.59    Compiling terminal_size v0.1.17
#15 28.79    Compiling memchr v1.0.2
#15 29.10    Compiling parking_lot_core v0.7.2
#15 30.33    Compiling aho-corasick v0.7.18
#15 40.82    Compiling num-complex v0.3.1
#15 41.61    Compiling petgraph v0.5.1
#15 42.25    Compiling rand_core v0.5.1
#15 42.96    Compiling colored v1.9.3
#15 44.81    Compiling console v0.14.1
#15 45.36    Compiling nom v3.2.1
#15 49.94    Compiling parking_lot v0.10.2
#15 50.58    Compiling regex v1.5.4
#15 65.71    Compiling chrono v0.4.19
#15 73.76    Compiling rand_chacha v0.2.2
#15 75.86    Compiling tree_magic v0.2.3
#15 89.60    Compiling indicatif v0.16.2
#15 94.75    Compiling simple_logger v1.13.0
#15 97.00    Compiling rand v0.7.3
#15 101.1    Compiling num v0.3.1
#15 101.2    Compiling clap v3.0.0-beta.2
#15 104.1    Compiling compress v0.2.1
#15 141.0    Compiling npktool v0.1.0 (/app/neox-tools)
#15 141.3 error[E0716]: temporary value dropped while borrowed
#15 141.3    --> neox-tools/src/main.rs:688:37
#15 141.3     |
#15 141.3 688 |                     pb.set_message(&format!("{}", file_name));
#15 141.3     |                     ----------------^^^^^^^^^^^^^^^^^^^^^^^^-- temporary value is freed at the end of this statement
#15 141.3     |                     |               |
#15 141.3     |                     |               creates a temporary which is freed while still in use
#15 141.3     |                     argument requires that borrow lasts for `'static`
#15 141.3     |
#15 141.3     = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
#15 141.3
#15 141.3 error: aborting due to previous error
#15 141.3
#15 141.3 For more information about this error, try `rustc --explain E0716`.
#15 141.3 error: failed to compile `npktool v0.1.0 (/app/neox-tools)`, intermediate artifacts can be found at `/app/target`
#15 141.3
#15 141.3 Caused by:
#15 141.3   could not compile `npktool`
#15 141.3
#15 141.3 To learn more, run the command again with --verbose.
------
executor failed running [/bin/sh -c cargo install --path neox-tools --root /usr/local/]: exit code: 101

macos 10.15.7
docker Server version 20.10.7

Hi
First thanks for your great work.
I used your prior version of the container, but since your last update I can not manage to build the image anymore.

I have an issue while building the docker image.
It failed to compile fsd2json v0.1.0 (/app/fsd2json)

Here are my errors:

Step 10/22 : RUN cargo install --path fsd2json --root /usr/local/
---> Running in f3dd406b601a
Installing fsd2json v0.1.0 (/app/fsd2json)
Updating crates.io index
warning: spurious network error (2 tries remaining): failed to resolve address for github.com: Name does not resolve; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for github.com: Name does not resolve; class=Net (12)
error: failed to compile 'fsd2json v0.1.0 (/app/fsd2json)', intermediate artifacts can be found at '/app/target'
Caused by:
failed to get 'byteorder' as a dependency of package 'fsd2json v0.1.0 (/app/fsd2json)'
Caused by:
failed to load source for dependency 'byteorder'
Caused by:
Unable to update registry 'https://github.com/rust-lang/crates.io-index'
Caused by:
failed to fetch 'https://github.com/rust-lang/crates.io-index'
Caused by:
network failure seems to have happened
if a proxy or similar is necessary 'net.git-fetch-with-cli' may help here
https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
Caused by:
failed to resolve address for github.com: Name does not resolve; class=Net (12)

I have tried to change the Dockerfile with RUN CARGO_NET_GIT_FETCH_WITH_CLI=true cargo install --path fsd2json --root /usr/local/ .... without success... now I have:

---> Running in 9bf64c8a4944
Installing fsd2json v0.1.0 (/app/fsd2json)
Updating crates.io index
error: failed to compile 'fsd2json v0.1.0 (/app/fsd2json)', intermediate artifacts can be found at '/app/target'
Caused by:
failed to get 'byteorder' as a dependency of package `fsd2json v0.1.0 (/app/fsd2json)'
Caused by:
failed to load source for dependency 'byteorder'
Caused by:
Unable to update registry 'https://github.com/rust-lang/crates.io-index'
Caused by:
failed to fetch 'https://github.com/rust-lang/crates.io-index'
Caused by:
could not execute process 'git fetch --force --update-head-ok 'https://github.com/rust-lang/crates.io-index' 'refs/heads/master:refs/remotes/origin/master'' (never executed)
Caused by:
No such file or directory (os error 2)

Am I doing something wrong ?

Thanks in advance for your help

Regards

Hi! This is not really an issue on your submission but a question.

Have you seen the new Netease's engine Messiah? It uses MPK files and they seem to have a different encryption system. Do you think they might be using something similar to the NPK you figured out here?

I uploaded a few files here in case you want to see what I mean:
https://www.mediafire.com/file/0t38kkqwvjxkqxd/e46a07c1c54d66e3e77251ce383cfc3a/file
https://www.mediafire.com/file/hslml9a0qddk75z/0f98fac7babf2dfab4c388ff67a281b1/file
https://www.mediafire.com/file/yl58ph0ytj96hxm/ef2f6ecf63a082be43aa04f839f7d578/file

Any help on how to get access to those files would be really appreciated, thanks!

Hi, i'm trying to run docker img, but always stoped on one place, can't understand what's wrong

docker run -v$(pwd):/home/root/ cookiemagic/evee-tools dump_static /home/root/eve.xapk /home/root/staticdata

/home/root/eve.xapk
/home/root/staticdata
/opt/eve-echoes-tools /opt/eve-echoes-tools
Traceback (most recent call last):
File "scripts/dump_static_data.py", line 510, in
dump_static_data_fsd(xapk_temp_dir)
File "scripts/dump_static_data.py", line 318, in dump_static_data_fsd
for filename in os.listdir(obb_path):
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpqld7cygx/Android/obb/com.netease.eve.en'
/opt/eve-echoes-tools

error.log
Hello again,
Thanks for your help on the previous issue. I'm attempting another round and I got another error . . .

This time there is a panicked thread with a error about an integer being too large. I have attached a log generated with RUST_BACKTRACE=full.

I went and recompiled fsd2json, and didn't recieve any errors at compile time.

Thanks again for all your time and help! Not to mention building this in the first place :)

Would love some assistance here if anyone has figured this out: how would one translate the zh_name field on all of the items to their English counterpart? My first attempt was looking up the actual string found in zh_name in the respective gettext/zh files, however, I only find results for about half of the total items.

Some of the cases that fail seem to be when the name contains ASCII characters, like this one:

衍生无人机 - LMH-M-1 - B

That string does not exist in the zh files, however, if we do a little reverse engineering we can find that the likely locality id is 14029 which is Metadrones - LMH-M-1 - B in English and 衍生无人机 - 露姆赫-姆-1 - 比 in Chinese. It's not clear to me why in the item file there is ASCII characters but in the locality file, there isn't. I believe I was getting like 29k items in total but only 16k of them could make a direct match back to the Chinese locality data.

Other than reverse looking up like I tried above, it's not clear how one is supposed to tie an item name to its translated name in the locality files. I tried searching around for various locality ID's and most only show up in the locality files and nowhere else. Am I missing data or am I looking at all of this wrong?