Request for jq and json2csv about ios_system HOT 19 OPEN

Used wabt to convert the wasm to c. It was over 32MB and 1M lines long 😅. It must have the entire Go stdlib in it. I tried compiling it, but killed it when it had consumed 18GB of RAM on my system.

from ios_system.

This similarly produces no output.

from ios_system.

Also, I pushed my entire working directory for the jq port in case you want to see the changes and add it before I get the "build recipe" script done.

https://github.com/fprimex/jq

The branch is iOS, which I set to the default.

from ios_system.

Hi,
the current TestFlight version for a-Shell mini now has jq, thanks to @fprimex: https://testflight.apple.com/join/REdHww5C

You can also install json2csv, which has been compiled to WebAssembly, using: curl -OL https://github.com/holzschu/a-Shell-commands/releases/download/0.1/json2csv, followed by mv json2csv ~/Documents/bin.

from ios_system.

I have jq working in the simulator and am trying to finish up getting it added. I'm not familiar with json2csv. Do you have a link to it?

from ios_system.

from ios_system.

I had a quick look and the difficulty here is that the program is written in Go. This language toolset is not yet working with ios_system. I compiled it and it produces neither output nor files.

GOOS=js GOARCH=wasm go build -o json2csv.wasm

Go can natively produce both wasm and ios/arm64 formats, however the wasm SDK is not the one needed for to produce compatibly ios_system output.

The ios/arm64 output would work, but Go does not support outputting dynamic, shared libraries on ios/arm64. I tried combinations of -buildmode=shared, buildmode=c-shared, and -linkshared but none were accepted.

Honestly, your best bet for getting json2csv is to port it to python or jq. jq has an @csv output target. I would like to see Go work with ios_system at some point, but I feel like that is probably a long ways off.

from ios_system.

I would be interested in decompiling the resulting Go wasm, replacing the needed SDK calls, and then reassembling it. That would enable Go programs in ios_system. I'll try that later.

from ios_system.

I've been thinking. WASM should be portable (it is designed for that). The only potentially non-portable parts are when it tries to access the file system, which is usually encapsulated into JavaScript commands. What kind of errors do you get with json2csv.wasm?

from ios_system.

IIRC it just doesn't work. No output, no errors, no files.

from ios_system.

That doesn't make a lot of clues, I'll grant you that. But normally, a webAssembly command that does not execute produces an error, so "no errors" could be a positive sign. Since webAssembly does not have access to the file system, I would try with stdin and stdout (json2csv < input.json > output.csv).

from ios_system.

Go wasm does not provide wasi and therefore cannot be used in the context ios_system / a-shell wants. The tinygo provider, however, does and compiling with it works fine. Unfortunately, tinygo does not support everything that Go does, and the resulting wasm "panics" with a reflect.NumMethod not implemented error. This is apparently common for tinygo programs using encoding/json.

I'm looking at one other option and then I think I'm out of ideas. The good news is that using jq to do this would be relatively simple.

from ios_system.

I haven't looked at the Go documentation, but if their WebAssembly version comes with a JavaScript container, maybe we can run this JS container rather than ours? jsc lets you run JavaScript commands, and the code a-Shell uses to run WebAssembly is in wasm.js. If it works, we will have to design a way to switch between JS containers depending on the WebAssembly flavour.

from ios_system.

Here is the working wasm file for json2csv to place in Documents/bin. GH made me zip it so I could upload it 🙄 , so it'll need to be unzipped as well.

json2csv.wasm.zip

from ios_system.

Thanks for compiling this command.
I've made it available on https://github.com/holzschu/a-Shell-commands/ but when I test, I get blank output every time. I might be using it wrong, though.

from ios_system.

I think that version is old and broken. I have attached a tar.gz with the wasm and some tests that can be run that are from the project's acceptance tests and docs. See the tests.txt file for what to run and the expected output (there will be some newline representation differences because of how I copied the tests over most likely).

json2csv_wasm.tar.gz

from ios_system.

It works with your tests, and it works with a metadata.json file that came with a Python package, but with this colors.json, it only produces a lot of blank lines.
colors.json.zip

I'm not knowledgeable enough to know whether this is normal or not.

from ios_system.

colors.json, and basically any single JSON value, isn't going to work for this.

Converts a stream of newline separated json data to csv format.

colors.json is a single JSON value that is on multiple lines. json2csv wants many JSON values, one on each line. So I think the blank lines are a result of it reading each line, failing to do anything with them, but continuing. I figured there would be some stderr since there are some messages in there, but 🤷

from ios_system.

Thanks a lot for cross-compiling jq. I'm including it in the next TestFlight version.

One point I haven't mentionned so far: ios_system does not clear memory or re-initialize variables when you leave a command (because there is no fork/exec, so the command is still in the process memory space, so variables keep their values). Depending on how the command is coded, this can result in memory leaks or options that keep being set on subsequent runs (you run "command -v", so this one is verbose, and then "command", but it is still verbose because the verbose flag is still set from the previous run). Again, that depends on how the command has been coded (some reinitialize all variables and free everything, others don't). I tried looking at the source of jq, but I couldn't find where it parses options.

Update: it seems to hold in my tests. I had a lot of issues with awk, who has a similar syntax. I guess the programmation style of jq is more modern and robust.

from ios_system.