dakusui / jq-front Goto Github PK
View Code? Open in Web Editor NEWA tool to empower your JSON
Home Page: https://dakusui.github.io/jq-front/
License: MIT License
A tool to empower your JSON
Home Page: https://dakusui.github.io/jq-front/
License: MIT License
As long as a yaml can be translated into a json file, it can be included in inheritance tree of jq-front's processing pipeline.
And it would be useful.
Provide a built-in function that performs multi-line replacement based on provided regex.
function replace_multilines() {
local string_before_replacement="${1}"
local begin_regex="${2}"
local end_regex="${3}"
local replacement="${4}"
...
echo "${string_after_replacement}"
}
replace_multilines 'function hi_there() {
echo "hi there"
}' 'hi_[a-z]+\(\)' '\}' 'hi_there() { echo "HI THERE" }'
Results in
function hi_there() { echo "HI THERE"}
I try to run the following json. But, I got error. How can I express for getting expectation?
json source
{ "k":"eval:string:\"'\\\"'\"" }
expectaion
{"k":"'\"'"}
Followiing doesn't render as expected.
{
"key": {
"hello world":"eval:string:$(parent $(curn))"
}
}
Expectation is following.
{
"key": {
"hello world":".\"key\""
}
}
And results in this.
{
"key": {
"hello world":""
}
}
Following should work.
{
"a":[1,2,3]
"v":"eval:array:$(ref $(cur).a)"
}
Right now, references in a string node is expanded in a following order and logic.
With this mechanism user can know and to some extent control how templating happens in case a node contains $(ref)
command.
However, we ultimately want to make it possible to reference a node from another without noticing such mechanism while in case a cyclic reference is warned as an error.
The current logic is awkward.
It can be improved as follows.
def path2pexp(v):
reduce .[] as $segment (""; .
+ ($segment
| if type == "string" then ".\"" + . + "\"" else "[\(.)]" end));
paths(scalars_or_empty)|path2pexp(.)
Since the docker mode execution has a significant performance overhead, we need to provide a way to process a number of files in batch.
Improve logic to merge objects.
Rather than simple multiplication (*
), following logic does a better job with regard to handling elements inside arrays.
def value_at($n; $p):
$n | getpath($p);
def merge_objects($a; $b):
$a | [paths(scalars_or_empty)]
| reduce .[] as $p ($b; setpath($p; value_at($a; $p)));
merge_objects({
"k":"k-a",
"k-a": "k-a",
"arr":[{"arro": "arro-a", "arro-a":"arro-a"}, {}, {}],
"o":{
"oo":"oo-a",
"oo-a":"oo-a"
}
}; {
"k":"k-b",
"k-b": "k-b",
"arr":[{"arro": "arro-b", "arro-b":"arro-b"}, {"arro1":"arro1-b"}],
"o":{
"oo":"oo-b",
"oo-b":"oo-b"
}
})
Following will achieve it.
function refexists() {
# shellcheck disable=SC2181
[[ $? == 0 ]] || abort "${_error_prefix}Failure was detected."
local _path="${1}"
has_value_at "${_path}" "$(cat "$(self)")"
}
function reftag() {
# shellcheck disable=SC2181
[[ $? == 0 ]] || abort "${_error_prefix}Failure was detected."
local _tagname="${1}"
local _curpath
_curpath="$(curn)"
while [[ "${_curpath}" != "" ]]; do
_curpath="$(parent "${_curpath}")"
if refexists "${_curpath}.${_tagname}"; then
ref "${_curpath}.${_tagname}"
return 0
fi
done
abort "${_error_prefix}The specified tag:'${_tagname}' was not found from the current path:'$(curn)'"
}
Improve behavior on an error during templating, where a failing command in a templated string is silently ignored.
It should abort the entire process of jq-front
.
Version number shown by docker version is not correct.
string node starts with 'template:' doesn't complain of circular dependency.
In order to avoid naming collision, I have renamed this project from 'powerjson' to 'jf'.
However, I found that there are quite a few projects in GitHub named 'jf'....
Let's rename it again to 'jq-front'.
When templating results in a malformed JSON, error message is not helpful.
jq-front
only gives an following message.
parse error: Invalid numeric literal at line 2, column 0
Stack trace on the situation is following
DEBUG: error trapped
at 6 _debug_abort /home/hiroshi/opt/jq-front/lib/helpers.sh
at 102 _render_text_node /home/hiroshi/opt/jq-front/lib/templating.sh
at 37 _perform_templating /home/hiroshi/opt/jq-front/lib/templating.sh
at 13 perform_templating /home/hiroshi/opt/jq-front/lib/templating.sh
at 40 perform_jqfront /home/hiroshi/bin/jq-front
at 118 main /home/hiroshi/bin/jq-front
at 161 main /home/hiroshi/bin/jq-front
Make it possible to use user custom function by following syntax.
{
"$extends": [
"SS.sh;SOURCE"
],
"key": "eval:string:hello_world=$(hello_world d),$(echo HELLO)"
}
If SS.sh
has a following content,
function hello_world() {
echo "Hello, world. My Function!"
}
then, following should be rendered.
{
"key": "hello_world=Hello, world. My Function!,HELLO"
}
jq-front version: The latest version as of Nov 5, 2019
{
"$local": {
"nodeA": {
"aa": "aa"
},
"nodeB": {
}
},
"a": [
{
"$extends": ["nodeA"],
"a": "a"
}
]
}
./jq-front test.json
{
"a": [
{
"aa": "aa",
"a": "a"
}
]
}
{
"a": [
{
"a": "a"
}
]
}
{
"$local": {
"database_default": {
"server": {
"ip": "192.168.1.5",
"port": 2000
},
"db_name": "test",
"user": {
"name": "root",
"password": "root"
}
}
},
"foo_database": {
"$extends": "database_default",
"server": {
"port": 2001
},
"db_name": "foo",
"user": {
"password": "foo_root"
}
}
}
{
"foo_database": {
"server": {
"ip": "192.168.1.5",
"port": 2001
},
"db_name": "foo",
"user": {
"name": "root",
"password": "foo_root"
}
}
}
jq: error (at <stdin>:1): Cannot iterate over string ("database_d...)
{
"foo_database": {
"server": {
"port": 2001
},
"db_name": "foo",
"user": {
"password": "foo_root"
}
}
}
ref function does not resolve recursively eval: when it is referencing an object (?)
Suppose we have
{
"obj": {
"c": "eval:string:$(ref $(cur).a)",
"a": "Howdy"
}
"key": "eval:object:$(ref $(cur).obj)"
}
In this situation, since .key.c
doesn't appear in the list of attributes to be templated initially created, the node will have a value eval:string:$(...
, not Howdy
.
Probably we should repeat perform_templating
function until all the templating keywords (eval:
and template:
) go away.
The condition to reproduce this issue is not clear yet, though, jq-front sometimes with following message.
jq: error (at <unknown>): Cannot index string with number
ERROR: Failed to merge object nodes:'{"
This hasn't been seen until v0.16.
Following is the stacktrace that jq-front prints.
}
}': error: /tmp/tmp.bTr8hOOqjN
at 36 abort /home/hiroshi/opt/jq-front/lib/shared.sh
at 99 _merge_object_nodes /home/hiroshi/bin/jq-front
at 568 expand_nodelevel_inheritances /home/hiroshi/bin/jq-front
at 632 run_jqfront /home/hiroshi/bin/jq-front
at 156 _expand_nodelevel_inheritances /home/hiroshi/bin/jq-front
at 565 expand_nodelevel_inheritances /home/hiroshi/bin/jq-front
at 632 run_jqfront /home/hiroshi/bin/jq-front
at 522 expand_filelevel_inheritances /home/hiroshi/bin/jq-front
at 624 run_jqfront /home/hiroshi/bin/jq-front
at 648 perform_jqfront /home/hiroshi/bin/jq-front
at 704 main /home/hiroshi/bin/jq-front
at 739 main /home/hiroshi/bin/jq-front
Support 'super' built-in function, which allows a user to access attributes in the "super" JSON node.
"super" node can be defined as a node where an empty object extends all the nodes specified by "$extends" in the current node.
In node-level inheritances, overriding priority is not consistent with file-level's and not correct.
{
"child": {
"$extends": [
"A.json",
"B.json"
]
}
}
Should become
{
"child": {
"b": "B",
"o": "A",
"a": "A"
}
}
But it gives
{
"child": {
"b": "B",
"o": "B",
"a": "A"
}
}
This issue was found in until v0.37
{
"releaseVersion": "2.12.0",
"snapshotVersion": "template:string:$(ref .releaseVersion)-SNAPSHOT"
}
{
"releaseVersion": "2.12.0",
"snapshotVersion": "template:$(ref .releaseVersion)-SNAPSHOT"
}
These will be rendered to
{
"releaseVersion": "2.12.0",
"snapshotVersion": "2.12.0-SNAPSHOT"
}
And
{
"objectNode": {"hello": "world"},
"snapshotVersion": "template:object:$(ref .objectNode)"
}
will become
{
"objectNode": {
"hello": "world"
},
"snapshotVersion": {
"hello": "world"
}
}
JF_PATH is not respected as designed when jf is used without docker
Make processing program configurable through a .rc
file.
If you have a file .jq-front.rc
with following content,
function resolve_processor() {
local _absfile="${1}"
local _processor=""
if is_localnode "${_absfile}" || [[ ${_absfile} == *.json ]]; then
_processor="jq ."
elif [[ "${_absfile}" == *.yaml || "${_absfile}" == *.yml ]]; then
_processor="yaml2json"
elif [[ "${_absfile}" == *.sh ]]; then
_processor="bash -eu"
elif [[ "${_absfile}" == *.py ]]; then
_processor="python"
fi
echo "${_processor}"
}
you become able to use python
not only JSON, yaml, and bash script files.
For instance, following prints the entire file, which can be very huge.
ERROR: Failed to expand node level inheritance for node:'{...
It is necessary and cumbersome to specify the number of levels to climb for the parent
built-in function.
That is,
"eval:$(parent $(parent $(parent $(cur))))"
Instead, I want to be able to write the same thing as follwos
"eval:$(parent $(cur) 3)"
The default value for the parameter will be 1
and we can keep the backward compatibility.
If you give 0
to the parameter, it will result in the same result as the given path itself.
This object results in an empty object {}
.
{
"key1": {
"key2": {
"key3": false
}
}
}
Following is rendered correctly.
{
"key":{
"booleanValue": true
}
}
By introducing following syntax, make it possible to inherit an optional file.
"$extends": [ "optionalFile.json?" ]
When a file name ends with ?
, extend it if it exists, otherwise simply ignore it without throwing an exception.
Following unnecessary message is written to stderr.
jq: error (at <stdin>:1): null (null) has no keys
Found in v0.28
I have a following key-value.
{
"hello": "HELLO",
"world": "WORLD",
"value": "eval: $(ref .hello) $(echo $(ref .world))"
}
The intention is to render following element.
"value": "HELLO $(echo WORLD)"
However, jq-front
renders this into :
"value": "HELLO WORLD"
Because it just considers the given data as a nested command substitution.
How can I define the element to get the desired string?
A node that references outside with relative(parent) manner cannot be referenced.
{
"key": "helloWorld",
"child": {
"childKey": "eval:string:$(ref $(parent $(cur)).key)"
}
}
This renders to following.
{
"key": "helloWorld",
"child": {
"childKey": "helloWorld"
}
}
However, this doesn't and results in an error.
{
"key": "helloWorld",
"child": {
"childKey": "eval:string:$(ref $(parent $(cur)).key)"
}
"a": {
"layer2" : {
"layer3": "eval:string:$(ref .child.childKey)"
}
}
}
Why because, the attribute layer3
is rendered to eval:string:$(ref $(parent $(cur)).key)
, first.
{
"key": "helloWorld",
"child": {
"childKey": "eval:string:$(ref $(parent $(cur)).key)"
}
"a": {
"layer2" : {
"layer3": "eval:string:$(ref $(parent $(cur)).key)"
}
}
}
And from .a.layer2.layer3
, there is no path specified by $(parent $(cur)).key
, which is computed to .a.key
.
Since jq-front is designed to render text nodes in alphabetical and path length order you can come up with a work around where the rendering for layer3
happens first.
However it is quite an error-prone way and you would not prefer it.
In this situation, it is better to resort to reftag
built-in function as follows.
{
"key": "helloWorld",
"child": {
"childKey": "eval:string:$(ref $(parent $(cur)).key)"
}
"a": {
"layer2" : {
"layer3": "eval:string:$(reftag key)"
}
}
}
reftag
tries to find a key which has a name specified by the argument given to it by climbing up a node path from current ($(cur)
).
Nevertheless, it is a very expensive built-in as of now and it is not advised to use it unnecessarily.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.