Giter Club home page Giter Club logo

juniper / libxo Goto Github PK

View Code? Open in Web Editor NEW
302.0 25.0 44.0 5.58 MB

The libxo library allows an application to generate text, XML, JSON, and HTML output using a common set of function calls. The application decides at run time which output style should be produced.

Home Page: http://juniper.github.io/libxo/libxo-manual.html

License: BSD 2-Clause "Simplified" License

Shell 1.30% C 68.87% Makefile 4.61% CSS 2.15% JavaScript 0.22% Perl 2.23% M4 1.91% Roff 16.14% Yacc 2.56%
freebsd xml json html cbor

libxo's Introduction

libxo

libxo - A Library for Generating Text, XML, JSON, and HTML Output

The libxo library allows an application to generate text, XML, JSON, and HTML output using a common set of function calls. The application decides at run time which output style should be produced. The application calls a function "xo_emit" to product output that is described in a format string. A "field descriptor" tells libxo what the field is and what it means.

Imagine a simplified wc that emits its output fields in a single xo_emit call:

    xo_emit(" {:lines/%7ju/%ju} {:words/%7ju/%ju} "
            "{:characters/%7ju/%ju}{d:filename/%s}\n",
            line_count, word_count, char_count, file);

Output can then be generated in various style, using the "--libxo" option:

    % wc /etc/motd
          25     165    1140 /etc/motd
    % wc --libxo xml,pretty,warn /etc/motd
    <wc>
      <file>
        <filename>/etc/motd</filename>
        <lines>25</lines>
        <words>165</words>
        <characters>1140</characters>
      </file>
    </wc>
    % wc --libxo json,pretty,warn /etc/motd
    {
      "wc": {
        "file": [
          {
            "filename": "/etc/motd",
            "lines": 25,
            "words": 165,
            "characters": 1140
          }
        ]
      }
    }
    % wc --libxo html,pretty,warn /etc/motd
    <div class="line">
      <div class="text"> </div>
      <div class="data" data-tag="lines">     25</div>
      <div class="text"> </div>
      <div class="data" data-tag="words">    165</div>
      <div class="text"> </div>
      <div class="data" data-tag="characters">   1140</div>
      <div class="text"> </div>
      <div class="data" data-tag="filename">/etc/motd</div>
    </div>

View the beautiful documentation at:

http://juniper.github.io/libxo/libxo-manual.html

Analytics

libxo's People

Contributors

0mp avatar akabaev avatar allanjude avatar brooksdavis avatar cbrueffer avatar gahr avatar olevole avatar philshafer avatar q66 avatar shamrin avatar trombonehero avatar ydnath avatar zvr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libxo's Issues

Output containing reserved characters isn't escaped allowing invalid JSON and XML

Some code we were running that used libxo added single quote characters to a string that ended up in a value (via a few levels of indirection) and we were surprised to find that invalid XML was produced. It would be helpful if libxo did this for us or at a minimum least provided a simple vis(3) like function that did the escaping required for the current format mode.

In our case I ended up using a strsnvis() to blindly encode XML and JSON special characters.

Linux: <sys/sysctl.h> missing

When I try to compile on Linux with GCC I get this error:

  CC       xo_syslog.lo
xo_syslog.c:61:10: fatal error: sys/sysctl.h: No such file or directory
   61 | #include <sys/sysctl.h>
      |          ^~~~~~~~~~~~~~

The only sysctl.h file I have on my system is linux/sysctl.h:

 $ find /usr/include/* -name 'sysctl.h'
/usr/include/linux/sysctl.h

Simply commenting out the line however helps too.
Maybe add a check for this header file to configure.ac?
On Ubuntu the path sys/sysctl.h seems to be only available in the dietlibc package.

xo_error() does not generate any JSON output

From Stefan Esser:

  1. xo_error() does not generate any JSON output. At least a warning
    should be generated in the XOF_WARN case, IMHO.

Yes, I don't have a good answer, but these are important enough
that a bad one will have to do. I'll emit these as "__error"
and "__warning".

create "pop until" clearing mechanism

Suggestion from Stefan Esser:

The overhead of clearing open containers, lists, and instances makes writing app code fairly annoying. Having a mechanism to record the current state and then clear to that state can simplify that logic:

xo_push_marker("some-internal-name");
call_some_function();
xo_pop_marker("some-internal-name");

The marker isn't emitted as output, but gives a means for the app to clear to a known state.

warn on implicit state transition

On 2015-02-11 13:19, Phil Shafer wrote:

Marcel Moolenaar writes:

BTW: Is there a way to have libxo do the state transitions
verbosely (i.e. emit a warning)? That way we don=E2=80=99t end up
with a bunch of broken utilities. Something cryptic would
be fine as I see it only being useful to check correctness
of utilities that are being modified (i.e. converted).

I can certainly add this, but there's no means currently. Should
it be part of the normal "warn" flag or something distinct?

In general, this implicit transitions are meant to help coders avoid
having to make some calls, as a convenience mechanism. Perhaps the
best think would be just to warn when the transition makes no sense
at all, such as when the xo_close_list("file") is seen after the
implicit transition has already closed it when the
xo_open_container("total") was seen.

Thanks,
Phil

That seems reasonable. But maybe also a 'strict' flag or mode,
especially for something like the test suite to use, to detect when
libxo was misused.

Allan Jude

XOF_UNDERSCORES: hyphens in topmost keys are not replaced

Tested on FreeBSD 13 (libxo 1.4.0), when adding the underscores flag, the hyphens in the topmost keys are never underscored.

For example:

% uptime --libxo json
{"uptime-information": {"time-of-day":"10:09AM", ...

% uptime --libxo json,underscores
{"uptime-information": {"time_of_day":"10:09AM", ...

For uptime-information, the hyphen is not replaced. I am not sure if this is intended.

xo_emit(3) in FreeBSD points to github page for manual and is not versioned appropriately; can diverge from source code

xo_emit(3) on FreeBSD points the reader to the github manual instead of providing a description of the lexicon, or an in-depth description of the APIs. The problem with this is that one day when libxo has matured enough in FreeBSD, and the libxo project has been under development long enough, the manual will diverge from the source code.

The documentation needs to be versioned appropriately, or needs to be distributed with FreeBSD and needs to reference appropriate paths under /usr/share/doc/... .

Fails to build with GCC on FreeBSD

Affects at least FreeBSD 11.3-RELEASE, 12.1-RELEASE, 13.0-RELEASE.

$ pkg install automake libtool gmake gcc9
$ sh bin/setup.sh
$ cd build
$ export MAKE=gmake CC=gcc9
$ ../configure --enable-warnings
$ gmake
[...]
../../libxo/xo_encoder.c: In function 'xo_encoder_discover':
../../libxo/xo_encoder.c:237:9: error: cast between incompatible function types from 'dlfunc_t' {aka 'void (*)(struct __dlfunc_arg)'} to 'int (*)(xo_encoder_init_args_t *)' {aka 'int (*)(struct xo_encoder_init_args_s *)'} [-Werror=cast-function-type]
  237 |  func = (xo_encoder_init_func_t) dlfunc(dlp, XO_ENCODER_INIT_NAME);
      |         ^

Core dump when compile with gcc 10

Hi all, i meet a segment fault when compiling libxo with gcc 10.
we can use these following commands to reproduce the problem:

yum install libtool git make
cp /usr/include/linux/sysctl.h /usr/include/sys/sysctl.h

git clone http://github.com/Juniper/libxo.git

sh bin/setup.sh
cd build
../configure
make
make install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/


[root@localhost build]# cat test.c
#include <libxo/xo.h>

int main(int argc, char *argv[])
{
   xo_emit("{d:/%-*.*s}{etk:name}{eq:flags/0x%x}", 0, 0, NULL, NULL, 0);
}

gcc test.c -g -L /usr/local/lib/ -lxo -I /usr/local/include/libxo -o test
./test
Segmentation fault (core dumped)

env info:

[lyc@localhost:lib]$ rpm -q glibc
glibc-2.34-124.oe2203sp2.aarch64
[lyc@localhost:lib]$ rpm -q kernel
kernel-5.10.0-153.12.0.92.oe2203sp2.aarch64
[lyc@localhost:lib]$ uname -a
Linux localhost 5.10.0-153.12.0.92.oe2203sp2.aarch64 #1 SMP Wed Jun 28 23:18:48 CST 2023 aarch64 aarch64 aarch64 GNU/Linux
[lyc@localhost:lib]$ gcc -v
using built-in specs。
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-linux-gnu/10.3.1/lto-wrapper
target:aarch64-linux-gnu
configure with:../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,fortran,objc,obj-c++,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl --without-cloog --enable-gnu-indirect-function --build=aarch64-linux-gnu --with-stage1-ldflags=' -Wl,-z,relro,-z,now' --with-boot-ldflags=' -Wl,-z,relro,-z,now' --disable-bootstrap --with-multilib-list=lp64 --enable-bolt
thread model:posix
Supported LTO compression algorithms: zlib

missing docs for 'l'

From Stefan Esser:

I just noticed, that the "l"
modifier (leaf list) is missing in xo_format.5 even in the
development branch - it is only documented in the source file ...

Missed it completely. Will add it.

Build broken in many ways for non-default prefix

When configuring with a non-default prefix, the package-config file is not correctly set up (it retains a load of ${} and @...@ values, and the rpath for the xo binary is set so that it can't find the shared library (it looks in /usr/local/lib).

1.3.1 Ubuntu 18.04 WSL missing so

./configure
make
make test
make install

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1# xo --version
xo: error while loading shared libraries: libxo.so.0: cannot open shared object file: No such file or directory

xo xohtml xolint xopo

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1# xohtml
xohtml: wrap libxo-enabled output in HTML
Usage: xohtml [options] [command [arguments]]
Valid options are:
-b | --base
-c | --command
-f | --file

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1# xolint
xolint [options] files ...
-c invoke 'cpp' on input
-C flags Pass flags to cpp
-d Show debug output
-D Extract xolint documentation
-I Print xo_info_t data
-p Print input data on errors
-V Print vocabulary (list of tags)
-X Print examples of invalid use

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1# xopo
xopo: error while loading shared libraries: libxo.so.0: cannot open shared object file: No such file or directory

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1#libxo-config
Usage: libxo-config [OPTION]

Known values for OPTION are:

--prefix=DIR change libxo prefix [default /usr/local]
--exec-prefix=DIR change libxo exec prefix [default /usr/local]
--libs print library linking information
--bindir print the bin directory
--cflags print pre-processor and compiler flags
--share print share directory
--help display this help and exit
--version output version information

root@DESKTOP:/mnt/c/wsl/libxo-1.3.1# libxo-config --cflags --libs --share
-I/usr/local/include -D_GNU_SOURCE -g -O2
-L/usr/local/lib -lxo
/usr/local/share/libxo

Man pages not escaping some newlines (PR included)

As the subject indicates, there are some cases where newlines are not escaped and found a case of a missing end quote. I have created a pull request: #78

I should look at the contribution guide in the docs, but wanted to get the PR and Issue in before I forgot.

Fix for man page xo_format.5

The following diff fixes quoting errors in the man-page:

Index: xo_format.5
===================================================================
--- xo_format.5 (revision 276665)
+++ xo_format.5 (working copy)
@@ -201,13 +201,13 @@
 .Pp
 .Bl -column M "Name12341234"
 .It Sy M "Name        Description"
-.It c "colon       " "A colon (":") is appended after the label"
+.It c "colon       " "A colon ("":"") is appended after the label"
 .It d "display     " "Only emit field for display styles (text/HTML)"
 .It e "encoding    " "Only emit for encoding styles (XML/JSON)"
 .It k "key         " "Field is a key, suitable for XPath predicates"
 .It n "no-quotes   " "Do not quote the field when using JSON style"
 .It q "quotes      " "Quote the field when using JSON style"
-.It w "white space " "A blank (" ") is appended after the label"
+.It w "white space " "A blank ("" "") is appended after the label"
 .El
 .Pp
 For example, the modifier string "Lwc" means the field has a label

Add a warning for duplicate siblings

From Stefan Esser:

Add a warning for duplicate siblings:

Any name must be only used once for all children of some node, either

  • for a leaf node (xo_emit)

  • for a container node

  • for a list (all instances in a contiguous list)

    If use of a name has completed (end of list, container or leaf) no
    further list, container or leaf is allowed to use the same name for
    a node belonging to the same parent.

.Nd entries in man pages are not correct for some man pages

According to mdoc(7), the ".Nd" macro is:
A one line description of the manual's content.

It looks like the .Nd macro has been cut and paste across several man pages
and is wrong.

./xo_create.3:.Nd emit formatted output based on format string and arguments
./xo_set_flags.3:.Nd emit formatted output based on format string and arguments
./xo_attr.3:.Nd emit formatted output based on format string and arguments
./xo_set_style.3:.Nd emit formatted output based on format string and arguments
./xo_set_info.3:.Nd emit formatted output based on format string and arguments
./xo_set_allocator.3:.Nd emit formatted output based on format string and arguments
./xo_flush.3:.Nd emit formatted output based on format string and arguments
./xo_open_list.3:.Nd emit formatted output based on format string and arguments
./xo_finish.3:.Nd emit formatted output based on format string and arguments
./xo_set_writer.3:.Nd emit formatted output based on format string and arguments
./xo_open_container.3:.Nd emit formatted output based on format string and arguments
./xo_emit.3:.Nd emit formatted output based on format string and arguments

libxo inconsistent field truncation

Hi again Phil!

Tested on FreeBSD 12-stable and 13.0-RELEASE:

I noticed that "ps" sets the maximum length of the command field to 2048 characters [POSIX2_MAX_LENGTH] via the instructlion "{:command/%-0..2048s}", even when"-ww" is specified. "procstat -e" doesn't. Sigh.

Anyway, that's another issue, but in the process of checking this, I discovered that whilst libxo honours the maxlength for text, and html, it doesn't for XML, JSON, or CSV

As an example, try running this, and replacing the libxo output format as appropriate:

env -i $(jot 500 | awk '{print "testvar_"$0"=dummydummydummydummy"}') sh -c 'ps -wwe -p $$ --libxo xml'

Is this intentional? I couldn't find anything in the docs mentioning this.

Is this related to a kludgy fix for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246514 ?

In my opinion, when or if or how fields should be truncated is a thorny issue, but whilst I find many implementations are messy, I still think it should be solely the responsibility of the calling program.

I.E. I don't think libxo should ignore the max width for some formats and not others...

Thoughts?

Cheers, Jamie

libxo output breaks if buffer exceeds 8kb

While converting the FreeBSD procstat utility to libxo, I noticed that if the output exceeds 8kb, and stdout is a file or a pipe, the content is truncated.

to reproduce, run some command that produces more than 8192 bytes of data via libxo, and redirect stdout to a file, or pipe it to less or similar. The output will be truncated.

Please make a compile-option that has libxo just become printf()s

Hi!
For (very) embedded applications we absolutely don't need any of the libxo plumbing for spitting out XML, JSON, etc. We may also not need it for smaller footprint builds like the rescue image builds.

Would it be possible/OK to have a version of libxo's API that just becomes printf()s?

close/reopen function for iterative output

NLD-json emits newline-delimited json objects. libxo can record open containers, lists, etc with keys to allow a 'recorking' function to close open constructs, emit a delimiter, and reopen those constructs.

Fix bug in xo_warn_c()

The xo_warn_c function calls xo_warn_hcv() with mis-ordered arguments:

xo_warn_hcv(NULL, 0, code, fmt, vap) should be xo_warn_hcv(NULL, code, 0, fmt, vap)

Index: libxo.c
===================================================================
--- libxo.c     (revision 276665)
+++ libxo.c     (working copy)
@@ -1027,7 +1033,7 @@
     va_list vap;

     va_start(vap, fmt);
-    xo_warn_hcv(NULL, 0, code, fmt, vap);
+    xo_warn_hcv(NULL, code, 0, fmt, vap);
     va_end(vap);
 }

check width of unicode characters

[potential problem which I want to record]

Perhaps a problem with unicode "variation selectors"?

\u+e29c85 \u+e284b9 \u+efb88f

should be:

✅ ℹ️

but maybe the second isn't zero width?

commas in numeric output

Add "hn-commas" to add commas in the output of large numeric values.

Will need a command line argument to turn it off.

Should be ignored when humanize is on.

No man page for xo_emit_field()

libxo seems to have convenient APIs to allow various compile-time field names to be used. However, sometimes in life, I have a set of dynamically determined field names, and it wasn't clear to me that the API made that convenient. Possibly I've suffered a documentation spelunking fail, in which case please just point me in the right direction and close. But here is an example:

#include <libxo/xo.h>

/*
 * Sometimes with libxo, I have a set of field names only known at run time.
 * For example, if a program self instruments using hardware performance
 * counters, and because of a limited number of hardware counter registers,
 * the user has to select up to 6 at a time, but we need to support hundreds
 * of potential values.  In this example code, I want to allow the field name
 * to be a string generated at run time .. but I wasn't sure how to do that in 
 * a way that was elegant with libxo.
 */
void
what_I_wanted_to_do(const char *variable, const char *value)
{

        xo_open_container("wanted_to_do");
        xo_emit("{:%s/%s}", variable, value);
        xo_close_container("wanted_to_do");
}

void
what_I_did(const char *variable, const char *value)
{
        char buffer[128];

        snprintf(buffer, sizeof(buffer), "{:%s/%%s}", variable);
        xo_open_container("did");
        xo_emit(buffer, value);
        xo_close_container("did");
}

int
main(int argc, char *argv[])
{

        xo_set_style(NULL, XO_STYLE_JSON);
        xo_set_flags(NULL, XOF_PRETTY);
        what_I_wanted_to_do("L1D_CACHE_ACCESS", "10");
        what_I_did("L1D_CACHE_ACCESS", "10");
        xo_finish();
        return(0);
}

Which outputs:

{
  "wanted_to_do": {
    "%s": "L1D_CACHE_ACCESS"
  },
  "did": {
    "L1D_CACHE_ACCESS": "10"
  }
}

In this scenario, the A72 core we're using has dozens of performance counter events that can be configured, but due to an architectural limit, only 6 can be used at a time. The strings for those registers are already defined in other headers, and I definitely don't want to retype them or do anything to replicate that. Instead, I want to have the user pass the names via the command line, validate them using the PMC library, and then use them as field names with libxo.

Pointers most welcome!

'plain' output with a defined field-separator

Hi. I notice that the output from "df" is tricky to parse, when the mount "device" and/or the mount-point contain spaces.. (yeah, i know... don't ask!!! )

"mount -p" uses (one or more) tab characters to separate fields, UNLESS the field is over 24 character in which case it uses space. (arrrrgh)

I see that the idea behind libxo is to enable processing of data, by not having to try and parse human-readable output. This is great, and just what I'd want.

'df' does indeed have libxo functionality, and whilst this would do the trick, parsing xml or json is overkill.
I was hoping it would be possible to have something like:

--libxo=delimited,delimiter="",record-separator=""

e.g.

--libxo=delimited,delimiter="\0",record-separator="\n"

or maybe being able to pass a format string in printf format?

Is this kind of thing generally useful?

In my particular case, I ended up writing a small C function just to grab the mount "device" and mount-point I needed, but it seemed to me that this what exactly the sort of thing libxo was meant to solve.

Am I missing something?

Cheers, Jamie

Feature request: Simplify definition of lists (omit need for explicit state in calling programs)

The functions xo_open_list() and xo_close_list() are only required for JSON and ignored for all other formats.

For well-formed JSON output, xo_open_list("tag") must be executed immediately before the first xo_open_instance("tag") and xo_close_list("tag") must follow the last xo_open_instance("tag").
Experience shows, that correct emission of xo_open/close_list() for optional lists requires explicit state variables and thus complicates the program code and creates significant effort to get tested and correct.

For examples of the complexity see the conditional xo_open_list() calls in https://reviews.freebsd.org/D1206 (pciconf) and https://reviews.freebsd.org/D1380 (netstat).

Both xo_open_list() and xo_close_list() can be automatically emitted, though:

  • xo_open_instance("tag") could check whether a list of the same name has already been opened and shall call xo_open_list("tag") if not.
  • Any xo_open_container(), xo_open_instance(), xo_emit() for a different tag needs to implicitly call xo_close_list("list_tag") to close the list, before a new node is generated for a tag not belonging into the list. The same is true when the enclosing element is closed.
  • Both xo_open_list() and xo_close_list() will continue to work and behave as before, but they need to maintain the state required for the automatic calls to these two functions as outlined above. This state consists of the name of the current list (parameter to xo_open_list()) and the information whether an list element has already been created with xo_open_instance() or xo_emit().

One effect of having "xo_open_list()" in a program is, that empty JSON arrays will be generated, if there is not a single xo_open_instance() for this list. This differs from XML output, where the xo_open_list() does not generate any output and lack of xo_open_instance() means, than no XML element will be emitted at all. If the empty JSON array is preferred, then use of xo_open_list() preserves this behaviour.

Not directly related but when the implicit xo_open_list() is implememted, it is just a minor modification:
xo_emit("{:tag/...}") within xo_open_list("tag") should generate the same output as if the "l" modifier was present (only for tags that are identical in xo_emit() and xo_open_list()).

To better warn in case of multiple lists with identical names being the child of one node, a new warning feature should be implemented: If a list has been closed (implicitly or explicitly), then any new xo_open_list for the same tag should emit a warning: In XML format, the lists will be considered to consinst of the concatenation of all elements. In JSON format, only the last list will be effective and all elements in any earlier list of the same name will be lost.

This situation is already an error when using the current API (which requires explicit xo_open/close_list() calls), and a warning should be generated for it.

Padding {P:} option can't take zero length string.

It seems like the following code will cause libxo to walk off the end of an array:

xo_emit("{P:}");

This ha something to do with it not accepting a zero length string for the 'P' (padding) modifier.

This causes some difficult to track down bugs due to what appears to be stack smashing.

Besides this bug, what this means that for potentially zero pads, you need additional logic. See the following workaround:

splbio/freebsd@31a2ccd

Better translation support

[ Writing up summary of discussion from the Ottawa departure lounge ]

It would be nice to make the formatting strings more amenable to translation, referring to fields by name and allowing reordering in the translated version. For example, if I want to print the string:

Push the red button.

In French this is:

Poussez le bouton rouge.

I want to translate this in two chunks, because the button colour is locale-dependent. This is typically done now with gettext as something like:

printf(_("Push the %s button."), _(red));

This becomes more complex in C if you have multiple formats, because you need to use the printf modifies to reorder them.
Languages like Python that have dictionaries as first-class citizens make this a bit easier, by passing a dictionary for the arguments with the format string including them by name.

It would be nice to change the xo_emit() calls to do the next best thing in C, where each of the variadic parameters are pairs of name and format string, followed by the variadic parameter. For example:

xo_emit("Push the {colour} button", "color,s", "red");

The translation of the format string would also allow other utilities to do the transformation, for example if we emit HTML, transform it, and then want to translate it at the end of a pipeline. libxo should assume that all format strings (though probably not all strings that are inserted, unless they have a modifier character) should be translated.

Feature request: binary output

We're using libxo as part of the SOAAP project to output the results of our security-oriented static analysis. It's working quite well, but when we apply the tool to applications like Chromium we get JSON files of over 1GB. These are hard to work with, not just because they're large, but because we can't seek in the file without parsing.

It would be helpful if libxo also supported a binary output format with some kind of length encoding so that we could skip large objects and arrays that we're not currently interested in. Perhaps an existing encoding could be employed... for instance, there are C bindings for Cap'n Proto (MIT license).

"xo" need option to make JSON object wrapper

$ xo --wrap "a/b/c" --json --pretty "{:name}{:value}\n" one a
"a": {
  "b": {
    "c": {
      "name": "one",
      "value": "a"
    }
  }
}

which isn't a complete json object. A "--object-wrapper" option should put the entire output inside a "{}" wrapper. This should be true for "--open" and "--close" as well.

Lots of implicit integer type conversion errors

Internally, it appears libxo is essentially only using two integer types, 'int' and 'unsigned'. Both of these are frequently 32bits even on 64bit systems.
Throughout libxo, pointer arithmetic is being done and storing the value in 'int' types. Nearly all of libxo/xo_buf.h's inline functions are this way.
Buffer sizes, lengths, and positive-only offsets should be size_t, and offsets that need negative values should be ssize_t.
Unfortunately, this affect nearly every type and function in libxo, including its API. Even the formatter function that is ostensibly derived from vsnprintf uses an int for buffer size instead of vsnprintf's size_t. Similarly, write() derived functions operate on ints instead of size_t and ssize_t.
This affects the internal wrapper functions xo_strn* that take a -1 and then do a strlen() on the input, since all strn* functions properly take a size_t for the buffer size, but the wrappers take an int in order to accommodate the -1 behavior.
There seems to be a lot of potential for vulnerabilities in just the integer type conversions and sign mismatches.

Versioned output

The outputs should be versioned, so if the content ever has to change, parses will be aware of the change.

Adding something like xo_version("version.string.here"); to the output, that is not displayed for the plain text version, but it part of the JSON/XML etc output.

Newline before closing brace in json output

libxo seems to be creating a newline before the last closing brace when doing non-pretty json output:

% uptime --libxo json
{"uptime-information": {"time-of-day":" 5:03PM","uptime":25201,"days":0,"hours":7,"minutes":0,"seconds":31,"uptime-human":" 7 hrs,","users":9,"load-average-1":0.41,"load-average-5":0.36,"load-average-15":0.34}
}

This is manifesting on all libxo tools I've tested on multiple versions of FreeBSD, up to 12-ALPHA6

add color role

Oleg Ginzburg writes:

I wanted to ask - is there some method to be used in the output xo ANSII
codes? For normal human text sometimes you want to draw a conclusion in color
to make it more human output ;)

Color would be a useful enhancement, but it can't be done by intermixing
escape sequnces with data. It would need to be something more explicit
and meaningful, something like:

xo_emit("The {k:name} {C:green}weighs{C:} {:weight/%d} pounds\n", name, weight);

The new "C" role would accept a name that would appear as escape
sequences for text and "style" attributes for HTML. The list of
color names should be freakishly small, to keep it portable, and
unknown colors would be ignored (with a warning).

Hmm... instead of directly making a color style attribute, I'd probably
opt for a color class, allowing CSS files to adjust it when/if needed.

And no, it won't support blinking ;^)

Thanks,
Phil

Add "--continuation" option to "xo" to allow multiple html lines

See https://lists.freebsd.org/pipermail/freebsd-hackers/2018-September/053304.html

If I choose HTML output I get the following invalid output which is missing
closing tags.

test1
,
2GB
test2
,
4GB

This can be fixed by adding "\n" to the end of the format lines, which may
be required, but it seems strange that I can so easily generate broken
output.

Yes, without a trailing newline, libxo can't declare the output
line "over", so it doesn't make the . The problem is that
there's no mechanism to "xo" that your output line is continued
from the previous one, and since "xo" is stateless, it will happily
start the next chunk of output with a new open div. I'll add a
"--continuation" option.

releases don't include manpages

Release tarballs have makefiles that reference manpages and they exist in the repository, but the manpages themselves are in the tarballs so the release don't build without patching.

FreeBSD's json format prints an unescaped new line before the final closing bracket

FreeBSD's --libxo json format prints an unescaped new line before the final closing bracket. Here is an example with a short output (jls with no jails, any other command will do):

% jls --libxo json
{"__version": "2", "jail-information": {"jail": []}
}

Notice the closing bracket in a new line (unescaped). The tests in libxo do not seem to include it. I am not sure if this is intended.

"xo" needs a means of handling lists and instances

See https://lists.freebsd.org/pipermail/freebsd-hackers/2018-September/053304.html

  1. "machines" is an object, which erroneously contains 2 "machine" keys

This is a bug. "xo" lacks a means of knowing that an object is a
list. One might expect that labeling "name as a key would perform
this, but it doesn't, and it's not clear that this is enough
information. Since "xo" is stateless, it cannot really handle
that "--wrapper machine" applies only to the first item. It
would have to know not to generate the closing "}" for machine,
but it's stateless, so I'd need a means of telling it that.

In C code, we have "xo_open_list" and "xo_open_instance", like:

https://libxo.readthedocs.io/en/latest/api.html?highlight=xo_open_instance#lists-and-insta
nces

But with "xo" I don't have lists and instances. I'll need to
find a means of adding these, though that would make "xo" more
cumbersome and it would lack the FSM (and the stack) that catches
missing calls in libxo.

    xo $opts --open-list machine
    for name in red green blue; do
        xo $opts --open-instance machine
        xo $opts "Machine {k:name} has {:memory}\n" $name `get-mem ~name`
        xo $opts --close-instance machine
    done
    xo $opts --close-list machine

Which is very exact but less that beautiful.

Thanks,
Phil

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.